Compare commits

...
19 Commits
Author SHA1 Message Date
ducoterra bef24e05e2 phase: 123_chat_image_questions
Build and Push Containers / build-and-push-app (push) Successful in 1m54s
Build and Push Containers / build-and-push-db (push) Failing after 1m50s
All gates green. Verification complete.

**Phase 123 — final verification pass (all 4 tasks already in `complete/`)**

- Verified the full implementation is in the working tree: `app/api/chat_images.py` (upload/serve pair), `ChatRequest.image`/`ChatMessage.image` (path-validated, omitted-when-None), toggle-off + stale-file hinted error frames, `build_user_content` multimodal build at both sites (chat.py deflected branch + `run_agent`), config-gated composer attach/preview/upload-then-send, restore + shared rendering, CSP `img-src 'self' data:` carve-out, mock-LLM capture buffer.
- `uv run pytest` → **2796 passed**, exit 0 (unit + integration).
- `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (29/4615 missed; phase-123 modules 99–100%).
- `uv run pytest tests/e2e/test_chat_image_questions.py -v --no-cov` → **5 passed** in isolation.
- `uv run ruff check . && uv run pyright` → clean (0 errors).

**Completion criteria:** (1) attach→send→multimodal text+image to the model, bubble/reload/shared all render it, saved chat stores the PATH with `"base64" not in json.dumps(stored)` — **verified** (E2E tests 1–4 + integration round-trip); (2) `BOR_IMAGES=false` — control hidden, exact hinted error frame, zero model calls / no query_log row — **verified** (E2E test 5 + integration); (3) text-only byte-identical (`content` stays a plain `str`) — **verified** (unit + integration); (4) all gates green — **verified**; (5) commit + phase move — left to the harness per pipeline rules (no `git add`/`commit` run).

No defects found; no live-infrastructure changes (repo + local dev DB only). **Next pending phase: none** — 123 is the last phase in `todo/`.
2026-09-25 05:19:18 -04:00
ducoterra a19d78d284 phase: 122_image_documents
Build and Push Containers / build-and-push-app (push) Successful in 1m57s
Build and Push Containers / build-and-push-db (push) Failing after 13s
**Phase 122 (image documents) — final verification pass: all green. No code changes were needed; defects found: none.**

**Verified (implementation already complete in working tree, reviewed end-to-end):**
- Toggle (`BOR_IMAGES`/`BOR_IMAGE_EXTENSIONS`/`BOR_IMAGE_DIR`, off by default) + `GET /api/config` `images` flag
- Ingest: bytes digest, `image_dir` persistent copy, `content = summary = vision description` (chat-model call; only text embedded), fail-soft skip + `images_failed` counter
- Serve/display: `/api/documents/{id}/image` route (404 matrix), viewer `<img>` + description, Sources 48px lazy thumbnails, chat inline source figure (alt = summary), agent `read` marker
- Prune guard: images-off syncs never prune `is_image` docs

**Test / lint / coverage (exact commands & outcomes):**
- `uv run pytest` → exit 0 (green; note: pytest 9.1.1 `-q` omits the final count line in output — exit code authoritative)
- `uv run pytest --cov=app --cov-report=term-missing` → **2715 passed, exit 0, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors, 0 warnings, 0 informations"
- `uv run pytest tests/e2e/test_image_documents.py -v --no-cov` → **4 passed, exit 0** (isolation)

**Completion criteria:** (1) images=true → described/embedded/displayed docs: ✅ (E2E + integration) · (2) images=false byte-identical + image docs survive sync: ✅ (E2E negative app + unit/integration) · (3) viewer + chat rendering with alt text; failed description skips + logs, sync completes: ✅ · (4) test/lint/coverage gates: ✅ · (5) commit + phase move: deferred to harness per this pass's rules (working tree left uncommitted).

**Notable deviation (pre-existing, documented in code):** image route uses `require_user` (phase-79 posture, same gate as the document content endpoint) rather than the phase text's "public" parenthetical — matches the endpoint it mirrors.

**Next pending phase:** `123_chat_image_questions`.
2026-09-25 01:54:23 -04:00
ducoterra 0f77e9a876 phase: 121_git_source_tokens
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Build and Push Containers / build-and-push-db (push) Failing after 14s
**Phase 121 final verification pass — all green** (all 4 tasks already in `complete/`; verified, no defects found, no changes needed)

- Verified implementation vs phase design: migration `0021` (reversible, round-tripped via `alembic downgrade base` + `upgrade head` → head `0021`), `GitSource.token` column, `normalize_credential`/`clone_url_for`/`sanitize_url`, clone callers switched (`sync.py`, `import_docs.py`), masked token fields in add form + editor, `extra="forbid"` output shapes
- Tests: `uv run pytest` → 2662 passed, 0 failed (exit 0); `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (≥90% gate)
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov` → **4 passed**

Completion criteria:
1. Private repo (UI add or pasted embedded-token URL) clones with injected token; token absent from every API response, page text, title attr, and full HTML — **PASS** (integration raw-JSON assertions + E2E `_assert_token_nowhere`)
2. Legacy embedded-token rows still clone from stored URL; output sanitized — **PASS** (`test_sync_legacy_row_clones_with_original_stored_url`, `test_get_masks_legacy_embedded_token_row`, env-fallback masking)
3. Public/local sources byte-identical — **PASS** (verbatim-URL + no-userinfo-unchanged tests)
4. pytest / coverage / ruff / pyright — **PASS** (see above)
5. Commit + phase move — harness responsibility; task files already in `complete/`, changes left in working tree (no commit made, per protocol)

Notable: no deviations; DB left at head, functional. Next pending phase: **122_image_documents** (then 123_chat_image_questions).
2026-09-24 20:51:39 -04:00
ducoterra 3a0fc3db05 phase: 120_failed_turn_retry
Build and Push Containers / build-and-push-app (push) Successful in 3m5s
Build and Push Containers / build-and-push-db (push) Failing after 2m13s
All verification complete. Final report:

**Phase 120 — Failed-turn retry: verification pass (all 3 tasks were done; final verification + 1 regression fix)**

**Verified:** `ChatMessage.failed`/`error` (≤500, `extra="forbid"` intact); `finalizeFailedTurn` funnel on the 3 failure paths (catch-else, stream-drop guard, zero-frame fallback) with `failed: true` + capped detail + `markLastRetryable`; `appendFailedNote` restore branch (Save-as-doc/Tune excluded); `showErrorBanner`/`retryLastTurn` byte-pinned untouched; only the three paths persist `failed: true` (grep + unit pin); no test asserts the old broken behavior.

**Defect found & fixed (rule 7):** a real navigate-away mid-turn let the browser's teardown fetch rejection (TypeError, not AbortError) leak into the failed funnel, persisting a phantom failed brain record — `test_sources_midstream_bug.py::test_no_orphan_brain_message_when_navigated_before_first_token` failed (2 `.msg` after reload) and violated the phase-20 navigate-away convention. Fixed: turn-scoped `leftThePage` flag (set unconditionally on `pagehide`, reset in `runTurn`) skips the funnel in the catch-else branch; pinned by new unit test `test_navigate_away_is_not_a_failed_turn`. No phase-overview/PLAN/todo/complete files touched; no commits made.

**Gates (exact):**
- `uv run pytest` → 2577 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL 4271 stmts, 99% (>90%)
- `uv run pytest tests/e2e/test_failed_turn_retry.py -v --no-cov` → 4 passed (isolated)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- Regression E2E, isolated: `test_sources_midstream_bug.py` 6/6 (was 5/6); `test_llm_retry`/`test_tool_scaffolding_guardrails`/`test_stop_generation`/`test_navbar_refresh` 17/17

**Completion criteria:** (1) network error → banner + in-bubble Retry, re-ask without re-typing ✅ (E2E A); (2) refresh restores failed bubble + working Retry, no "new chat" ✅ (E2E C); (3) stopped/successful turns byte-identical ✅ (negative E2E, stop suite, byte-identity units); (4) pytest/coverage/lint/types ✅; (5) commit + phase move — left to the harness per pass rules.

**Notable:** deviation = the regression fix above (a navigation is not a failed turn; phase-20 partial-persist convention restored). Next pending phase: `121_git_source_tokens`.
2026-09-24 18:50:36 -04:00
ducoterra 0ff1f8c4d6 chore(agent): phase roadmap from TODO.md — 4 phases (120–123)
Protocol B append: failed-turn retry (L3–4), git source tokens (L5),
image documents (L6 ingest), chat image questions (L6 chat side).
TODO.md items now live in .agents/phases/todo/ and the file is cleared.

LLM-Generated: true
2026-09-24 16:27:44 -04:00
ducoterra a5b63f83ad phase: 119_name_signal_read_chips
Build and Push Containers / build-and-push-app (push) Successful in 2m1s
Build and Push Containers / build-and-push-db (push) Successful in 18s
All verification complete. Final report:

**Phase 119 final verification pass — all criteria verified, one stale pin fixed.**
- Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry.
- Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged.
- New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2.
- Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed).
- Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met).
- Next pending phase: **none** — `todo/` holds only phase 119.
2026-09-16 15:50:48 -04:00
ducoterra 795fb56425 add phase 119
Build and Push Containers / build-and-push-app (push) Successful in 15s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-16 10:05:52 -04:00
ducoterra 9820c361b0 phase: 118_summary_seed_context
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 14s
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.

- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
2026-09-16 06:57:49 -04:00
ducoterra 21aad84a6d chore(agent): phase roadmap from TODO.md — phase 118 summary seed context (top-5 summary seeding, all-doc summaries)
Build and Push Containers / build-and-push-app (push) Successful in 17s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-15 22:43:01 -04:00
ducoterra c851d1a1c5 phase: 117_tool_call_compact
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification complete — every gate green. Final report:

**Phase 117 final verification pass — all criteria verified** (tasks 01–04 were already in `complete/`; I re-verified the full gate myself, no code changes needed)
- **Verified:** `appendToolLine` wraps lines in `details.tool-calls-disclosure` (open live, `closeToolCalls` at delta/done/stop + restore-closed, count via textContent); `addToolLines` parity (closed); `.tool-call` deboxed / `code` dechipped; new disclosure/summary CSS; only the one pinned CSS assert updated
- `uv run pytest` → **2466 passed**
- `uv run pytest --cov=app --cov-report=term-missing` → **99%** on `app/` (>90% floor)
- E2E in isolation: `test_tool_call_compact.py` **4 passed** (fold/expand/deboxed-flow/restored-folded); `test_agent_document_tools.py` **4 passed**; `test_big_read_progress.py` **4 passed** (live disclosure open + `.tool-elapsed` visible); `test_thinking_display.py` **5 passed**; `test_smoke.py` **3 passed**
- `uv run ruff check . && uv run pyright` → clean (pyright 0 errors, 0 warnings)
- Completion criteria: pins 1–4 ✅ (E2E above); live "calling tool" state byte-identical ✅ (`test_frontend_tool_states.py` 10 passed incl. `test_calling_tool_label_strings`); scope ✅ — `git diff` limited to the 3 frontend assets + 2 unit tests + E2E story + phase files, **0 changes in `app/`**
- Notable: `TODO.md` carries a **pre-existing uncommitted owner edit** (new retrieval-context idea) already in the tree before this pass — left untouched, will ride along in the harness commit; pre-existing committed quirk: the brain-wave commit's styles.css comment mislabels itself "Phase 117" (cosmetic, out of this diff)
- No commit made (harness commits + moves the phase); **next pending phase: none** — `todo/` holds only phase 117
2026-09-15 18:26:38 -04:00
ducoterra 2ac3fc89c2 feat(ui): swap the in-turn loader for a brain-wave sweep left of the send button
Build and Push Containers / build-and-push-app (push) Successful in 2m11s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Phase 117 (owner request, live-mockup-confirmed): the phase-109 3-dot
cue becomes a compact ECG trace (49px, P/QRS/T) with a brand sweep
traveling the path (bwdraw, 42/140 dash segment, 0.9s loop), the
loader repositioned left of the button so its appearance never shifts
it. setUiState stays the sole owner of the loader's hidden attribute;
the reduced-motion variant stills the sweep. Unit + lifecycle-E2E pins
updated for the new contract.
2026-09-15 11:55:59 -04:00
ducoterra adf31a4a35 phase: 116_modal_scrollbar_theme
Build and Push Containers / build-and-push-app (push) Successful in 2m1s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification complete — every gate re-run independently and green. Final report:

**Phase 116 final verification pass — all green (work was already in the tree; verified, nothing to fix)**

- **Verified** `styles.css`: `--scrollbar-thumb`/`--scrollbar-track` tokens in `:root` (color-mix derivations — follow the saved theme, not in BUILTIN_COLORS; contrast 3.73:1 ≥ 3:1, recomputed independently) + scoped rules `.doc-modal .doc-raw` (overflow-x already owned by the element)
- **Verified** E2E test 8 `test_raw_code_block_themed_scrollbar`: overflow, scroll action, computed-style theme assertions, screenshot
- **Screenshots visually confirmed**: headed `doc_modal_scrollbar_themed_1280.png` shows muted rounded thumb on dark track, no native light-gray bar; headless suite shot records scrolled state (headless Chromium paints no bar UI — documented in test docstring)

**Test / lint / coverage (re-run by me):**
- `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` → **8 passed** (isolation)
- `uv run pytest --cov=app --cov-report=term` → **2457 passed, app/ 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_dark_tech_theme.py -v --no-cov` → **6 passed** (no token collision)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:**
1. Modal scrolls horizontally, scrollbar themed, screenshot shows no native bar — ✅
2. No global scrollbar change (all new selectors under `.doc-modal`; grep-confirmed), regressions green — ✅
3. pytest green, coverage >90%, e2e isolated green, ruff+pyright clean — ✅
4. `--no-gpg-sign` commit + phase move — harness job (left uncommitted in working tree, per rules)

**No defects found; no deviations. Next pending phase:** none — `todo/` contains only phase 116.
2026-09-15 06:11:38 -04:00
ducoterra 990c8adf13 phase: 115_doc_draft_discard
Build and Push Containers / build-and-push-app (push) Successful in 2m12s
Build and Push Containers / build-and-push-db (push) Successful in 14s
All green — this was the final verification pass; everything from the four completed tasks was already in the working tree and verified.

**Phase 115 — Doc drafts: Discard + DELETE route + title fix — verification report**

- Verified all 4 task deliverables present: DELETE route (`app/api/doc_drafts.py`), Discard UI (`doc-edit.html` + `doc-edit.js` + `.discard-draft` CSS), title fix (`defaultDocTitle(wrap)` pairing + `saveAsDoc` call site), and all test pins (integration, frontend unit, E2E). No code changes needed.
- **Completion criteria:**
  1. ✅ Orphaned draft discardable from edit screen; row gone — `test_delete_removes_row_and_invalidates_token` (204 → GET 404), unknown-token 404, admin-gate 403 on all routes, E2E `test_discard_draft_from_edit_screen` all pass.
  2. ✅ Title after retry redo = redone answer's own question — E2E `test_save_title_is_the_redo_question_after_retry` passes.
  3. ✅ Push flow byte-identical — `git diff` shows only the new DELETE route + module docstring; all 7 existing push tests green.
  4. ✅ `uv run pytest --cov=app` → **2457 passed**, app coverage **99%** (>90%); `uv run pytest tests/e2e/test_save_doc_session.py -v --no-cov` → **4 passed**; `uv run ruff check .` → clean; `uv run pyright` → 0 errors.
  5. ⏳ Commit + phase-dir move left to the harness (per executor rules, no `git` run; all changes left in the working tree).
- No defects found; no deviations.
- Next pending phase: none in `todo/` other than this one (`115_doc_draft_discard` is the last).
2026-09-15 05:35:05 -04:00
ducoterra 3846f26a58 phase: 114_embed_question_length
Build and Push Containers / build-and-push-app (push) Successful in 2m6s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification passes complete — the phase was already fully implemented in the working tree, and every gate is green. No defects found; no code changes were needed.

**Final verification pass — Phase 114 (embed question length):**
- Verified truncation: `chat.py:459` embeds `request.message[:settings.embed_question_max_chars]` (default 1200, `BOR_EMBED_QUESTION_MAX_CHARS`, `>0` validator); full question still reaches the LLM prompt/log.
- Verified error mapping: `EmbeddingInputTooLargeError(EmbeddingError)` (byte-identical message) caught **before** `EmbeddingError` → no retry, terminal frame `detail="Question too long — trim it and re-ask."` + reachability-fine hint; `ChatErrorEvent.hint` additive.
- Verified frontend chain: frame `hint` → `err.hint` → `setUiState(error, …, {hint})` → `showErrorBanner(…, {retryable: true})` — hint replaces default `ERROR_HINT`, phase-111 `#banner-retry` button revealed. 4,000-char clamp untouched (A2).
- `uv run pytest tests/unit/test_embed_question_length.py -v --no-cov` → 21 passed
- `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov` (isolation, DB up) → 1 passed (4,000-char question → done, no banner)
- Regression: `test_llm_retry.py` 4 passed · `test_oneshot_llm_retry.py` 2 passed · `test_chip_sizing_question_cap.py` 6 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2444 passed, TOTAL **99%** (>90% gate)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:** (1) 4,000-char question embeds prefix + full prompt ✅ · (2) too-large → accurate frame + hint + Retry button ✅ · (3) reachability failure byte-identical (retries + old copy) ✅ · (4) all gates green ✅ · (5) commit/phase-move → left to the harness per instructions (no `git add`/`commit` run).
**Deviations:** none. **Next pending phase:** `115_doc_draft_discard`.
2026-09-15 04:16:55 -04:00
ducoterra 97d663d16d phase: 113_source_chip_quality
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 15s
All gates green — no defects found; this pass was verification only.

**Phase 113 final verification pass — report**

- Verified (no code changes needed): `select_documents_tiered` cited/related tiering + `select_documents` wrapper, `TurnPlan.related_docs`, `ChatDoneEvent.related` (additive, old payloads parse), `appendRelated` UI row (`.related-doc`, never `.source-chip`), done-frame + restore-path wiring, two settings with validators, `.env.example` entries
- `uv run pytest --cov=app --cov-report=term-missing` → 2422 passed, app/ coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (isolated) → 2 passed
- Regression E2E `test_retrieval_quality.py` + `test_honest_deflection.py` + `test_chat_rag.py` + `test_sources_midstream_bug.py` → 17 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors); `bash .agents/validate.sh` → "validation OK"

Completion criteria:
1. Single-doc question → exactly one `.source-chip` (E2E): ✅ passed
2. Weak 2nd doc only in de-emphasized related row, never `.source-chip` (unit + E2E): ✅ passed
3. Deflected turn → zero citation chips, weak hits in related row: ✅ passed
4. Full suite green, coverage >90%, isolated E2E green, lint/types clean: ✅ passed
5. `--no-gpg-sign` commit + phase dir move: left to harness per pass rules (task files already in `complete/`)

No deviations. Next pending phase: `114_embed_question_length`.
2026-09-15 03:11:05 -04:00
ducoterra 1374faf136 phase: 112_honesty_gate_weak_hits
Build and Push Containers / build-and-push-app (push) Successful in 2m15s
Build and Push Containers / build-and-push-db (push) Successful in 14s
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):**

- Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`.
- Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length).
- Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented.
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW).
- E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**.
- Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors.

**Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree).

**Deviations:** none. Next pending phase: `113_source_chip_quality`.
2026-09-15 00:37:38 -04:00
ducoterra 2683128876 phase: 111_chat_banner_retry
Build and Push Containers / build-and-push-app (push) Successful in 2m24s
Build and Push Containers / build-and-push-db (push) Successful in 14s
## Phase 111 Completion Report

**Implemented/Verified:**
- `#kb-banner` contains a `<button type="button" class="banner-retry" id="banner-retry">` (hidden by default, Retry label + SVG)
- `showErrorBanner(detail, opts)` reveals the button only when `opts.retryable` is true AND `lastBrainWrap` exists
- Turn-error path passes `{ retryable: true }`; all non-turn callers (share, save-doc, stale) remain text-only
- `clearErrorBanner()` re-hides the button
- `ERROR_HINT` changed from "Try again — …" to "If this persists, check the LLM is reachable."
- `.banner-retry` CSS styled as a pill (matching `.stale-regenerate` family)
- 12 source-assertion unit tests in `tests/unit/test_frontend_banner_retry.py`

**Test / Lint / Coverage:**
- `uv run pytest tests/unit/test_frontend_banner_retry.py -v --no-cov` → 12 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2362 passed, 99% coverage
- `uv run ruff check .` → All checks passed
- `uv run pyright` → 0 errors
- `tests/e2e/test_llm_retry.py` → 4 passed (in isolation)
- `tests/e2e/test_smoke.py` → 3 passed (in isolation)

**Completion Criteria:**
- ✅ Retry button visible after failed chat turn, re-runs last question
- ✅ Non-turn callers show text-only banner (no button)
- ✅ pytest green, coverage >90%, ruff + pyright clean
- ✅ Phase dir to be moved by pipeline gate

**Next pending phase:** `112_honesty_gate_weak_hits`
2026-09-14 23:07:56 -04:00
ducoterra f37c517590 chore(agent): phase roadmap from TODO.md — 6 phases (111–116): banner retry, honesty gate, chip quality, embed length, draft discard, modal scrollbar
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 13s
2026-09-14 22:07:32 -04:00
ducoterra 2b75f3cc85 docs(todo): log live testing findings L1-L7 (brain.reeseapps.com session)
Interactive browser test session (15 tests) against the deployed
instance. Findings: L1 banner 'Try again' is plain text, L2 honesty
gate eagerness + stochastic deflection compliance, L3 unstyled modal
scrollbar, L5 recurring weak-hit 2nd source chip, L6 4000-char
question clamp exceeds the embed input cap (500 surfaced as
'unreachable'), L7 save-as-doc drafts have no discard path. Each
entry carries observed evidence, suggested fixes and acceptance.
2026-09-14 21:31:08 -04:00
422 changed files with 34597 additions and 1924 deletions
+1 -1
View File
@@ -46,7 +46,7 @@ accounts (one admin + hand-out tokens is the model).
| A5 | LLM backend | OpenAI-compatible self-hosted endpoint `https://aipi.reeseapps.com/v1` via the `openai` async client: **`turbo`** (chat, streams `reasoning_content` thinking), **`embed`** (embeddings), **`lite`** (one-shot: document summaries, KB overview, folder summaries) | | A5 | LLM backend | OpenAI-compatible self-hosted endpoint `https://aipi.reeseapps.com/v1` via the `openai` async client: **`turbo`** (chat, streams `reasoning_content` thinking), **`embed`** (embeddings), **`lite`** (one-shot: document summaries, KB overview, folder summaries) |
| A6 | Embedding dim | **768** (verified against the live endpoint); `chunks.embedding` is fixed at table creation — a dim mismatch must **fail loudly**, never silently re-embed | | A6 | Embedding dim | **768** (verified against the live endpoint); `chunks.embedding` is fixed at table creation — a dim mismatch must **fail loudly**, never silently re-embed |
| A7 | Retrieval→context | **Hybrid:** cosine top-100 ∪ Postgres FTS top-30 (OR tsquery, `ts_rank`), fused with **RRF (k=60)** → parent docs ranked by best fused chunk → **full text of top-N=2 documents, never truncated** on the retrieval path (owner 2026-08-24: "this should never happen"; the agent `read`-tool cap is a separate owner-permitted path) | | A7 | Retrieval→context | **Hybrid:** cosine top-100 ∪ Postgres FTS top-30 (OR tsquery, `ts_rank`), fused with **RRF (k=60)** → parent docs ranked by best fused chunk → **full text of top-N=2 documents, never truncated** on the retrieval path (owner 2026-08-24: "this should never happen"; the agent `read`-tool cap is a separate owner-permitted path) |
| A8 | Honesty gate | Deflect (LOW mode) **only when** best cosine < `BOR_RELEVANCE_THRESHOLD` (0.62) **and** zero FTS hits; LOW prompt carries weak-hit *titles only* + the `DEFLECT_MODE` marker (the E2E mock keys on its presence) + the plain-text no-tools line | | A8 | Honesty gate | Deflect (LOW mode) **only when** best cosine < `BOR_RELEVANCE_THRESHOLD` (0.62) **and** (zero FTS hits **or** best cosine < `BOR_LEXICAL_SUPPORT_FLOOR` (0.35)); an FTS hit flips HIGH only when `best_cosine >= lexical_support_floor` — the vector signal must corroborate the lexical match (A8 revised 2026-09-14, owner-confirmed, TODO L2a: lexical-only hits without vector support deflect). LOW prompt carries weak-hit *titles only* + the `DEFLECT_MODE` marker (the E2E mock keys on its presence) + the plain-text no-tools line |
| A9 | Content scope | Default `md, markdown, txt, yaml, yml, json, py` + Podman quadlet family + `j2`; `BOR_IMPORT_EXTENSIONS` may name **any** well-formed extension or narrow the set; hidden (dot) paths + exclusion list + per-source `ignore_paths` (raw prefixes, no globs) always apply | | A9 | Content scope | Default `md, markdown, txt, yaml, yml, json, py` + Podman quadlet family + `j2`; `BOR_IMPORT_EXTENSIONS` may name **any** well-formed extension or narrow the set; hidden (dot) paths + exclusion list + per-source `ignore_paths` (raw prefixes, no globs) always apply |
| A10 | State & auth | `POST /api/chat` is **stateless** (client-provided `history` only, budget-trimmed — nothing stored per conversation). Auth = single admin, signed `bor_session` cookie (Starlette `SessionMiddleware` + itsdangerous; no server-side session store); admin-issued SHA-256-hashed access tokens are the only other identity; **only** shared chats stay anonymous. Fail-loud at boot while `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` are empty | | A10 | State & auth | `POST /api/chat` is **stateless** (client-provided `history` only, budget-trimmed — nothing stored per conversation). Auth = single admin, signed `bor_session` cookie (Starlette `SessionMiddleware` + itsdangerous; no server-side session store); admin-issued SHA-256-hashed access tokens are the only other identity; **only** shared chats stay anonymous. Fail-loud at boot while `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` are empty |
| A11 | Frontend | Vanilla HTML/CSS/JS in git, **no CDN** — everything served by FastAPI `StaticFiles`; system font stack; the navbar views are views of ONE shell document (`frontend/index.html` + `router.js` deep-links) | | A11 | Frontend | Vanilla HTML/CSS/JS in git, **no CDN** — everything served by FastAPI `StaticFiles`; system font stack; the navbar views are views of ONE shell document (`frontend/index.html` + `router.js` deep-links) |
@@ -0,0 +1,42 @@
# Phase 111 — Chat error banner: a real Retry button (TODO L1)
**Source:** `TODO.md` L3–21 — "L1 — Chat error banner: 'Try again' is plain text, not a button (2026-09-15, brain-of-reese interactive test)"
**Story:** n/a (interactive-test follow-up fix; extends the phase-06 loading-feedback and phase-49/53 retry assets).
**Context:** `frontend/index.html:112` renders `#kb-banner` (chat page only — the Sources/document pages do not render it) with `#kb-banner-text`; `frontend/assets/app.js` — `showErrorBanner(detail)` (L2084) writes `${detail} ${ERROR_HINT}` as **plain text**; `ERROR_HINT` (L362) begins "Try again — …" so "Try again" reads as a clickable action but is not. `retryLastTurn(wrap)` (L2131) re-asks the last question in place (phase 49); `#stale-regenerate` (index.html:134, handler at app.js ~L1898) is the existing banner-button → `retryLastTurn` pattern.
## Objective
Give the chat-view error banner a real Retry control after a failed/dropped turn: the banner shows a visible Retry button that re-runs the last question without re-typing (reusing the phase-49 redo-in-place and the stale-banner button pattern). The banner text stops mimicking a button, and every existing `showErrorBanner` caller (share, save-doc, stale chat) keeps working text-only — the Retry button appears only on failed chat turns.
## Dependencies
- `110_fix_sse_db_pool_exhaustion` (complete) — pipeline predecessor (execution order) only; no code dependency (this phase touches `frontend/index.html`, `frontend/assets/app.js`, `frontend/assets/styles.css`, and frontend unit tests).
## Design (shared by all tasks — the executor reads this, not the chat)
- **Banner button (task 01):** add `<button type="button" class="banner-retry" id="banner-retry" hidden>` inside `#kb-banner` (after `#kb-banner-text`), mirroring the `#stale-regenerate` markup (same refresh SVG + visible "Retry" label). Hidden by default; `showErrorBanner(detail, opts)` gains an optional second arg — when the caller flags the error as a **failed chat turn** (the UI state-machine path at app.js:1281 `if (state === UI_STATE.error) showErrorBanner(errorDetail)`), the button is revealed and wired to `retryLastTurn(lastBrainWrap)` — the same last-brain-bubble targeting the `#stale-regenerate` handler uses. No retryable brain bubble → no button.
- **Copy:** `ERROR_HINT` becomes "If this persists, check the LLM is reachable." — the "Try again —" prefix moves to the button (the text must no longer read as a fake control).
- **Non-turn callers** (share failures L1857/L1870/L1892, save-doc L735/L742, stale L1620/L1950/L1962, …) pass no opts → text-only banner, no button — no behavior change for them.
- **CSS:** `.banner-retry` in `styles.css` reuses the `.stale-regenerate` pill look (same component family); the banner keeps `role="alert"`.
- **NOT touched:** the per-answer Retry pill (phase 49), the stale banner, `retryLastTurn` itself, the server, and the RAG/document views (no `#kb-banner` there — no split needed, ASSUMPTION in task 01).
## Tasks
1. `01_banner_retry_button.md` — banner Retry button markup + handler + hint-copy fix.
2. `02_banner_retry_tests.md` — frontend unit tests for the button's presence/handler + stale hint-copy assertions updated.
## Testing & Quality
- Unit: `tests/unit/test_frontend_banner_retry.py` (new, task 02) — house-style source assertions: `#banner-retry` exists in the `#kb-banner` markup (hidden by default, `type="button"`); `showErrorBanner` wires the click → `retryLastTurn`; the button is revealed only on the turn-error path; `clearErrorBanner` re-hides it; `ERROR_HINT` no longer starts with "Try again".
- E2E: no new file — the turn-error path is exercised by the existing `tests/e2e/test_llm_retry.py` and `tests/e2e/test_smoke.py` suites, which must stay green (no banner behavior change for non-turn callers).
- Coverage: **>90%** on `app/` (validate.sh gate; the frontend JS is pinned by the source-assertion unit tests — no app/ code changes in this phase).
## Completion Criteria
- [ ] After a dropped/failed chat turn, `#kb-banner` shows a visible Retry button; clicking it re-runs the last question without re-typing.
- [ ] Share/save-doc/stale-chat errors show a text-only banner (no button) — unchanged.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **A1 — Retry = re-ask the last question in place via the existing `retryLastTurn` (owner-confirmed 2026-09-14, roadmap confirmation).** No new retry mechanism; the phase-49 redo-in-place is reused.
- **A2 — the button is offered only on the UI state-machine's turn-error path (a dropped/failed chat turn); all other banner callers stay text-only (owner-confirmed 2026-09-14).** Matches L1's acceptance ("after a dropped/failed turn …").
## Commit
```bash
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(ui): give the chat error banner a real Retry button that re-asks the last question"
```
@@ -0,0 +1,34 @@
# Task 01 — Banner Retry button: markup, handler, hint-copy fix
**Phase:** `111_chat_banner_retry` · **Source:** `TODO.md:3–18` — "L1 — Chat error banner: 'Try again' is plain text, not a button … `showErrorBanner()` (`frontend/assets/app.js`, `ERROR_HINT` ~line 362) renders `${detail} ${ERROR_HINT}` as **plain text** into `#kb-banner-text` … Suggested fix: give the chat-view error banner a real Retry control that re-asks the last question (the stale-chat banner already has the pattern: `#stale-regenerate` → `retryLastTurn`; the phase-49 Retry pill asset exists). Keep `#kb-banner` dual-use working for the RAG view, or split the two banners if the RAG view's banner has different recovery semantics."
## Objective
The chat error banner gets a real Retry button (turn failures only) that re-asks the last question through the existing `retryLastTurn` redo-in-place; the hint copy stops mimicking the button.
## Work
1. `frontend/index.html` — inside `#kb-banner` (L112), add after `<span id="kb-banner-text">`:
```html
<button type="button" class="banner-retry" id="banner-retry" hidden>
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M21 12a9 9 0 1 1-9-9c2.52 0 4.93 1 6.74 2.74L21 8"/><path d="M21 3v5h-5"/></svg>
<span>Retry</span>
</button>
```
(the same refresh SVG the `#stale-regenerate` button at L134 uses). Add a comment block: revealed only for failed chat turns (task 01 of this phase); hidden for every other banner caller.
2. `frontend/assets/app.js`:
- `ERROR_HINT` (L362): change to `"If this persists, check the LLM is reachable."` (the action moves to the button).
- `showErrorBanner(detail, opts = {})` (L2084): keep the single-arg behavior byte-identical; when `opts.retryable` is true AND a retryable last brain bubble exists (the same lastBrainWrap lookup the `#stale-regenerate` handler at ~L1898 uses), unhide `#banner-retry` and bind its click **once** to `() => retryLastTurn(lastBrainWrap)`; re-binding on every reveal must be guarded (one listener per button lifetime). `clearErrorBanner()` re-hides the button.
- The turn-error path (L1281 `if (state === UI_STATE.error) showErrorBanner(errorDetail)`): pass `{ retryable: true }`. Every other caller (L735, L742, L1620, L1857, L1870, L1892, L1950, L1962, …) is left unchanged.
- If `retryLastTurn` would no-op (no retryable bubble), do not reveal the button — reveal only when a bubble exists.
3. `frontend/assets/styles.css` — `.banner-retry`: same pill treatment as `.stale-regenerate` (color, border, hover, `focus-visible` ring per the theme), laid out inline after the banner text (the `.kb-banner` flex row + gap already handles spacing).
4. ASSUMPTION: no banner split — `#kb-banner` exists only in `frontend/index.html` (the chat page); `document.html`/the Sources pages do not render it, so "keep dual-use" is trivially satisfied and the RAG view is untouched.
5. ASSUMPTION: the button is revealed only on the UI state-machine's turn-error path (a dropped/failed chat turn) — non-turn errors (share, save-doc, stale chat) stay text-only (locked A2).
## Testing & Quality
- Unit: `tests/unit/test_frontend_banner_retry.py` (added by task 02 — this task ships the code, task 02 ships the pin).
- Coverage: n/a (frontend) — the validate.sh `app/` gate must stay green.
## Completion Criteria
- [ ] `#kb-banner` contains `#banner-retry` (hidden by default); a failed chat turn reveals it; clicking re-asks the last question without re-typing.
- [ ] `ERROR_HINT` no longer contains "Try again".
- [ ] No non-turn call site passes `retryable` (grep the call sites).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,23 @@
# Task 02 — Unit tests for the banner Retry button
**Phase:** `111_chat_banner_retry` · **Source:** `TODO.md:19–21` — "Acceptance: after a dropped/failed turn, the banner shows a visible Retry button that re-runs the last question without re-typing; unit test for the banner's button presence/handler in the frontend test suite."
## Objective
Pin the banner contract in the frontend unit suite: the button's presence, its handler, its reveal condition, and the new hint copy.
## Work
1. `tests/unit/test_frontend_banner_retry.py` (new) — house-style source assertions (pattern: `tests/unit/test_frontend_feedback.py`):
- `frontend/index.html`: `#banner-retry` exists inside the `#kb-banner` block, `hidden` by default, `type="button"`, with a visible "Retry" label.
- `frontend/assets/app.js`: `showErrorBanner` binds the click handler to `retryLastTurn`; the `UI_STATE.error` turn path passes the retryable flag; `clearErrorBanner` re-hides the button.
- `ERROR_HINT` does not start with "Try again".
2. Grep the whole `tests/` tree for the old hint copy (`Try again — if this persists`) and update any stale assertion (loading-feedback and banner-related frontend tests).
3. Run the full unit + integration suite — no regressions (in particular `tests/unit/test_frontend_*.py`).
## Testing & Quality
- Unit: the new file above (≥4 assertions across markup/handler/reveal/copy).
- Coverage: **>90%** on `app/` unchanged (no app/ code touched by this phase).
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_frontend_banner_retry.py -v` green.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] No test anywhere asserts the old "Try again — …" hint.
@@ -0,0 +1,46 @@
# Phase 112 — Honesty gate: FTS hits need cosine corroboration (TODO L2a/b + README nit)
**Source:** `TODO.md` L23–86 — "L2 — Deflection test: non-KB question got answered parametrically + unrelated source chips (2026-09-15, brain-of-reese interactive test)" — part a (L36–44), part b (L46–55) + the stochastic follow-up (L64–75), the README nit (L77–81), acceptance (L83–86). Part c (chips on non-grounded answers, L57–62) is delivered by phase 113.
**Story:** the completed story 04 (`04_story_honest_deflection`) — this phase fixes the A8 gate eagerness the live deflection test exposed.
**Context:** `app/api/chat.py::plan_turn` (L272–277): HIGH when `best_cosine >= settings.relevance_threshold (0.62) or fts_hits > 0` — a single weak FTS token hit (suspected: the token "capital" inside a quest file) promoted a non-KB question into grounded mode and injected two irrelevant docs into the HIGH prompt, which then tempted the model into a parametric answer. The follow-up one-tap re-run of the identical question produced a **clean, textbook deflection** with the same docs injected → HONESTY GATE compliance is stochastic across runs; the deterministic lever is the gate (don't inject irrelevant docs), not prompt copy alone.
## Objective
Make the A8 honesty gate deterministic against weak lexical hits: an FTS hit flips the turn to HIGH (grounded) only when the vector signal corroborates it (best cosine clears a new `lexical_support_floor`); below the floor the turn stays LOW (deflected) even with FTS matches. The owner-confirmed locked-prompt contract decision (treat the stochastic disclosed-general-knowledge behavior as acceptable — documented, prompt text byte-identical) and the stale README deflection copy are fixed in the same phase.
## Dependencies
- `111_chat_banner_retry` (todo) — pipeline predecessor (execution order) only; no code dependency.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Gate rule (task 01):** HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`). LOW otherwise — including the fts>0 / cosine<floor case (the Mongolia case). `lexical_support_floor` is a new setting (default 0.35, env `BOR_LEXICAL_SUPPORT_FLOOR`), validated `0 <= floor <= relevance_threshold`. Rationale: a real lexical match on a genuinely similar doc (cosine ≥ floor) still grounds; a single weak token match with vector-unsupported docs no longer promotes. `plan_turn`'s docstring (the A8 bullets) and the A8 entry in `.agents/PLAN.md` are updated with the revision note (house precedent: "A8 revised 2026-08-21" — an owner-confirmed change to a LOCKED decision is recorded in the plan, not silently deviated from).
- **No schema/API change:** `query_log.top_score` / `fts_hits` are recorded exactly as today (observability unchanged); `TurnPlan` shape unchanged; the LOW branch (deflect prompt, weak-hit titles, derived suggestions) unchanged.
- **Prompt contract (task 03):** owner decision (iii) — the model's stochastic disclosed-general-knowledge answer (when misleading docs are injected) is documented as acceptable; the `app/rag/prompts.py` module docstring (the house location for locked-prompt revision history, e.g. the 2026-08-22 note) gains a dated entry; the prompt strings stay byte-identical (LOCKED verbatim); the README deflection section notes the behavior.
- **README nit (task 04):** the quoted deflection opening *"I haven't done anything like that"* was removed in the 2026-08-22 locked-prompt revision — the two README spots (L11, L575) are updated to describe the current behavior (admit no notes + 2–3 concrete alternative questions).
## Tasks
1. `01_gate_meaningful_fts.md` — the `plan_turn` gate fix + `lexical_support_floor` setting + the A8 plan revision note.
2. `02_gate_tests.md` — unit pins for the HIGH/LOW quadrants on weak single-token FTS hits + the E2E deflection check.
3. `03_prompt_contract_documentation.md` — document the owner decision (iii) (prompts.py docstring + README); prompt text unchanged.
4. `04_readme_deflection_copy.md` — README stale deflection copy (L11, L575) updated.
## Testing & Quality
- Unit: `tests/unit/test_chat_gate.py` (existing — extend) + a new quadrant file (task 02): cosine ≥ threshold → HIGH regardless of FTS; fts>0 + cosine ≥ floor → HIGH; **fts>0 + cosine < floor → LOW** (the new behavior, the Mongolia regression pin); no hits → LOW.
- E2E: `tests/e2e/test_honest_deflection.py` (existing — extend, task 02): a known-out-of-KB question (mock LLM, so the test pins the gate not the model) → deflected, no citation chips, 2–3 alternative questions. Run in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov`.
- Regression: `tests/unit/test_chat_gate.py`, `tests/e2e/test_chat_rag.py`, `tests/e2e/test_retrieval_quality.py` stay green.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] A weak single-token FTS hit with vector-unsupported docs (cosine < floor) → LOW/deflected (unit-pinned).
- [ ] A known-out-of-KB question produces no false citations and 2–3 concrete alternative questions (E2E).
- [ ] `app/rag/prompts.py` prompt strings byte-identical to pre-phase (test-pinned); README deflection copy matches current behavior.
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — lever: cosine corroboration; `lexical_support_floor` default 0.35, env-tunable via `BOR_LEXICAL_SUPPORT_FLOOR` (owner-confirmed 2026-09-14, roadmap confirmation).** Of the TODO's three options (rank threshold / stopword-short-token exclusion / cosine corroboration), cosine corroboration is the deterministic one; the tests pin the decision logic, not the default value.
- **A2 — locked-prompt contract: option (iii) — treat the stochastic disclosed-general-knowledge behavior as acceptable and document it (owner-confirmed 2026-09-14, roadmap confirmation).** The prompt text is unchanged (LOCKED verbatim); the deterministic protection is the gate (A1). Options (i) tighten copy / (ii) amend the prompt via the plan remain open to a future owner decision.
- **A3 — the A8 entry in `.agents/PLAN.md` is amended in this phase with a dated revision note** (owner-confirmed 2026-09-14) — house precedent for recording owner-confirmed LOCKED-decision changes.
## Commit
```bash
git add app/ tests/ .agents/ README.md && git commit --no-gpg-sign -m "fix(rag): require cosine-corroborated FTS hits before the honesty gate flips HIGH — document the disclosed-answer behavior, refresh README deflection copy"
```
@@ -0,0 +1,30 @@
# Task 01 — Gate: FTS hits must be cosine-corroborated
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:36–44` — "a) **Honesty gate too eager** (`app/api/chat.py` L272–277, A8 revised 2026-08-21): HIGH/grounded when `best_cosine >= threshold` OR `fts_hits > 0`. A single weak FTS token hit (suspected: the token 'capital' inside a quest file) promotes a non-KB question into grounded mode and injects two irrelevant top-docs into the HIGH prompt — which then tempts the model into a parametric answer instead of deflection. Consider: require FTS hits to be *meaningful* (e.g. rank threshold, stopword/short-token exclusion, or cosine corroboration) before flipping to HIGH; or run the LOW prompt when top-docs score below a usefulness bar." (+ the follow-up, L64–75: "the deterministic lever is the gate (don't inject irrelevant docs — part a), not prompt copy alone")
## Objective
An FTS hit flips `plan_turn` to HIGH only when the best cosine clears the new `lexical_support_floor`; a weak single-token hit with vector-unsupported docs stays LOW (deflected).
## Work
1. `app/config.py` — add `lexical_support_floor: float = Field(default=0.35)` (env `BOR_LEXICAL_SUPPORT_FLOOR`) next to `relevance_threshold` (L119), with validation `0 <= lexical_support_floor <= relevance_threshold` (mirror the file's existing validator style); document it in `.env.example`.
2. `app/api/chat.py::plan_turn` — replace the gate line (`if best_cosine >= settings.relevance_threshold or fts_hits > 0:`):
```python
lexical_supported = fts_hits > 0 and best_cosine >= settings.lexical_support_floor
if best_cosine >= settings.relevance_threshold or lexical_supported:
...
```
Update the `plan_turn` docstring's gate section (the A8 bullets): HIGH when `best_cosine >= threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); LOW otherwise — a lexical-only hit without vector support deflects (A8 revised 2026-09-14).
3. `.agents/PLAN.md` — amend the A8 locked-decision entry with a dated revision note (house precedent: "A8 revised 2026-08-21"): "A8 revised 2026-09-14 (owner-confirmed, TODO L2a): an FTS hit flips HIGH only when `best_cosine >= lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`); lexical-only hits without vector support deflect."
4. `app/rag/prompts.py` — prompt strings NOT touched (LOCKED verbatim).
5. ASSUMPTION: the chosen lever is cosine corroboration (one of the TODO's three listed options); floor default 0.35 (≈ half the 0.62 threshold) is tunable via env against the live KB — the unit/E2E tests pin the decision logic, not the default value (locked A1).
## Testing & Quality
- Unit: covered by task 02 (`tests/unit/test_chat_gate.py` extension) — this task's code must keep it green.
- Coverage: **>90%** on `app/` including the new setting and the modified gate line.
## Completion Criteria
- [ ] `plan_turn`: fts_hits>0 + best_cosine < floor → LOW (`TurnPlan.deflected` True, LOW prompt, suggestions derived from weak-hit titles).
- [ ] fts_hits>0 + floor <= best_cosine < threshold → HIGH (the new "corroborated lexical" path).
- [ ] best_cosine >= threshold → HIGH regardless of FTS (unchanged); no hits → LOW (unchanged).
- [ ] `.env.example` documents `BOR_LEXICAL_SUPPORT_FLOOR`; the config validation rejects floor > threshold.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,27 @@
# Task 02 — Pin the gate's HIGH/LOW decision on weak FTS hits
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:83–86` — "Acceptance: a known-out-of-KB question (LLM-known, e.g. capitals, sports results) produces no false citations, follows the HONESTY GATE (or the amended contract), and unit/E2E tests pin the gate's HIGH/LOW decision on a weak single-token FTS hit."
## Objective
Unit tests pin all four gate quadrants — especially the new fts>0 / cosine<floor → LOW quadrant (the Mongolia/"capital" regression); the E2E deflection story asserts a known-out-of-KB question deflects with no false citations and 2–3 alternatives.
## Work
1. `tests/unit/test_chat_gate.py` (existing — extend; reuse its chunk-building helpers) — the quadrants:
- cosine ≥ threshold, fts=0 → HIGH (unchanged).
- fts>0, cosine ≥ floor (e.g. 0.50 with default settings) → HIGH (corroborated lexical — the new path).
- **fts>0, cosine < floor** (e.g. one `fts_hit=True` chunk with cosine 0.10 — the "capital" case) → LOW: `deflected=True`, LOW prompt, `suggestions` non-empty, the weak docs do not enter a HIGH prompt.
- no chunks → LOW (unchanged).
- boundary: cosine exactly at the floor → HIGH (`>=`, mirroring the threshold's convention); config with floor > threshold → validation error.
2. `tests/e2e/test_honest_deflection.py` (existing — extend): a test asking a known-out-of-KB question (e.g. "What is the capital of Mongolia?" — LLM-known, absent from the fixture KB; the mock LLM's deflection path keeps the test deterministic on the gate, not on model compliance) → the done frame is `deflected: true`, `sources` is empty (no false citations), `suggestions` has 2–3 items.
3. Run the E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` (DB up).
4. Regression: `tests/unit/test_chat_gate.py` (all), `tests/e2e/test_chat_rag.py`, `tests/e2e/test_retrieval_quality.py` green.
## Testing & Quality
- Unit: the quadrant table (the new quadrant is the regression pin for TODO L2a).
- E2E: the deflection story extension.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] All quadrants green, including fts>0 + cosine<floor → LOW.
- [ ] E2E: known-out-of-KB question → deflected, zero source chips, 2–3 alternatives.
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,21 @@
# Task 03 — Document the disclosed-answer contract (owner decision iii)
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:46–55, 64–75` — "b) **Model violates the locked prompt** (Rule 1 'Answer ONLY from the provided document context', Rule 3 HONESTY GATE): with irrelevant docs injected it answered from general knowledge. The disclosure is better UX than silence, but the contract says no pretending to know + 2-3 concrete alternative questions — decide whether to (i) tighten the prompt copy … (ii) amend the locked prompt via the plan to explicitly permit disclosed general-knowledge answers, or (iii) treat the observed behavior as acceptable and document it. Owner decision required — the prompt text is locked verbatim (change through the plan, not here)." + the follow-up: "HONESTY GATE compliance is **stochastic** across runs (run 1: parametric answer; run 2: perfect deflection). Implication for the fix direction: the deterministic lever is the gate …, not prompt copy alone; a small local model cannot be relied on to obey Rules 1/3 100% when handed misleading context."
## Objective
Record the owner-confirmed decision (iii): the stochastic disclosed-general-knowledge answer (when misleading docs are injected) is acceptable and documented — the prompt text stays byte-identical (LOCKED verbatim); the deterministic protection is the task-01 gate fix.
## Work
1. `app/rag/prompts.py` — module docstring (the house location for the locked-prompt revision history, e.g. the 2026-08-22 note): add a dated entry recording the 2026-09-15 interactive-test finding (parametric "Ulaanbaatar" answer with a disclosure, on the Mongolia question with two irrelevant docs injected; the clean textbook deflection on the identical one-tap re-run — stochastic compliance) and the owner decision (2026-09-14, roadmap confirmation): treat the disclosed general-knowledge answer as acceptable; the deterministic lever is the gate (A8 revised, task 01); options (i)/(ii) remain open to a future plan amendment. Prompt strings: byte-identical.
2. `README.md` — in the deflection paragraph (the one task 04 rewrites), one sentence: with a small local model, a rare turn may answer from general knowledge with an explicit disclosure when retrieval was borderline — the gate (phase 112) minimizes this; the disclosure is surfaced, never silent.
3. Prompt-lock pin: if no existing test byte-pins the prompt text, add a small `tests/unit/test_prompt_lock.py` asserting the HIGH/LOW prompt constants against pre-phase anchor strings (the executor extracts the pre-phase values when writing the test — e.g. exact prefix/suffix + total length, so any byte change fails).
4. ASSUMPTION: the documentation lives in the prompts.py docstring (the existing revision-history location) + the README — no new docs file.
## Testing & Quality
- Unit: the prompt-lock pin (work item 3).
- Coverage: n/a (docs-only change) — the suite stays green.
## Completion Criteria
- [ ] `app/rag/prompts.py` docstring carries the dated decision entry; the prompt strings are byte-identical (test-pinned).
- [ ] The README deflection section notes the rare disclosed-answer behavior.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,20 @@
# Task 04 — README: fix the stale deflection copy
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:77–81` — "Plus a docs nit: the README still promises the exact deflection copy *'I haven't done anything like that'* — the mandated deflection opening was removed in the 2026-08-22 locked-prompt revision (`app/rag/prompts.py` module docstring). Update the README's 'If it doesn't have notes…' paragraph to match current behavior."
## Objective
The README describes the current deflection behavior (admit no notes + 2–3 concrete alternative questions) instead of the removed mandated opening.
## Work
1. `README.md` L11 ("If it doesn't have notes for your question, it admits it: *'I haven't done…*") — rewrite the quoted copy to match the current locked-prompt behavior: it admits it has no notes on that and offers 2–3 concrete alternative questions about things it DOES have notes on. No exact-copy promise (the opening is no longer mandated).
2. `README.md` L575 ("**Honest deflection** (the amber *'I haven't done anything like that'*…") — same update; keep the amber-banner description accurate.
3. Grep the README for any other occurrence of the old quoted opening and update it.
## Testing & Quality
- Docs-only: no code tests. Grep `tests/` for `haven't done anything` — if any test asserts the old copy, update it to the new behavior.
- Coverage: n/a.
## Completion Criteria
- [ ] `grep -rn "haven't done anything" README.md` → no hits (the quoted opening is gone).
- [ ] The deflection paragraph matches current behavior (admit + 2–3 concrete alternatives).
- [ ] `uv run pytest` green.
@@ -0,0 +1,46 @@
# Phase 113 — Source chip quality: usefulness bar + related-docs tier (TODO L5 + L2c)
**Source:** `TODO.md` L101–147 — "L5 — Recurring weak-hit source chips: the 2nd chip is often noise the answer never used (2026-09-15, brain-of-reese interactive test)" — root-cause chain (L125–129), suggested directions (L131–142), acceptance (L144–146) — plus `TODO.md` L57–62 (L2 part c): "c) **Misleading chips on non-grounded answers** (`chat.py:252` — `done.sources` = weak hits when deflected; by design, but visually a citation). … At minimum: never render them as answer citations."
**Story:** n/a (interactive-test follow-up fix; extends the phase-09 retrieval-quality and phase-05 chip assets).
**Context:** `top_n_docs = 2` (`app/config.py:114`) forces two docs into `plan.docs`; `done.sources` (`app/api/chat.py` ~L796, `ChatDoneEvent.sources`) carries every entry and `appendSources` (`frontend/assets/app.js:1369`) chips them all with identical visual weight — "the answer used this" vs "this also scored" is indistinguishable. Deflected turns carry weak hits in `sources` "by design" but they render as citations. Owner-confirmed direction: the deterministic **usefulness bar** (server-side) + the **visual split** (UI) — cite-gated chips (parsing the model's cited paths) are rejected for now (stochastic with a small model).
## Objective
A document earns a citation slot only when its retrieval signal is vector-corroborated (or the agent explicitly read it via a tool call); everything else that scored is demoted to a clearly secondary "nearby docs" row that never reads as a citation. For a single-document question the turn shows exactly one citation chip; a deflected turn shows no citation chips at all.
## Dependencies
- `112_honesty_gate_weak_hits` (todo) — the gate fix stops weak hits being injected into the HIGH prompt; this phase stops weak docs earning a `done.sources` slot. Same workstream, ordered after the gate.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Usefulness bar (task 01):** new settings `source_usefulness_floor: float = 0.35` (env `BOR_SOURCE_USEFULNESS_FLOOR`, validated `0 <= floor <= relevance_threshold`, mirroring phase 112's floor) and `related_max_docs: int = 2` (env `BOR_RELATED_MAX_DOCS`, validated `>= 0`). In `plan_turn`, retrieval docs are tiered: **cited** = distinct parent docs (best fused-score order, at most `top_n_docs`) whose best hit-chunk **cosine** clears the floor; **related** = the next scored distinct docs (at most `related_max_docs`) that did not clear it. Agent-read docs (`holder.read_docs`, the phase-37 agent tool reads) always stay cited — the model read them via tool calls, so they were used by definition. `TurnPlan` gains `related_docs: list[Document] = []`; the tiering is a new `select_documents_tiered(chunks, n, floor, related_cap) -> tuple[list[Document], list[Document]]` in `app/rag/retriever.py`, with `select_documents` becoming a thin wrapper (legacy behavior byte-identical for existing callers/tests). `query_log.sources` is unchanged (it records retrieval, not citations — locked A3).
- **Done frame (task 01):** `ChatDoneEvent` (`app/schemas.py`) gains `related: list[SourceRef] = []` — additive; old clients ignore unknown fields (house contract, PLAN §4). Built from `plan.related_docs` with the same (source, path) dedupe against the cited list as `cited_docs` already does.
- **UI split (task 02):** `frontend/assets/app.js` — new `appendRelated(wrap, related)` renders a `.msg-meta.related-docs` row under the bubble (only when `related` is non-empty): a small de-emphasized label "Nearby docs, in case:" + one link per doc with the class `related-doc` (NOT `source-chip`) — same `documentUrl(...)` href and left-click → `openDocumentModal` behavior as citation chips, visually secondary (reduced opacity/size/dashed border via theme variables; link contrast ≥4.5:1, WCAG 2.1 AA). The done-frame handler (~L2421) also calls `appendRelated(wrap, ev.related)`; the restored-chat path (~L1561) likewise when the stored payload carries `related` (pre-phase chats don't — graceful). Deflected turns: `ev.sources` is empty (the server change) → no chips; the weak hits arrive in `ev.related` → the row only.
- **NOT touched:** the citation-chip component (`.source-chip` / `appendSources`) for the cited tier; the suggestion chips; `top_n_docs` (ceiling, not quota); the phase-37 agent-read dedupe; the Sources/RAG pages.
## Tasks
1. `01_usefulness_bar_sources.md` — retriever tiering + `TurnPlan.related_docs` + `ChatDoneEvent.related` + the two settings.
2. `02_secondary_related_docs_ui.md` — the related-docs row in the chat UI (app.js + styles.css); deflected turns show no chips.
3. `03_chip_filter_tests.md` — unit pins on the four observed live shapes + E2E chip-count assertions.
## Testing & Quality
- Unit: `tests/unit/test_retriever.py` (extend — the tiering table), `tests/unit/test_source_chip_quality.py` (new, task 03 — the four observed shapes), the done-frame schema tests (`related` defaults `[]`; old payloads without the field still parse).
- E2E: `tests/e2e/test_source_chip_quality.py` (new, task 03; run in isolation: `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov`) — a known single-source question → exactly one citation chip; a deflected question → zero `.source-chip` elements (the row, if any, is `.related-doc`, never `.source-chip`).
- Regression: `tests/e2e/test_retrieval_quality.py`, `test_honest_deflection.py`, `test_chat_rag.py`, `test_sources_midstream_bug.py` stay green.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] For a single-document question, the turn shows one citation chip (E2E).
- [ ] A weak 2nd doc renders only in the de-emphasized related row, never as a `.source-chip` (unit + E2E).
- [ ] A deflected turn renders zero citation chips (the weak hits, if any, live in the related row).
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — direction: usefulness bar (server, deterministic) + visual split (UI); cite-gated chips rejected for now (owner-confirmed 2026-09-14, roadmap confirmation).**
- **A2 — citation slot = vector-corroborated retrieval doc (best hit-chunk cosine >= `source_usefulness_floor`, default 0.35, env-tunable) OR agent-read doc; `top_n_docs` stays a ceiling, not a quota (owner-confirmed 2026-09-14).**
- **A3 — `query_log.sources` keeps recording the full retrieval (observability); `done.sources` records only the cited tier (owner-confirmed 2026-09-14).**
- **A4 — the related tier is capped at 2 docs (`related_max_docs`, env `BOR_RELATED_MAX_DOCS` — owner-confirmed 2026-09-14).**
## Commit
```bash
git add app/ tests/ frontend/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): tier sources by a usefulness bar — weak hits become a de-emphasized related-docs row, never citation chips"
```
@@ -0,0 +1,28 @@
# Task 01 — Usefulness bar: tier done.sources into cited + related
**Phase:** `113_source_chip_quality` · **Source:** `TODO.md:125–129, 137–140` — "Root cause chain: `top_n_docs = 2` (`app/config.py` L114) forces retrieval to return two documents, `done.sources` carries both (`app/api/chat.py` L252), and the UI chips every entry without distinguishing 'the answer used this' from 'this also scored'." + "**Usefulness bar on the 2nd doc** — only include a document in `done.sources` when its fused/cosine score clears a threshold (a single weak FTS token hit should not earn a citation slot); `top_n_docs` stays a ceiling, not a quota." + `TODO.md:57–62` (L2 part c: "`done.sources` = weak hits when deflected; by design, but visually a citation … At minimum: never render them as answer citations.")
## Objective
Retrieval docs are tiered at the honesty gate: cited (vector-corroborated, ≤ `top_n_docs`) vs related (scored but under the floor, ≤ `related_max_docs`); the SSE done frame carries both; agent-read docs always stay cited.
## Work
1. `app/config.py` — add `source_usefulness_floor: float = Field(default=0.35)` (env `BOR_SOURCE_USEFULNESS_FLOOR`) and `related_max_docs: int = Field(default=2)` (env `BOR_RELATED_MAX_DOCS`); validators: `0 <= source_usefulness_floor <= relevance_threshold`, `related_max_docs >= 0`; `.env.example` entries.
2. `app/rag/retriever.py` — add `select_documents_tiered(chunks, n, floor, related_cap) -> tuple[list[Document], list[Document]]`:
- rank distinct parent docs by best fused score (the existing `select_documents` ordering), tracking each doc's best hit-chunk cosine;
- **cited** = the docs whose best-chunk cosine >= `floor`, up to `n` (ceiling — a single strong doc yields one cited doc);
- **related** = the next docs in rank order (any cosine, including 0.0 lexical-only), up to `related_cap`, never overlapping the cited list.
- `select_documents` becomes a wrapper: `cited, _ = select_documents_tiered(chunks, n, 0.0, 0)` — floor 0.0 + cap 0 keeps the legacy "any score, top-N" behavior byte-identical for existing callers/tests.
3. `app/api/chat.py` — `TurnPlan` gains `related_docs: list[Document] = []` (after `docs`); `plan_turn` calls `select_documents_tiered(chunks, settings.top_n_docs, settings.source_usefulness_floor, settings.related_max_docs)` → `docs`, `related_docs` (both the HIGH and LOW branches — deflected turns: the weak hits fall to related, cited is usually empty). The done-frame build (~L796): add `related=[SourceRef(source=d.source, path=d.path, title=d.title) for d in <plan.related_docs deduped against cited_docs by (source, path)>]` — the same dedupe pattern `cited_docs` already uses.
4. `app/schemas.py` — `ChatDoneEvent` gains `related: list[SourceRef] = []` (additive; docstring note: old clients ignore unknown fields, PLAN §4).
5. `query_log.sources` — unchanged (locked A3).
6. ASSUMPTION: the bar is on the **cosine** of the doc's best hit chunk, not the RRF fused score — the fused `score` is a rank key, not a similarity; a lexical-only hit has cosine 0.0 and is vector-unsupported by definition (consistent with the phase-112 gate; locked A2).
## Testing & Quality
- Unit: `tests/unit/test_retriever.py` (extend) — the tiering table: both clear → both cited; strong + weak → 1 cited + 1 related; both weak → 0 cited + 2 related; related cap respected; `select_documents` wrapper legacy behavior unchanged (existing tests stay green without edits).
- Unit: the existing done-frame/schema tests — `related` defaults to `[]`; a payload without the field still parses (back-compat).
- Coverage: **>90%** on `app/` including the new function.
## Completion Criteria
- [ ] `plan_turn` returns tiered docs; HIGH and LOW branches both populated correctly.
- [ ] The done frame carries `related` (≤ `related_max_docs`, deduped against cited); old frames (no field) parse.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,27 @@
# Task 02 — UI: the related-docs row (never a citation chip)
**Phase:** `113_source_chip_quality` · **Source:** `TODO.md:141–142` — "**Visual split** — keep both, but render uncited/weak docs as a clearly secondary 'related docs' row, not citation chips." + `TODO.md:59–62` — "When the answer is disclosed general knowledge (or a deflected turn), the chips should be visually de-emphasized / labeled (e.g. 'nearby docs I have, in case'), or omitted when the brain says it didn't use them. At minimum: never render them as answer citations."
## Objective
The chat bubble renders the cited tier exactly as today (`.source-chip` via `appendSources`) and the related tier as a clearly secondary labeled row; a deflected turn renders zero citation chips.
## Work
1. `frontend/assets/app.js`:
- new `appendRelated(wrap, related)` next to `appendSources` (L1369): early-return when empty; a `.msg-meta.related-docs` row (`role="list"`, `aria-label="Nearby docs, in case"`) + a small `<span class="related-docs-label">Nearby docs, in case:</span>` + one link per doc — class `related-doc` (NOT `source-chip`), the same `documentUrl(s.source, s.path, "/")` href and left-click → `openDocumentModal(s.source, s.path, link)` behavior, `title`/`aria-label` carrying the full path.
- the done-frame handler (~L2421, next to `appendSources(wrap, ev.sources)`): also `appendRelated(wrap, ev.related)`.
- the restored-chat path (~L1561): same, when the stored payload carries `related` (pre-phase saved chats don't — the row is simply absent, graceful).
- deflected turns: `ev.sources` is empty (the task-01 server change) → `appendSources` no-ops; the weak hits arrive in `ev.related` → row only.
2. `frontend/assets/styles.css` — `.related-docs` (muted row: smaller font, theme-variable color — link text contrast ≥4.5:1, WCAG 2.1 AA), `.related-doc` (dashed border, no hover elevation of the citation chips; `focus-visible` ring), `.related-docs-label` (small caps or muted small text); the row stacks below the citation `.msg-meta` row with the existing gap.
3. Frontend unit test (house source-assertion style — lives in `tests/unit/test_source_chip_quality.py`, extended by task 03): `appendRelated` exists and uses `related-doc` (assert `source-chip` is NOT in the `appendRelated` body); the row renders only when related is non-empty; the label text is present; the done handler calls `appendRelated`.
4. ASSUMPTION: label copy "Nearby docs, in case:" (the TODO's suggested wording, trimmed).
5. ASSUMPTION: pre-phase saved chats (stored payload without `related`) restore exactly as today — no related row.
## Testing & Quality
- Unit: the frontend source-assertion tests (task 03's file).
- E2E: pinned by task 03.
- Coverage: n/a (frontend) — the `app/` gate stays green.
## Completion Criteria
- [ ] A related doc renders only as `.related-doc` in the labeled row — never as `.source-chip`.
- [ ] A deflected turn (mock) renders zero `.source-chip` elements under the bubble.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,31 @@
# Task 03 — Pin the chip contract: the four observed shapes + E2E chip counts
**Phase:** `113_source_chip_quality` · **Source:** `TODO.md:110–123, 144–146` — the four observed live cases (1. "What is the capital of Mongolia?" → `Trooper_Nagraz.pl` + `Trooper_Begzei.pl`, both unrelated; 2. phase-gate question answered from `brain-of-reese/.agents/validate.sh` → second chip `ServMon/README.md` unused; 3. Trooper_Nagraz question answered from `Trooper_Nagraz.pl` → second chip `Trooper_Byzin.pl` uncited; 4. meta question about the conversation's own history → chips `app/api/suggestions.py` + `108_history_wire_check/00_phase.md`, neither used) + "Acceptance: for a single-document question, the turn shows one citation chip; a unit test pins `done.sources` filtering (or the chip renderer's cite-gate) on the four observed shapes; E2E asserts chip count for a known single-source question."
## Objective
A unit test pins the cited/related tiering on the four live shapes; the E2E suite asserts the visible chip counts.
## Work
1. `tests/unit/test_source_chip_quality.py` (new) — model each of the four observed shapes as a `plan_turn`/done-frame fixture (retrieval chunks with controlled cosine/`fts_hit`/fused `score`) and assert the tiering:
1. **both docs weak** (cosine < floor, the Mongolia case) → `sources` empty, `related` ≤ 2.
2. **one strong + one weak** (the validate.sh case: `validate.sh` cosine ≥ floor, `ServMon/README.md` below) → exactly 1 in `sources`, the weak one in `related`.
3. **the Nagraz case** — same shape, different fixtures (`Trooper_Nagraz.pl` strong, `Trooper_Byzin.pl` weak) → 1 cited, 1 related.
4. **the meta/history question** (no doc clears the floor, the agent reads nothing) → `sources` empty, `related` ≤ 2; assert the frame shape that the UI renders as row-only (the rendering is pinned by task 02's source tests + the E2E).
5. **agent-read exemption**: a doc under the floor that is in `holder.read_docs` (agent tool read) still lands in `sources` (cited).
- plus the frontend source-assertion tests from task 02 work item 3 (same file).
2. `tests/e2e/test_source_chip_quality.py` (new; conftest/mock-LLM/fixture-KB pattern per `tests/e2e/test_retrieval_quality.py`):
- a known single-source question (a fixture-KB question whose answer comes from one doc) → the done bubble has **exactly one** `.source-chip`.
- a deflected question (known-out-of-KB) → **zero** `.source-chip`; if a `.related-docs` row exists, its links are `.related-doc`, never `.source-chip`.
- if the fixture KB cannot produce a strong+weak two-tier shape, say so in the test docstring and rely on the unit table for that shape.
3. Run in isolation: `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (DB up).
4. Regression: `tests/e2e/test_retrieval_quality.py`, `test_honest_deflection.py`, `test_chat_rag.py` green.
## Testing & Quality
- Unit: the four-shape table (the acceptance pin) + the agent-read exemption.
- E2E: the chip-count assertions (the acceptance pin).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] The four observed shapes are unit-pinned (plus the agent-read exemption).
- [ ] E2E: single-source question → exactly one citation chip; deflected turn → zero.
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,45 @@
# Phase 114 — Embed question length: truncation + accurate error (TODO L6)
**Source:** `TODO.md` L149–181 — "L6 — 4,000-char question clamp exceeds the embed model's input cap → misleading 'couldn't reach the embedding model' error (2026-09-15, brain-of-reese interactive test)"
**Story:** n/a (interactive-test follow-up fix; extends the phase-67 LLM-retry and phase-06 loading-feedback assets).
**Context:** The composer clamps at 4,000 chars; `app/api/chat.py:423` embeds the **full** question via `llm.embed_one`; aipi's litellm rejects the ~903-token input with **HTTP 500**: "input (903 tokens) is too large to process. increase the physical batch size (current batch size: 512)". The single-text path in `app/rag/llm.py` (`_embed_batch` → `_TooLarge`, ~L279–284) turns that into `EmbeddingError("a single …-char chunk exceeded the endpoint's per-request input token cap — lower BOR_CHUNK_TARGET_CHARS and re-import")` — an **import-oriented** message — and the chat endpoint's catch-all (~L428–444) maps EVERY `EmbeddingError` to "I couldn't reach the embedding model — please try again." Both diagnoses are wrong (reachability is fine; the chunker constant is irrelevant to a question). The chunker's own `HARD_MAX_CHARS = 1200` (`app/rag/chunker.py:51`, ~1024 tokens at ~1.4 chars/token) shows the question path never got the same treatment.
## Objective
Every legal question (≤ the UI clamp) is embeddable: the embed step gets a bounded prefix of the question (the chunker's 1200-char budget) while the full question still reaches the LLM prompt; and if the input is still too large (a smaller-cap model, a misconfiguration), the turn fails with an accurate "question too long" error — no false reachability diagnosis, no wasted retries — and the banner carries the phase-111 Retry button.
## Dependencies
- `111_chat_banner_retry` (todo) — L6's acceptance: "the L1 'Try again' button fix should also apply to this banner" — the too-long error flows through the same turn-error state machine, so the phase-111 Retry button is offered on it.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Truncation (task 01):** new setting `embed_question_max_chars: int = 1200` (env `BOR_EMBED_QUESTION_MAX_CHARS`, default = the chunker's `HARD_MAX_CHARS` budget, validated `> 0`). The chat embed step (chat.py:423) embeds `request.message[:settings.embed_question_max_chars]`; the LLM prompt build is unchanged (the full question still reaches the model). Questions shorter than the budget are byte-identical to today.
- **Error mapping (task 02):** `app/rag/llm.py` — new `EmbeddingInputTooLargeError(EmbeddingError)` subclass; the single-text `_TooLarge` branch of `_embed_batch` raises it (same message text — the importer path is byte-identical, it still catches `EmbeddingError`). The chat endpoint catches `EmbeddingInputTooLargeError` **before** `EmbeddingError` inside the phase-67 retry loop → no retry (a deterministic failure — locked A3) → terminal `ChatErrorEvent` with `detail="Question too long — trim it and re-ask."` and a new optional `hint` field: `hint="The app reached the embedding model fine — only the question length is the problem."` `ChatErrorEvent` gains `hint: str | None = None` (additive; PLAN §4 old-client ignore contract). The frontend's phase-111 reworked `showErrorBanner(detail, opts)` shows `opts.hint` when the frame carries one, else the default `ERROR_HINT`.
- **Retry:** the too-long frame flows through the turn-error state machine → the phase-111 banner Retry button is offered (re-asking is the user's call after trimming; the composer clamp still applies).
- **NOT touched:** the importer's embed path and its batch-halving `_TooLarge` behavior/error copy, the 4,000-char composer clamp (locked A2 — truncation, not a lower clamp), the reachability-failure retry semantics (phase 67 — byte-identical).
## Tasks
1. `01_embed_truncation.md` — the bounded-prefix embed + the setting.
2. `02_too_long_error_mapping.md` — `EmbeddingInputTooLargeError`, the chat-path mapping, `ChatErrorEvent.hint`, the frontend hint support.
3. `03_embed_length_tests.md` — the unit pins + the 4,000-char E2E.
## Testing & Quality
- Unit: `tests/unit/test_embed_question_length.py` (new, task 03) — truncation (long → prefix embedded, LLM prompt carries the full text; short → byte-identical), error mapping (too-large failure → exact detail + hint, no retry frame, one attempt; transport failure → legacy reachability path with retries — the regression pin), the config validator.
- E2E: `tests/e2e/test_embed_question_length.py` (new, task 03; run in isolation: `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov`) — a 4,000-char question (the composer clamp) streams to done (mock LLM), no error banner.
- Regression: `tests/e2e/test_llm_retry.py`, `test_oneshot_llm_retry.py`, `test_chip_sizing_question_cap.py` (the 4,000-char counter) stay green.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] A 4,000-char question embeds (bounded prefix) and the turn succeeds; the LLM prompt carries the full question.
- [ ] A too-large embed failure (forced in a unit test) → the accurate "Question too long" frame + the reachability-fine hint; the banner offers the phase-111 Retry button.
- [ ] A reachability embed failure behaves byte-identically to pre-phase (retries + old copy).
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — both fixes combined: 1200-char embed truncation (default = the chunker budget, env-tunable) + the precise too-long error mapping (owner-confirmed 2026-09-14, roadmap confirmation).**
- **A2 — the 4,000-char composer clamp stays (owner-confirmed 2026-09-14) — truncation, not a lower clamp.**
- **A3 — a too-large embed failure is NOT retried (deterministic failure) — it short-circuits the phase-67 retry loop (owner-confirmed 2026-09-14).**
## Commit
```bash
git add app/ tests/ frontend/ .agents/phases/ && git commit --no-gpg-sign -m "fix(rag): embed a bounded question prefix (1200-char budget) and map the embed too-large failure to an accurate too-long error with a reachability-fine hint"
```
@@ -0,0 +1,21 @@
# Task 01 — Embed a bounded question prefix
**Phase:** `114_embed_question_length` · **Source:** `TODO.md:151–164, 168–170` — "Repro: type/paste a question to the UI maximum (the composer clamps at 4,000 chars — char counter shows '4000/4000 — character limit') and send. Result, **100% reproducible**: the turn dies pre-token with the banner 'I couldn't reach the embedding model — please try again.' … a short question embeds fine (HTTP 200), but the 4,000-char question (~903 tokens) gets **HTTP 500** from aipi … So the maximum legal question length exceeds the embed model's maximum legal input — and the chunker's own 1200-char cap (set to stay under the ~1024-token per-request cap) shows the question path never got the same treatment." + "**Truncate for embedding** — embed a bounded prefix of the question (e.g. the same 1200-char budget as chunks) while the full question still reaches the LLM prompt."
## Objective
The chat embed step embeds at most `embed_question_max_chars` (default 1200 — the chunker's `HARD_MAX_CHARS` budget) of the question; the full question still reaches the LLM prompt.
## Work
1. `app/config.py` — add `embed_question_max_chars: int = Field(default=1200)` (env `BOR_EMBED_QUESTION_MAX_CHARS`), validator `> 0`; `.env.example` entry with a comment citing the chunker rationale (`app/rag/chunker.py:30–51` — ~1.4 chars/token, stays under the ~1024-token per-request cap).
2. `app/api/chat.py` — the embed step (~L423): `question_vec = await llm.embed_one(request.message[: settings.embed_question_max_chars])`. Everything downstream is unchanged: retrieval runs on the prefix vector (intended — the prefix is the question's head); the LLM prompt build (`hist` + the full `request.message`) is untouched; the per-turn log line is untouched (`question=%r` logs the full text).
3. One-line comment at the call site: the prefix is bounded to the embed model's input cap (the chunker budget); the full question still reaches the LLM prompt (TODO L6).
4. ASSUMPTION: the budget is a setting (env-tunable), default 1200 — not a hard-coded constant — so a model with a larger/smaller cap is accommodated without a code change (locked A1).
## Testing & Quality
- Unit: `tests/unit/test_embed_question_length.py` (new, task 03) — a question > the budget → `embed_one` receives exactly the prefix (mock LLM client); the LLM request messages carry the full question; a question ≤ the budget → byte-identical call.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] A 4,000-char question → `embed_one` called with the 1200-char prefix; the LLM request carries the full 4,000-char message.
- [ ] A short question → no behavior change (byte-identical call).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,31 @@
# Task 02 — Map the too-large embed failure to an accurate error
**Phase:** `114_embed_question_length` · **Source:** `TODO.md:171–173` — "**Map the 500 to a precise error** — detect the 'too large' embed failure and surface 'question too long — trim it' (and fix the ERROR_HINT for this case: reachability is fine)." + `TODO.md:179–181` — "Acceptance: a 4,000-char question either succeeds (truncated embedding) or fails with an accurate too-long error; unit test pins the error mapping; the L1 'Try again' button fix should also apply to this banner."
## Objective
A deterministic "input too large" embed failure surfaces as a precise "question too long" terminal error with a hint that reachability is fine — no false reachability diagnosis, no wasted retries; the error banner carries the phase-111 Retry button (the turn-error path).
## Work
1. `app/rag/llm.py` — add `class EmbeddingInputTooLargeError(EmbeddingError)` (near `EmbeddingError`, L42), with a docstring: the single-text input exceeded the endpoint's token cap — deterministic, not a reachability failure. In the single-text `_TooLarge` branch of `_embed_batch` (~L279–284): raise `EmbeddingInputTooLargeError(<the existing import-oriented message>)` instead of plain `EmbeddingError` — the message text is **identical** (the importer path is byte-identical; it still catches `EmbeddingError`, and the subclass is a drop-in).
2. `app/schemas.py` — `ChatErrorEvent` gains `hint: str | None = None` (additive; docstring: the client shows the hint in place of its default reachability hint when present; old clients ignore the field — PLAN §4).
3. `app/api/chat.py` — the embed-failure handling (~L428–444, inside the phase-67 retry `while` loop): catch `EmbeddingInputTooLargeError` **before** `EmbeddingError` → do NOT restart (locked A3 — deterministic) → `settled = True`, log an error line (the existing format plus a `too-large` marker), and yield:
```python
ChatErrorEvent(
detail="Question too long — trim it and re-ask.",
hint="The app reached the embedding model fine — only the question length is the problem.",
).model_dump()
```
The existing `EmbeddingError` branch (reachability) is unchanged, including the retry semantics and the old copy.
4. `frontend/assets/app.js` — the phase-111 reworked `showErrorBanner(detail, opts)`: honor `opts.hint` — `bannerText.textContent = detail ? \`${detail} ${opts.hint ?? ERROR_HINT}\` : (opts.hint ?? ERROR_HINT)`. The SSE error-frame handler in the stream state machine (~L1281): pass `{ retryable: true, hint: ev.hint }` when the frame carries a hint.
5. ASSUMPTION: detail copy "Question too long — trim it and re-ask." (the TODO's "question too long — trim it", phrased as a banner sentence); hint copy as in work item 3.
6. ASSUMPTION: no retry on too-large (locked A3) — the phase-67 retry loop is for transient failures; a size failure is guaranteed to repeat.
## Testing & Quality
- Unit: `tests/unit/test_embed_question_length.py` (new, task 03) — force the `_TooLarge` branch (a fake httpx response: HTTP 500 + a "too large to process" body) → the chat SSE stream yields exactly one error frame with the precise detail + hint and **no** retry frame; a transport failure (no "too large" signature) → the legacy reachability path with the retry loop and old copy (the regression pin).
- Coverage: **>90%** on `app/` including the new exception class and branch.
## Completion Criteria
- [ ] A too-large embed failure → the frame `{type: "error", detail: "Question too long — trim it and re-ask.", hint: "…fine…"}` — no "couldn't reach" copy, no retry frame.
- [ ] A reachability embed failure → byte-identical to pre-phase (retries + old copy).
- [ ] The frontend shows the frame's hint when present; the banner offers the phase-111 Retry button on this error.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,25 @@
# Task 03 — Unit + E2E pins for the 4,000-char question
**Phase:** `114_embed_question_length` · **Source:** `TODO.md:179–181` — "Acceptance: a 4,000-char question either succeeds (truncated embedding) or fails with an accurate too-long error; unit test pins the error mapping; the L1 'Try again' button fix should also apply to this banner."
## Objective
The acceptance is pinned: a full-clamp (4,000-char) question succeeds end-to-end (truncated embed), and the too-long mapping is unit-pinned.
## Work
1. `tests/unit/test_embed_question_length.py` (new):
- **truncation:** a mock `LLMClient` records the `embed_one` input; a 4,000-char question → exactly the prefix (default budget); the chat request to the LLM carries the full question; a 100-char question → byte-identical call.
- **error mapping:** a fake embed transport returning HTTP 500 + "too large to process" body for the input → the chat SSE stream yields exactly one error frame with the precise detail + the reachability-fine hint and no retry frame; short input + a 500 WITHOUT the "too large" signature → the legacy reachability path (retry frames + old copy) — the regression pin.
- **config:** the `embed_question_max_chars` default (1200) and validator.
2. `tests/e2e/test_embed_question_length.py` (new; the `tests/e2e/` conftest + mock-LLM pattern): type a 4,000-char question into the composer (the counter shows "4000/4000") → send → the turn streams to done (mock LLM) — no error banner.
3. Run in isolation: `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov` (DB up).
4. Regression: `tests/e2e/test_llm_retry.py`, `test_oneshot_llm_retry.py`, `test_chip_sizing_question_cap.py` (the 4,000-char counter) stay green.
## Testing & Quality
- Unit: as above (the acceptance pin: the error mapping).
- E2E: the 4,000-char success path (the acceptance pin: the truncated embed).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] A 4,000-char question → a successful turn (E2E); the embed input was the prefix (unit).
- [ ] The too-long mapping is unit-pinned (exact frame, no retry).
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,45 @@
# Phase 115 — Doc drafts: Discard + DELETE route + title fix (TODO L7)
**Source:** `TODO.md` L183–209 — "L7 — 'Save as doc' has no Discard: orphan drafts are invisible and un-deletable (2026-09-15, brain-of-reese interactive test)"
**Story:** n/a (interactive-test follow-up fix; extends the phase-59/75 doc-draft and save-as-doc assets).
**Context:** "Save as doc" (`frontend/assets/app.js::saveAsDoc`, L720) POSTs `/api/doc-drafts` (201) → `/doc-edit.html?draft=<token>`. The edit screen offers exactly one action: **"Push to docs branch"**. The drafts router (`app/api/doc_drafts.py`) has POST/GET/PUT/POST-push only — no DELETE, no TTL/pruning; a draft created by an accidental click (or a tester) sits orphaned in the DB forever — invisible (no UI lists drafts) and only consumable by actually pushing a doc to the repo. Side observation: the draft's default title comes from `defaultDocTitle()` (app.js:619) — the **last user record** in the conversation — and after a Retry redo-in-place (phase 49) the redone answer sits at the end, so its title came from an unrelated trailing question (a junk 4,000-char test question, not the question the answer answered). The body (full session transcript) is correct by design; only the title derivation mismatches.
## Objective
An orphaned doc draft can be discarded from the edit screen (a new admin-gated `DELETE /api/doc-drafts/{token}` + a Discard control), and the draft's default title is the question the saved answer actually answered (its paired user record) — fixing the retry-redo mismatch.
## Dependencies
- `114_embed_question_length` (todo) — pipeline predecessor (execution order) only; no code dependency.
## Design (shared by all tasks — the executor reads this, not the chat)
- **DELETE route (task 01):** `DELETE /api/doc-drafts/{token}` → **204**. The whole router already sits behind `require_admin` (phase 59, `dependencies=[Depends(require_admin)]` at L63) — the new route inherits it; the uuid4 token is the screen's credential (same trust model as GET/PUT/push). Unknown token → 404 via the existing `_get_draft_or_404` helper. No migration (a row delete); no push-side state (the git push happens only on push).
- **Discard UI (task 02):** the doc-edit screen — a "Discard draft" control next to "Push to docs branch" (secondary/danger treatment per the theme). `frontend/assets/doc-edit.js`: `confirm()` (destructive + irreversible — no undo exists), `DELETE /api/doc-drafts/${token}` (the same token the screen already uses for GET/PUT) → 204 → `location.assign("/")` (back to the chat page). Non-204 → the page's existing inline-error pattern, no navigation.
- **Title fix (task 03):** `defaultDocTitle(wrap)` — takes the saved brain bubble's wrap (the `.save-as-doc-btn`'s bubble); the title is the text of the user bubble **paired** with that brain bubble (the nearest preceding user message in the DOM conversation flow), falling back to the current last-user-record-in-`conversation` logic when no wrap is given or no paired user bubble is found (first-turn edge / DOM mismatch). The `DOC_TITLE_MAX` slice + "Note" fallback are unchanged.
- **NOT touched:** the push flow (byte-identical), the draft body (`buildSessionTranscript` — full session, correct by design), the drafts schema (no field change), no TTL/pruning (locked A1 — out of scope).
## Tasks
1. `01_delete_draft_route.md` — `DELETE /api/doc-drafts/{token}` (204 / 404 / admin-gated).
2. `02_discard_ui.md` — the Discard control on the doc-edit screen wired to the route.
3. `03_draft_title_fix.md` — the title from the answer's own question (the paired user record).
4. `04_draft_discard_tests.md` — the API + frontend + E2E pins.
## Testing & Quality
- Integration: `tests/integration/test_doc_drafts_api.py` (existing — extend, task 04): DELETE removes the row (204; subsequent GET 404); unknown token → 404; the admin gate applies (same assertions the sibling routes use).
- Unit (frontend, house source-assertion style, task 04): the Discard control's presence/handler (confirm → DELETE → 204 → redirect; non-204 → inline error, no navigation); the `defaultDocTitle` pairing logic + the call site.
- E2E: `tests/e2e/test_save_doc_session.py` (existing — extend, task 04): the discard flow (save → edit screen → discard → confirm → back on the chat, draft gone); a save-as-doc after a Retry redo-in-place → the title matches the redone answer's own question. Run in isolation: `uv run pytest tests/e2e/test_save_doc_session.py -v --no-cov`.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] An orphaned draft can be discarded from the edit screen; the draft row is gone afterward (integration).
- [ ] The title of a save-as-doc after a retry redo matches the redone answer's own question (E2E).
- [ ] The push flow is byte-identical (regression green).
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — scope: the Discard control + the DELETE route + the title fix; NO TTL/pruning (owner-confirmed 2026-09-14, roadmap confirmation — the discard covers the acceptance; pruning can be a future phase).**
- **A2 — the DELETE route is admin-gated by the router-level `require_admin` (phase 59) with the uuid4 token as the credential — same trust model as the sibling routes (owner-confirmed 2026-09-14).**
## Commit
```bash
git add app/ tests/ frontend/ .agents/phases/ && git commit --no-gpg-sign -m "feat(docs): discard doc drafts from the edit screen (DELETE /api/doc-drafts/{token}) + derive the draft title from the answer's own question"
```
@@ -0,0 +1,37 @@
# Task 01 — DELETE /api/doc-drafts/{token}
**Phase:** `115_doc_draft_discard` · **Source:** `TODO.md:185–199, 201–204` — "Clicking 'Save as doc' on an answer POSTs `/api/doc-drafts` (201) and navigates to `/doc-edit.html?draft=<token>`. The edit screen offers exactly one action: **'Push to docs branch'**. There is no Discard/cancel control, the drafts API has no DELETE route (`app/api/doc_drafts.py`: POST, GET, PUT, POST /push only) and no TTL/pruning. A draft created by an accidental click (or a tester) sits orphaned in the DB forever — invisible (no UI lists drafts) and only consumable by actually pushing a doc to the repo." + "Suggested fix: add a Discard control to `/doc-edit.html` wired to a new `DELETE /api/doc-drafts/{token}` (admin-gated, token = the screen's credential) …"
## Objective
A new admin-gated `DELETE /api/doc-drafts/{token}` removes a draft row (204) so an orphaned draft can be discarded instead of only pushed.
## Work
1. `app/api/doc_drafts.py` — add (after the `update_draft` PUT route, before the push route — or at the file's route-order idiom):
```python
@router.delete("/{token}", status_code=status.HTTP_204_NO_CONTENT)
def delete_draft(
token: uuid.UUID,
db: Session = Depends(get_db), # noqa: B008
) -> None:
"""Discard a draft (the edit screen's Discard control, phase 115).
Admin-gated like the whole router (phase 59); the uuid4 token is
the screen's credential — after a successful discard, GET/PUT/push
all 404.
"""
row = _get_draft_or_404(db, token)
db.delete(row)
db.commit()
```
Match the file's existing import style (check whether `status` from `fastapi` is already imported; use the file's idiom for the 204 response).
2. No migration, no schema change, no other route touched.
3. ASSUMPTION: 204 No Content (no body) — the token is a one-way credential; nothing else references the row (no FK targets, no push-side state).
## Testing & Quality
- Integration: extended in task 04 (`tests/integration/test_doc_drafts_api.py`).
- Coverage: **>90%** on `app/` including the new route.
## Completion Criteria
- [ ] `DELETE /api/doc-drafts/{token}` → 204; the subsequent `GET` → 404.
- [ ] Unknown token → 404; the admin gate applies exactly like the sibling routes.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,33 @@
# Task 02 — Discard control on the doc-edit screen
**Phase:** `115_doc_draft_discard` · **Source:** `TODO.md:201–204` — "Suggested fix: add a Discard control to `/doc-edit.html` wired to a new `DELETE /api/doc-drafts/{token}` (admin-gated, token = the screen's credential), or a TTL/prune for stale drafts; …" + `TODO.md:207–208` — "Acceptance: an orphaned draft can be discarded from the edit screen; the draft row is gone afterward (API test) …"
## Objective
The doc-edit screen offers a Discard control: confirm → DELETE → back to the chat.
## Work
1. The doc-edit template (find the exact HTML file — the edit screen that renders "Push to docs branch"; `frontend/doc-edit.html` or the template it uses): add next to the push control:
```html
<button type="button" id="discard-draft" class="discard-draft"
title="Delete this draft permanently — this cannot be undone">Discard draft</button>
```
visually secondary to the push button (the theme's muted/danger treatment).
2. `frontend/assets/doc-edit.js` — a handler near the push handler:
- `if (!confirm("Discard this draft? This cannot be undone.")) return;`
- `fetch(\`/api/doc-drafts/${token}\`, { method: "DELETE" })` (the same token the screen already uses for GET/PUT, per its existing load code ~L136–143);
- 204 → `location.assign("/")` (back to the chat page);
- non-204 → the page's existing inline-error pattern (message + no navigation, no crash).
3. `frontend/assets/styles.css` — `.discard-draft`: the secondary/danger button style (contrast ≥4.5:1, `focus-visible` ring per the theme), laid out next to the push button.
4. Frontend unit tests (house source-assertion style, shipped in task 04's `tests/unit/test_frontend_doc_draft_discard.py`): the button's presence; `confirm(...)` before the DELETE; 204 → redirect; non-204 → inline error, no navigation.
5. ASSUMPTION: after a successful discard the user lands on the chat page (`/`) — the draft has no other home (no drafts list exists).
6. ASSUMPTION: a native `confirm()` is acceptable for this one destructive action (the codebase has no custom dialog asset — if the executor finds one in the theme, use it instead).
## Testing & Quality
- Unit: the frontend source-assertion tests (task 04).
- E2E: the discard flow (task 04, `tests/e2e/test_save_doc_session.py`).
- Coverage: n/a (frontend) — the `app/` gate stays green.
## Completion Criteria
- [ ] The edit screen shows "Discard draft" next to the push control; the confirm dialog appears; a 204 returns the user to `/`.
- [ ] A failed DELETE (e.g. a 404 race) shows the inline error; no navigation; no crash.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,26 @@
# Task 03 — Draft title from the answer's own question
**Phase:** `115_doc_draft_discard` · **Source:** `TODO.md:193–199` — "Side observation (edge case, same test): the draft's default title is the user record *immediately preceding* the saved answer. After a Retry redo-in-place (L-see app.js `retryLastTurn`), the redone answer sits at the end of the conversation, so its save-as-doc title came from an unrelated trailing question (the doc was titled with a junk 4,000-char test question, not the question the answer answered). Body is the full session transcript (correct by design); title derivation is just the mismatch." + `TODO.md:203–204` — "consider deriving the default title from the question the answer actually answered (its paired user record) rather than the preceding record."
## Objective
The draft's default title is the user question paired with the saved brain bubble (its own question) — after a Retry redo-in-place the title matches the redone answer's question.
## Work
1. `frontend/assets/app.js` — `defaultDocTitle()` (L619) → `defaultDocTitle(wrap)`:
- when *wrap* (the brain bubble) is given, walk the DOM conversation flow backwards from *wrap* to the **nearest user message bubble** (confirm the exact user-bubble class — the `.msg` variant used for user turns) and use its text;
- fall back to the current last-user-record-in-`conversation` logic when *wrap* is absent or no paired user bubble is found (first-turn edge / DOM mismatch);
- the `DOC_TITLE_MAX` slice + whitespace collapse + "Note" fallback are unchanged.
2. `saveAsDoc(btn)` (L720) — pass the bubble: `const title = defaultDocTitle(btn.closest(<the bubble class>))` — the save button lives in the bubble's meta (the `addSaveAsDocButton` code ~L704–710 shows the exact ancestor; use the same class `lastBrainWrap` uses at L577).
3. The `docSlug`/path logic is unchanged (it derives from the title).
4. Frontend unit tests (house source-assertion style, task 04's file): `defaultDocTitle` takes a wrap arg and prefers the paired user bubble over the last conversation record; the `saveAsDoc` call site passes the bubble ancestor.
5. ASSUMPTION: the pairing is DOM-structural (nearest preceding user bubble), not index-based — the redo-in-place reorders the DOM, and the structural pair IS the answer's question by construction.
## Testing & Quality
- Unit: the frontend source-assertion tests (task 04).
- E2E: the title-after-retry pin (task 04, `tests/e2e/test_save_doc_session.py`).
- Coverage: n/a (frontend) — the `app/` gate stays green.
## Completion Criteria
- [ ] A save-as-doc on a normal (non-redone) answer → the same title as pre-phase (no regression).
- [ ] A save-as-doc after a Retry redo-in-place → the title is the redone answer's own question (E2E).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,30 @@
# Task 04 — API, frontend, and E2E pins for the discard + title
**Phase:** `115_doc_draft_discard` · **Source:** `TODO.md:207–209` — "Acceptance: an orphaned draft can be discarded from the edit screen; the draft row is gone afterward (API test); title of a save-as-doc after a retry redo matches the redone answer's own question."
## Objective
The acceptance is pinned at all three layers: the API (the row is gone), the frontend (the control + handler + the pairing), and the E2E (the full discard flow + the title after a retry redo).
## Work
1. `tests/integration/test_doc_drafts_api.py` (existing — extend):
- DELETE an existing draft → 204; the subsequent GET → 404.
- DELETE an unknown token → 404.
- the admin gate: the same assertions the sibling routes use (mirror the existing test's auth fixtures — anonymous → 401 / non-admin → 403 per the router's `require_admin`).
2. `tests/unit/test_frontend_doc_draft_discard.py` (new, house source-assertion style):
- the doc-edit template carries `#discard-draft`; `doc-edit.js` calls `confirm(...)` before the `DELETE /api/doc-drafts/` fetch; 204 → the redirect; non-204 → the inline error, no navigation.
- `app.js`: `defaultDocTitle` takes a wrap arg and prefers the paired user bubble; the `saveAsDoc` call site passes the bubble ancestor.
3. `tests/e2e/test_save_doc_session.py` (existing — extend):
- **the discard flow:** ask (mock LLM) → save as doc → the edit screen → click Discard → confirm → back on the chat page; an API check (test client) confirms the draft row is gone (GET 404).
- **the title after a retry:** ask → the answer → Retry (redo-in-place) → save as doc on the redone answer → the edit screen's title field value == the redone question (not an unrelated trailing question).
4. Run in isolation: `uv run pytest tests/e2e/test_save_doc_session.py -v --no-cov` (DB up).
5. Regression: `tests/integration/test_doc_drafts_api.py` (all), the existing `tests/e2e/test_save_doc_session.py` tests green.
## Testing & Quality
- Integration: the DELETE contract (the acceptance: "the draft row is gone afterward (API test)").
- Unit: the frontend pins.
- E2E: the flow + the title (the acceptance).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] All the acceptance pins green (API row gone; title after a retry).
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,39 @@
# Phase 116 — Document modal: themed code-block scrollbar (TODO L3)
**Source:** `TODO.md` L88–99 — "L3 — Document modal: native unstyled horizontal scrollbar in the code block (cosmetic) (2026-09-15, brain-of-reese interactive test)"
**Story:** `document-viewer.md` — the modal belongs to the document-viewer story (the same-page chip viewer, phase 26).
**Context:** The document modal (the almost-fullscreen chip viewer, `.doc-modal` — `frontend/assets/styles.css:4117+`) shows the raw content in a code block; long lines (e.g. a `quest::say(...)` line in a quest `.pl` file) overflow horizontally and reveal the **browser-native, unstyled scrollbar** (light-gray bar) — it clashes with the dark theme, and long lines clip at the right edge.
## Objective
The modal's code block scrolls horizontally with a themed scrollbar (`scrollbar-color` for Firefox, `::-webkit-scrollbar` pseudos for Chromium), consistent with the dark theme; long lines stay unwrapped (code stays code) and scroll instead of clipping.
## Dependencies
- `115_doc_draft_discard` (todo) — pipeline predecessor (execution order) only; no code dependency (CSS + one E2E file).
## Design (shared by all tasks — the executor reads this, not the chat)
- **Theme tokens (task 01):** two CSS custom properties in the theme's token block: `--scrollbar-thumb` (a muted theme color, ≥3:1 against the track — a scrollbar is a UI component, non-text contrast AA) and `--scrollbar-track` (near the code-block background). Scope: the modal's raw-content code element **only** (confirm the exact selector from `frontend/assets/document-modal.js` / the `.doc-modal` rules) — no global scrollbar restyle (out of scope; the TODO asks only about the modal).
- **Rules (task 01):** on the modal code block: `overflow-x: auto` (confirm present — add if the element relies on an ancestor), `scrollbar-width: thin`, `scrollbar-color: var(--scrollbar-thumb) var(--scrollbar-track)` (Firefox), and the Chromium pair: `::-webkit-scrollbar { height: 8px }`, `::-webkit-scrollbar-track { background: var(--scrollbar-track) }`, `::-webkit-scrollbar-thumb { background: var(--scrollbar-thumb); border-radius: 4px }`.
- **E2E (task 02):** the document-viewer story gains a check: open the modal on a document with a long line → the code block is horizontally scrollable (`scrollWidth > clientWidth`; a scroll action moves it) → a screenshot to `.agents/screenshots/` (house convention) as the visual record.
## Tasks
1. `01_themed_scrollbar.md` — the theme tokens + the scoped scrollbar rules.
2. `02_scrollbar_e2e_check.md` — the E2E scroll check + screenshot.
## Testing & Quality
- E2E: `tests/e2e/test_document_viewer.py` (existing — extend, task 02); run in isolation: `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov`.
- CSS: no unit layer for CSS — the E2E check + screenshot are the gate; the dark-theme suite (`tests/e2e/test_dark_tech_theme.py`) must stay green (no token collision).
- Coverage: n/a (no app/ code change) — the validate.sh gate stays green.
## Completion Criteria
- [ ] The modal's code block scrolls horizontally; the scrollbar is themed — the screenshot in `.agents/screenshots/` shows no native light-gray bar.
- [ ] No global scrollbar change (the new selector is scoped under `.doc-modal` — grep); the other pages are visually unchanged (regression suites green).
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — keep horizontal scroll (NO line-wrap) for the code content (owner-confirmed 2026-09-14, roadmap confirmation — the TODO's "consider wrapping long lines per content type" is rejected for code: wrapping breaks code readability; the themed scrollbar is the fix).**
## Commit
```bash
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "style(ui): theme the document modal's code-block horizontal scrollbar (scrollbar-color + webkit pseudos)"
```
@@ -0,0 +1,23 @@
# Task 01 — Themed scrollbar for the modal code block
**Phase:** `116_modal_scrollbar_theme` · **Source:** `TODO.md:90–96` — "In the document modal (the almost-fullscreen chip viewer), the raw-content code block overflows horizontally and reveals the **browser-native, unstyled scrollbar** (light-gray bar) — it clashes with the dark theme, and long lines clip at the right edge (observed on a quest `.pl` file whose `quest::say(...)` line exceeds the modal width). Style the scrollbar to match the theme (`scrollbar-color` for Firefox, `::-webkit-scrollbar` pseudos for Chromium), or consider wrapping long lines per content type."
## Objective
The modal's raw-content code block gets a themed horizontal scrollbar (both engine families), scoped to the modal.
## Work
1. `frontend/assets/styles.css` — find the modal code block's exact selector (the `.doc-modal` raw-content `<pre>`/code element — how `frontend/assets/document-modal.js` renders the content; the `.doc-modal-panel` rules start ~L4135):
- ensure `overflow-x: auto` on the scrolling element (add it if the element relies on an ancestor for the overflow);
- add the scoped rules: `scrollbar-width: thin`, `scrollbar-color: var(--scrollbar-thumb) var(--scrollbar-track)`, and `::-webkit-scrollbar { height: 8px }` / `::-webkit-scrollbar-track { background: var(--scrollbar-track) }` / `::-webkit-scrollbar-thumb { background: var(--scrollbar-thumb); border-radius: 4px }`.
2. `frontend/assets/styles.css` — define `--scrollbar-thumb` / `--scrollbar-track` in the theme's token block (derive from existing theme colors — the thumb must be visibly distinct from the track, ≥3:1 non-text contrast).
3. Do NOT restyle scrollbars elsewhere — the new selector stays scoped under `.doc-modal` (the TODO's scope).
4. ASSUMPTION: no line-wrap (locked A1, phase level) — the code stays unwrapped and scrolls.
## Testing & Quality
- E2E: the scroll check (task 02).
- Coverage: n/a (CSS) — the suite stays green.
## Completion Criteria
- [ ] The modal code block's horizontal scrollbar is themed for Chromium (webkit pseudos) and Firefox (`scrollbar-color`) — the task-02 screenshot shows no native light-gray bar.
- [ ] No other page's scrollbar changes (the new selector is scoped under `.doc-modal`).
- [ ] `uv run pytest` green (in particular `tests/e2e/test_dark_tech_theme.py`).
@@ -0,0 +1,24 @@
# Task 02 — E2E: the modal scrollbar check + screenshot
**Phase:** `116_modal_scrollbar_theme` · **Source:** `TODO.md:98–99` — "Acceptance: the modal's code block scrolls horizontally with a themed scrollbar; screenshot check in the document-viewer E2E story."
## Objective
The document-viewer E2E story asserts the modal code block scrolls horizontally and records a screenshot of the themed scrollbar.
## Work
1. `tests/e2e/test_document_viewer.py` (existing — extend) — a test:
- open the document modal on a document with a line longer than the modal width (check the fixture KB's contents first; if no fixture doc qualifies, pick the longest existing line and assert `scrollWidth > clientWidth` against it — note the choice in the test docstring);
- the code block element: `scrollWidth > clientWidth` (the overflow is real);
- a horizontal scroll action (`scrollLeft` / `scrollBy`) actually moves it (the scroll works, no clipping);
- the themed rule is in effect: `getComputedStyle` exposes `scrollbar-color` (the webkit pseudos are not exposed via computed style — that half is covered by the screenshot);
- a screenshot to `.agents/screenshots/` (house convention — check the exact path other screenshot tests use).
2. Run in isolation: `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` (DB up).
3. Regression: the existing `test_document_viewer.py` tests stay green.
## Testing & Quality
- E2E: the scroll behavior + the screenshot (the acceptance).
- Coverage: n/a.
## Completion Criteria
- [ ] The new E2E test green in isolation; the screenshot shows the themed scrollbar (no native light-gray bar).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,90 @@
# Phase 117 — Compact, well-wrapped tool-call lines
**Source:** owner visual-glitch report (live chat, 2026-09-15) — "how much space the tool calls take up, and the tool call text is spit and wrapped poorly."
**Story:** n/a (owner bug report — mobile viewport, `https://brain.reeseapps.com`, reproduced 2026-09-15)
**Context:** `frontend/assets/app.js` (`appendToolLine` — renders one `.tool-call` line per `tool` SSE frame into a `.tool-calls` list above the answer bubble; `ensureThinkingBlock` — places the Thinking `<details>` above the `.tool-calls` list via `anchor = body.querySelector(".tool-calls") ?? body.querySelector(".bubble")`; the delta/done/stop handlers each call `closeThinkingBlock(wrap)`; `renderStoredMessage` re-renders persisted tools through the SAME `appendToolLine`); `frontend/assets/shared.js` (`addToolLines` — the shared page's local copy of the same renderer, pinned byte-parity with `appendToolLine`); `frontend/assets/styles.css` (`.tool-calls` = flex column + gap; `.tool-call` = `display:flex; align-items:baseline` + a full card: `background` + `border` + `border-left:3px var(--accent-line)` + `border-radius` + `padding`; `.tool-call code` = a chip: `var(--brand-soft)` background + padding + radius on top of `var(--mono)`/`var(--ink)`); `tests/unit/test_frontend_tool_states.py` (pins the exact tool-line template literals + the `.tool-calls`/`.tool-call` DOM shape + the `.tool-call` CSS — `test_tool_call_style_is_accent_and_contrast_safe` asserts `display: flex` + `var(--accent-line)` on `.tool-call`); `tests/unit/test_big_read_progress.py` (pins phase 87's `.tool-elapsed` clock, which queries `.tool-calls .tool-call:last-child` and appends a SIBLING suffix — the line's own text/literals stay byte-identical); `tests/e2e/test_agent_document_tools.py` + `tests/e2e/test_big_read_progress.py` (the tool-line E2E — the former asserts `.tool-call` count + `to_contain_text` after a completed turn, the latter asserts the FIRST `.tool-call` line and the `.tool-elapsed` suffix are `to_be_visible` DURING the live frameless gap, before the answer's delta).
## Bug basis (code-traced + reproduced live, 2026-09-15)
Reproduced at 390×844 (mobile) on the live site. One answer to "Generate a change log … last 5 phases" rendered **6+ stacked full-width cards** — one per `tool` frame — each a complete bordered card (accent left border + surface background + radius) holding a mono path *chip* inside it. Two distinct defects:
- **Space:** each `tool` frame appends a full-width bordered `.tool-call` card spanning the whole chat column. The agent loop is round-capped and fires several calls per turn (`ls → read → read …`), so a single answer stacks N cards above it. On a phone the tool process visually swamps the answer.
- **Wrapping:** `.tool-call` is `display:flex; align-items:baseline`, so the label text node (`"📄 Reading "`) and the `<code>` path are **two separate flex items**. The long mono path squeezes the label flex-item, and `overflow-wrap: anywhere` breaks the label **mid-word** (`Reading` → `Rea` / `ding`); the path wraps to 3 lines indented to the right of that narrow broken label. The "spit and wrapped poorly" look.
- **The live-feedback that must be preserved:** the "calling tool" live state lives in `#send-status` (aria-live) + the typing-indicator aria-label (pinned by `test_calling_tool_label_strings`), NOT the tool lines — so folding the tool lines does not remove any live feedback. Phase 87's ticking `.tool-elapsed` suffix, however, IS on a `.tool-call` line and its E2E asserts it is **visible during the live gap** — so the lines must stay visible while the turn is in flight and fold only once the answer begins.
## Objective
Turn the per-call tool "cards" into a single collapsible disclosure — one compact "Tool calls (N)" line by default, expandable to the individual calls — and restyle the individual lines as deboxed, inline-flowing text so the path wraps to the left edge like a normal sentence and the label never breaks mid-word. Collapsed by default for completed/restored turns (the space fix), open during a live turn (keeps phase 87's live suffix visible), with the "calling tool" live state unchanged.
## Owner decisions (chat, 2026-09-15 — recorded per AGENTS.md rule 3)
- **D1 — Frontend-only.** No `app/` change, no new SSE frame, no persistence-format change. The stored record stays `{name, argument}` (+ `truncated`); only the *rendering* changes. The server and the SSE event set are byte-identical.
- **D2 — Reuse the Thinking-block convention.** The tool calls become a native `<details>/<summary>` disclosure in the same `.msg-body` wrap, mirroring `details.thinking` (open-while-active, closed-when-done). No new component, no new JS dependency.
- **D3 — Open live, folded at rest.** The disclosure is created **open** on the first live `tool` frame (the calls + phase 87's `.tool-elapsed` suffix stay visible during the turn); it folds when the answer's first `delta` arrives, on `done`, and on stop/abort (`closeToolCalls`, idempotent — the exact sites that call `closeThinkingBlock`). The restore path (`renderStoredMessage`) and the shared page (`addToolLines`) render it **closed**. This is what makes the user's screenshot (a completed turn) collapse to one line while keeping the live behavior green.
- **D4 — Debox + inline flow.** The `.tool-call` line loses its card (no `display:flex` / background / border / left border / radius / padding) so the label + inline `<code>` flow as one continuous run (fixes the mid-word label break + indented wrap); the `<code>` loses its chip background but keeps `var(--mono)` + `var(--ink)`. The accent is carried by the line's `color` (`var(--accent-ink)`), not a border.
- **D5 — Plain-text summary, no emoji.** The summary reads `Tool call (1)` / `Tool calls (N)` — plain text, matching the emoji-free chrome (the Thinking summary is just "Thinking") and the phase-08 emoji-free-chrome lean. The existing `📄`/`🔎` line glyphs stay (they are pinned literals + the emoji-guard strip set).
- **D6 — Keep the pinned literals + DOM shape.** The four `line.textContent = "…"` label literals, the `.tool-calls` list (role=list, aria-label "Tool calls"), the `.tool-call` listitems, and the `<code>` `textContent` arguments stay **byte-identical** — so `test_frontend_tool_states.py` (all but the one CSS assert), `test_big_read_progress.py`, the emoji guard, and phase 87's `.tool-calls .tool-call:last-child` query all stay green unchanged.
## Design (shared by all tasks — the executor reads this, not the chat)
### `frontend/assets/app.js`
- **`appendToolLine(wrap, name, argument)`** — wrap the list in a disclosure; keep everything else byte-identical:
- `let container = body.querySelector(".tool-calls-disclosure");` (idempotent per wrap — was `.tool-calls`).
- On first frame: `container = document.createElement("details"); container.className = "tool-calls-disclosure"; container.open = true;` then a `<summary class="tool-calls-summary">` (createElement, textContent only) appended first, then the existing list — `const listEl = document.createElement("div"); listEl.className = "tool-calls"; listEl.setAttribute("role","list"); listEl.setAttribute("aria-label","Tool calls");` — appended second; then `body.insertBefore(container, body.querySelector(".bubble"))` (the disclosure goes where the list did: above the answer, below an existing Thinking block).
- `const list = container.querySelector(".tool-calls");` — build the line EXACTLY as today (`line.className="tool-call"`, `role="listitem"`, the four pinned `line.textContent` label literals, the `<code>` with `code.textContent = argument`), `list.appendChild(line)`.
- Update the summary count on every append (live + restore): `const n = list.children.length; container.querySelector("summary").textContent = \`Tool call${n === 1 ? "" : "s"} (${n})\`;`.
- **No `innerHTML` anywhere in the function** (house rule; pinned). The function stays flat (no nested function declaration) so the `js[fn : js.find("\n}\n", fn)]` body-extraction pins keep working.
- **`closeToolCalls(wrap)`** (new, next to `closeThinkingBlock`): `const disc = wrap?.querySelector?.(".tool-calls-disclosure"); if (disc) disc.open = false;` — idempotent, no-op without a disclosure. Called at **every** existing `closeThinkingBlock(wrap)` site: the `delta` handler (≈L2514), the `done` handler (≈L2519), and the stop/abort settle (≈L2646).
- **`renderStoredMessage`** — after the existing `if (Array.isArray(m.tools)) { …appendToolLine(wrap, t.name, arg)… }` loop, add `closeToolCalls(wrap);` so a restored turn renders folded (the space fix; mirrors the thinking restore rendering collapsed).
- **`ensureThinkingBlock`** — the anchor becomes `body.querySelector(".tool-calls-disclosure") ?? body.querySelector(".tool-calls") ?? body.querySelector(".bubble")` so the Thinking block lands above the WHOLE disclosure (not inside it). `block.open = true` unchanged.
### `frontend/assets/shared.js`
- **`addToolLines(wrap, tools)`** — parity with `appendToolLine`, but **created closed** (pure render, always folded): build the `details.tool-calls-disclosure` (`container.open = false`) + `summary.tool-calls-summary` + the existing `.tool-calls` list (role=list, aria-label) exactly as today; render each line byte-identical (the same four label literals + `<code>` + the phase-95 `truncated-note`); set the summary count once at the end (`Tool call (N)` / `Tool calls (N)`). Keep the `code.textContent = argument` count at 3 and no `innerHTML` (the `test_shared_page_tool_lines_…` / `test_frontend_tool_states.py` shared pins).
### `frontend/assets/styles.css`
- **`.tool-call`** — DEBOX: remove `display:flex`, `align-items:baseline`, `gap`, `background`, `border`, `border-left`, `border-radius`, `padding`. KEEP `color: var(--accent-ink)`, `font-size: 0.8rem`, `line-height: 1.4`, `overflow-wrap: anywhere`. As a flex item of the `.tool-calls` column it stays block-level per line, but its label + inline `<code>` now flow as one continuous run → the path wraps to the left edge and the label no longer breaks mid-word.
- **`.tool-call code`** — DECHIP: remove `background: var(--brand-soft)`, `padding`, `border-radius`. KEEP `font-family: var(--mono)`, `font-size: 0.95em`, `color: var(--ink)` (≈11.5:1, AA).
- **NEW `.tool-calls-disclosure` + `.tool-calls-summary`** — model the disclosure on the existing `details.thinking` styling (house AA palette, no new hue): the summary is a native focusable toggle that carries the house 3px `:focus-visible` ring (the `details.thinking summary` already has it — reuse that language), small status font, `color: var(--accent-ink)` (≈10.4:1 on the surface — same pairing the deboxed line uses). No new color literal (phase-92 zero-literal invariant; B5 text+color, never color alone — the count is text).
- **`.tool-calls`** (the list) — UNCHANGED: stays `display:flex; flex-direction:column; gap: 0.25rem` (spacing between the now-deboxed lines).
- **`.tool-elapsed` / `.truncated-note`** — UNCHANGED (phase 87 / phase 95).
### `tests/unit/test_tool_call_compact.py` (NEW — source-level house pattern)
- **app.js:** `appendToolLine` body contains `document.createElement("details")` + `className = "tool-calls-disclosure"` + `document.createElement("summary")`; `container.open = true` (the live default, D3); the count literal `Tool call${n === 1 ? "" : "s"} (${n})` (or the equivalent template) is built with `textContent` (no `innerHTML`); `closeToolCalls` is defined and called at **3** handler sites (delta/done/stop) + once in `renderStoredMessage` (pin: `closeToolCalls(wrap)` appears ≥4×); `ensureThinkingBlock`'s anchor includes `.tool-calls-disclosure`; the four pinned `line.textContent = "…"` label literals are STILL present (mirror the guard); `innerHTML` is NOT in the `appendToolLine` body.
- **shared.js:** `addToolLines` body contains `document.createElement("details")` + `className = "tool-calls-disclosure"` + `document.createElement("summary")` + `open = false` (closed on the shared page, D3); the four label literals are still present; `body.count("code.textContent = argument") == 3`; no `innerHTML`.
- **styles.css:** the `.tool-call` rule has **NO** `display: flex` and **NO** `var(--accent-line)` (deboxed, D4) but still has `var(--accent-ink)`; the `.tool-call code` rule still has `var(--mono)` + `var(--ink)` and has **NO** `background`; `.tool-calls-disclosure` and `.tool-calls-summary` rules exist.
### `tests/unit/test_frontend_tool_states.py` (UPDATE — the only existing test that changes)
- **`test_tool_call_style_is_accent_and_contrast_safe`** — the `.tool-call` rule is deboxed: REMOVE the `assert "display: flex" in row` and `assert "var(--accent-line)" in row` lines; KEEP `assert "var(--accent-ink)" in row` (the accent is now the line's color) and the `.tool-call code` `var(--mono)` + `var(--ink)` asserts. Update the docstring to describe the deboxed, inline-flow line (phase 117). Every OTHER test in this file stays green unchanged (the DOM-shape, literal, branch-order, persist, restore, shared-parity, and no-CDN pins are all preserved by D6).
### `tests/e2e/test_tool_call_compact.py` (NEW story suite — mock "use your tools" flow, 3 tool calls)
The conftest `app_server`/`page` fixtures + the phase-37 admin login + the marker question that drives the mock's `ls → ls(scoped) → read` flow (mirror `test_agent_document_tools.py`). Four tests:
1. **Folds at rest:** after the turn completes, the `.tool-calls-summary` is visible with text `Tool calls (3)`; the disclosure is **not** open (`.tool-calls-disclosure` has no `[open]`); the `.tool-call` lines are present in the DOM (count 3) but hidden; the answer bubble is present.
2. **Expands on tap:** clicking the summary opens the disclosure; the three `.tool-call` lines become visible with the correct text (nth 0 "Listing documents", nth 1 "Listing documents in", nth 2 "Reading ").
3. **Deboxed inline flow:** on a visible (expanded) `.tool-call` line, `getComputedStyle(line).display !== "flex"` (the label + path are one inline run, not two flex items) and the line `to_contain_text("Reading ")` (label immediately followed by the path in the same run).
4. **Restored folded:** RELOAD (same context — the phase-14/50 persisted conversation restores); the restored brain message's `.tool-calls-disclosure` is present, closed, summary `Tool calls (3)`, `.tool-call` count 3 in the DOM — no auto-expand on load.
## Dependencies
- `95_read_truncation_cap`, `87_big_read_progress`, `70_harness_aligned_tools`, `37` (the tool-line rendering this restyles — complete). NO code dependency beyond the shared frontend files; the only pipeline predecessor is execution order (todo/ is empty — this is the next phase).
## Tasks
1. `01_disclosure_wrapper.md` — the `<details>/<summary>` wrapper + count + open-live/close-at-rest + restore-closed + shared.js parity (app.js + shared.js + the new unit module's JS pins).
2. `02_debox_and_inline_flow.md` — the CSS debox/dechip + disclosure/summary rules, and the one `test_frontend_tool_states.py` CSS-pin update (styles.css + unit CSS pins + the existing CSS test).
3. `03_e2e_story_suite.md` — `tests/e2e/test_tool_call_compact.py` (fold / expand / deboxed-flow / restored-folded).
4. `04_verify_and_commit.md` — full gate (unit + integration, coverage >90%, the new E2E story in isolation, `test_agent_document_tools.py` + `test_big_read_progress.py` + `test_thinking_display.py` + smoke in isolation, ruff + pyright) + atomic commit + move to complete/.
## Testing & Quality
- Unit — `tests/unit/test_tool_call_compact.py` (new): the JS/CSS pins listed above (disclosure + summary + count + open-live, close-at-4-sites, restore-closed, shared parity, deboxed CSS, disclosure/summary rules, the pinned literals still present, no innerHTML).
- Existing unit suites MUST stay green: `tests/unit/test_big_read_progress.py` (phase-87 clock + the byte-identical tool-line literals), `tests/unit/test_shared_page.py` (shared-page tool-line parity), and `tests/unit/test_frontend_tool_states.py` (all tests except the ONE deboxed CSS assert updated in task 02).
- E2E — `tests/e2e/test_tool_call_compact.py` (new; isolation gate per AGENTS.md rule 9): the four tests above, mock LLM (no slow proxy needed — the fold is about a completed turn).
- Regression E2E (run in isolation by task 04): `test_agent_document_tools.py` (tool-line count + text after a completed turn — the lines stay in the DOM inside the folded disclosure), `test_big_read_progress.py` (the FIRST `.tool-call` line + the `.tool-elapsed` suffix are visible DURING the live gap — the disclosure is still open then), `test_thinking_display.py` (the Thinking block ordering vs. the disclosure), `test_smoke.py`.
- Coverage: **>90%** on `app/` — no `app/` code changes (the floor is held by the untouched suite).
## Completion Criteria
- [ ] A completed tool turn renders as ONE collapsed "Tool calls (N)" line (E2E pin 1); tapping it reveals the individual calls (E2E pin 2); the restored/shared view is folded on load (E2E pin 4). The 6+ stacked cards from the screenshot are gone.
- [ ] An expanded `.tool-call` line is deboxed inline-flow text (E2E pin 3 + the CSS unit pins): the path wraps to the left edge and the label never breaks mid-word — the "spit and wrapped poorly" glitch is fixed at 390×844 and at desktop.
- [ ] Live behavior unchanged: during a live frameless gap the calls + phase-87 `.tool-elapsed` suffix stay visible (the disclosure is open until the first delta) — `test_big_read_progress.py` green in isolation; the "calling tool" `#send-status`/aria state is byte-identical (`test_calling_tool_label_strings` green).
- [ ] `uv run pytest` green (including `test_big_read_progress.py`, `test_shared_page.py`, and the updated `test_frontend_tool_states.py`); `uv run pytest --cov=app --cov-report=term-missing` >90%; the new E2E story + `test_agent_document_tools.py` + `test_big_read_progress.py` + `test_thinking_display.py` + `test_smoke.py` green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] `git diff --stat` limited to `frontend/assets/app.js`, `frontend/assets/shared.js`, `frontend/assets/styles.css`, `tests/unit/test_tool_call_compact.py`, `tests/unit/test_frontend_tool_states.py`, `tests/e2e/test_tool_call_compact.py`, and the phase files — nothing in `app/`.
- [ ] One atomic `--no-gpg-sign` commit (e.g. `feat(ui): fold tool calls into a compact collapsible disclosure`); phase dir moved to `.agents/phases/complete/`.
## Locked decisions
- **Frontend-only (D1)** — `app/`, the SSE event set, and the persistence format are byte-identical; only the rendering changes.
- **Reuse the native `<details>` Thinking-block convention (D2/D3)** — open while the turn is live, folded at rest and on restore; no new component or dependency. This is what keeps phase 87's live-suffix E2E and the tool-line count/text E2E green.
- **Pinned literals + DOM shape stay (D6)** — the four `line.textContent` labels, the `.tool-calls` list, the `.tool-call` listitems, and the `<code>` `textContent` arguments are byte-identical; the emoji guard and phase 87's `.tool-calls .tool-call:last-child` query are unaffected. Only the ONE CSS assert in `test_frontend_tool_states.py` changes (the debox).
- **Debox, don't restyle the accent away (D4)** — the line keeps `var(--accent-ink)` as its text color (≈10.4:1 AA on the surface); the accent moves off the border onto the text, so the line still reads as distinct from the brand-ink Thinking summary.
@@ -0,0 +1,63 @@
# Task 01 — The `<details>/<summary>` disclosure wrapper + count + open-live/close-at-rest
**Phase:** `117_tool_call_compact` · **Source:** owner visual-glitch report (2026-09-15) — tool-call cards take too much space and wrap poorly.
**Story:** n/a (owner bug report)
## Objective
Wrap the existing `.tool-calls` list in a native `details.tool-calls-disclosure` with a `summary.tool-calls-summary` that carries a `Tool call (N)` / `Tool calls (N)` count. The disclosure is created **open** on the first live `tool` frame, **folds** when the answer starts / the turn ends / is stopped (`closeToolCalls`), and renders **closed** on the restore path and the shared page. The `.tool-calls` list, the `.tool-call` lines, the four label literals, and the `<code>` arguments stay byte-identical (D6) — only the wrapper is added.
## Work
1. `frontend/assets/app.js` — `appendToolLine` (house comment style, citing phase 117):
- `let container = body.querySelector(".tool-calls-disclosure");` (was `.tool-calls`).
- First-frame branch (the `if (!container)` block): create the disclosure + summary + the existing list, in this order:
```js
container = document.createElement("details");
container.className = "tool-calls-disclosure";
container.open = true; // D3: open while the turn is live; closeToolCalls folds it
const summary = document.createElement("summary");
summary.className = "tool-calls-summary";
container.appendChild(summary);
const listEl = document.createElement("div");
listEl.className = "tool-calls";
listEl.setAttribute("role", "list");
listEl.setAttribute("aria-label", "Tool calls");
container.appendChild(listEl);
body.insertBefore(container, body.querySelector(".bubble"));
```
- After the `if`: `const list = container.querySelector(".tool-calls");` then build the line EXACTLY as today (`line.className="tool-call"`, `role="listitem"`, the four pinned `line.textContent` label literals, the `<code>` with `code.textContent = argument`) and `list.appendChild(line)`.
- Summary count on every append (live + restore): `const n = list.children.length; container.querySelector("summary").textContent = \`Tool call${n === 1 ? "" : "s"} (${n})\`;`.
- **No `innerHTML`** anywhere in the function; keep it flat (no nested function declaration) so the `js[fn : js.find("\n}\n", fn)]` body-extraction pins keep working.
2. `frontend/assets/app.js` — `closeToolCalls(wrap)` (new, defined next to `closeThinkingBlock`):
```js
function closeToolCalls(wrap) {
const disc = wrap?.querySelector?.(".tool-calls-disclosure");
if (disc) disc.open = false; // idempotent; no-op without a disclosure
}
```
Add `closeToolCalls(wrap);` on the line next to each existing `closeThinkingBlock(wrap);` — the `delta` handler (≈L2514), the `done` handler (≈L2519), and the stop/abort settle (≈L2646). Comment each: the answer began / the turn ended / the turn was stopped — fold the record (mirrors the Thinking block settling closed).
3. `frontend/assets/app.js` — `renderStoredMessage`: immediately after the existing `if (Array.isArray(m.tools)) { …appendToolLine(wrap, t.name, arg)… }` loop, add `closeToolCalls(wrap);` (a restored turn renders folded — the space fix; mirrors the thinking restore rendering collapsed).
4. `frontend/assets/app.js` — `ensureThinkingBlock`: change the anchor to
`const anchor = body.querySelector(".tool-calls-disclosure") ?? body.querySelector(".tool-calls") ?? body.querySelector(".bubble");`
so the Thinking block lands above the whole disclosure. `block.open = true` and everything else unchanged.
5. `frontend/assets/shared.js` — `addToolLines` (parity, but **created closed** — pure render, always folded):
- Build the same `details.tool-calls-disclosure` with `container.open = false`, a `summary.tool-calls-summary` (first child), and the existing `.tool-calls` list (role=list, aria-label "Tool calls") — appended to the same `.msg-body` anchor as today.
- Render each line byte-identical (the same four label literals + `<code>` + the phase-95 `truncated-note`), appending to the list.
- Set the summary count once at the end: `const n = list.children.length; summary.textContent = \`Tool call${n === 1 ? "" : "s"} (${n})\`;`.
- Keep `body.count("code.textContent = argument") == 3` and no `innerHTML` (the shared-parity pins).
6. `tests/unit/test_tool_call_compact.py` (NEW module) — the source-level JS pins (house pattern, `_js()` / `_shared_js()` readers like `test_frontend_tool_states.py`):
- **app.js `appendToolLine`:** body contains `document.createElement("details")`, `className = "tool-calls-disclosure"`, `document.createElement("summary")`; `container.open = true`; the count template `Tool call${n === 1 ? "" : "s"} (${n})` (assert the template fragment, not the rendered value); `innerHTML` NOT in the body; the four pinned `line.textContent = "…"` label literals are still present.
- **app.js `closeToolCalls`:** defined (`function closeToolCalls`); `closeToolCalls(wrap)` appears **≥4×** (the 3 handler sites + `renderStoredMessage`).
- **app.js `ensureThinkingBlock`:** the body contains `.tool-calls-disclosure` (the new anchor term) AND still contains `querySelector(".tool-calls")` + `querySelector(".bubble")` + `block.open = true`.
- **shared.js `addToolLines`:** body contains `document.createElement("details")`, `className = "tool-calls-disclosure"`, `document.createElement("summary")`, `open = false` (closed on the shared page); the four label literals present; `body.count("code.textContent = argument") == 3`; no `innerHTML`.
## Testing & Quality
- Unit: `uv run pytest tests/unit/test_tool_call_compact.py -v` green (all JS pins). `uv run pytest tests/unit/test_frontend_tool_states.py tests/unit/test_big_read_progress.py tests/unit/test_shared_page.py -v` green **unchanged** (the CSS pin is task 02; the JS/DOM pins are preserved by D6).
- Quick live sanity (session log): dev server, trigger a tool turn — during the turn the disclosure is open (calls + any `.tool-elapsed` suffix visible); when the answer starts it folds to "Tool calls (N)"; a reload shows it folded. (The deterministic E2E is task 03 — this is a wiring smoke only.)
- Coverage: **>90%** on `app/` unaffected (no `app/` change).
## Completion Criteria
- [ ] `git diff frontend/assets/app.js` shows: the disclosure/summary/list creation in `appendToolLine`, the summary count line, the new `closeToolCalls`, the three handler call sites + the `renderStoredMessage` call, and the `ensureThinkingBlock` anchor — and NOTHING else; the four `line.textContent` label literals and the `<code>` `textContent` bytes byte-identical.
- [ ] `git diff frontend/assets/shared.js` shows the parallel closed-disclosure build + count + the unchanged line/literal bytes.
- [ ] `uv run pytest tests/unit/test_tool_call_compact.py -v` green; `tests/unit/test_frontend_tool_states.py` + `tests/unit/test_big_read_progress.py` + `tests/unit/test_shared_page.py` green unchanged (CSS test still asserts the OLD flex/border — that flips in task 02, so run it here to confirm only that one assert is the pending delta).
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] No behavior change in completed work (the fold/expand proof is task 03; the live-suffix regression is task 04).
@@ -0,0 +1,53 @@
# Task 02 — Debox + inline-flow CSS, and the one CSS-pin update
**Phase:** `117_tool_call_compact` · **Source:** owner visual-glitch report (2026-09-15) — the tool-call "cards" are too heavy and the label breaks mid-word.
**Story:** n/a (owner bug report)
## Objective
Debox the `.tool-call` line (no card, no flex) so the label + inline `<code>` flow as one continuous run (fixes the `Rea`/`ding` mid-word break and the indented 3-line wrap), dechip the `.tool-call code` (no background chip, keep mono + ink), and add the `.tool-calls-disclosure` / `.tool-calls-summary` rules modeled on the Thinking disclosure. Update the ONE existing CSS assert in `test_frontend_tool_states.py` to the deboxed contract, and add the CSS pins to the new unit module.
## Work
1. `frontend/assets/styles.css` — **`.tool-call`** (the debox, D4). Remove `display: flex;`, `align-items: baseline;`, `gap: 0.45rem;`, `background: var(--surface);`, `border: 1px solid var(--line);`, `border-left: 3px solid var(--accent-line);`, `border-radius: var(--radius-sm);`, `padding: 0.3rem 0.75rem;`. KEEP `color: var(--accent-ink);`, `font-size: 0.8rem;`, `line-height: 1.4;`, `overflow-wrap: anywhere;`. Update the rule's comment: phase 117 deboxed the line — the accent now rides the text color (≈10.4:1 on the surface) instead of a border; the label + inline `code` flow as one run so the path wraps to the left edge (no more mid-word label break).
2. `frontend/assets/styles.css` — **`.tool-call code`** (the dechip). Remove `background: var(--brand-soft);`, `padding: 0.05em 0.35em;`, `border-radius: 5px;`. KEEP `font-family: var(--mono);`, `font-size: 0.95em;`, `color: var(--ink);`, `overflow-wrap: anywhere;`. Comment: the path keeps the mono + ink treatment (≈11.5:1 AA) but loses the chip background — it is inline text in the line's run now.
3. `frontend/assets/styles.css` — **NEW `.tool-calls-disclosure` + `.tool-calls-summary`** rules, placed next to the `details.thinking` rules and modeled on them (house AA palette, no new hue — the phase-92 zero-literal invariant):
```css
/* Phase 117: the tool-call record folds into a single disclosure line,
reusing the Thinking block's native <details> convention. Open while
the turn is live (phase 87's elapsed suffix stays visible), folded at
rest / on restore. The summary is a native focusable toggle — it
inherits the house 3px :focus-visible ring (see details.thinking
summary). Accent rides the TEXT (var(--accent-ink), ≈10.4:1 on the
surface) — text + color, never color alone (B5). No new literal. */
.tool-calls-disclosure { margin: 0.25rem 0; }
.tool-calls-summary {
cursor: pointer;
color: var(--accent-ink);
font-size: 0.8rem;
line-height: 1.4;
padding: 0.15rem 0;
list-style: none; /* the native marker is redundant with the count text */
}
.tool-calls-summary::-webkit-details-marker { display: none; }
```
(If the `details.thinking summary` uses a chevron/marker rather than the native one, match that treatment so the two disclosures read as one family — the count text is the accessible label either way.)
4. `frontend/assets/styles.css` — **`.tool-calls`** (the list) — UNCHANGED: keep `display: flex; flex-direction: column; gap: 0.25rem;` (spacing between the now-deboxed lines). **`.tool-elapsed`** and **`.tool-call .truncated-note`** — UNCHANGED (phase 87 / phase 95).
5. `tests/unit/test_tool_call_compact.py` (extend the module from task 01) — the CSS pins:
- the `.tool-call` rule (regex `\.tool-call \{([^}]*)\}`) has **NO** `display: flex` and **NO** `var(--accent-line)` (deboxed), but still has `var(--accent-ink)`;
- the `.tool-call code` rule still has `var(--mono)` + `var(--ink)` and has **NO** `background`;
- `.tool-calls-disclosure` and `.tool-calls-summary` rules exist (the summary carries `var(--accent-ink)`).
6. `tests/unit/test_frontend_tool_states.py` — **`test_tool_call_style_is_accent_and_contrast_safe`** (the ONE existing test that changes, D4/D6):
- REMOVE the two now-false asserts: `assert "display: flex" in row` and `assert "var(--accent-line)" in row`.
- KEEP `assert "var(--accent-ink)" in row` (the accent is now the line's color) and the `.tool-call code` `var(--mono)` + `var(--ink)` asserts and the `.tool-calls` `gap` assert.
- Update the docstring: the line is now a deboxed inline-flow row (phase 117) — the accent rides the text color; the `code` is inline mono text (no chip). Everything else in this file is untouched.
## Testing & Quality
- Unit: `uv run pytest tests/unit/test_tool_call_compact.py -v` green (JS pins from task 01 + the new CSS pins); `uv run pytest tests/unit/test_frontend_tool_states.py -v` green with the updated CSS test; `tests/unit/test_big_read_progress.py` + `tests/unit/test_shared_page.py` green unchanged.
- Quick live sanity (session log): dev server, expand a completed tool turn at 390×844 — the lines are plain text rows (no card border/background), the path wraps to the left edge, and "Reading" never breaks as "Rea/ding". (The deterministic E2E is task 03.)
- Coverage: **>90%** on `app/` unaffected.
## Completion Criteria
- [ ] `git diff frontend/assets/styles.css` shows: the deboxed `.tool-call`, the dechipped `.tool-call code`, the two new disclosure/summary rules — and NOTHING else (`.tool-calls` gap, `.tool-elapsed`, `.truncated-note` unchanged).
- [ ] `git diff tests/unit/test_frontend_tool_states.py` shows ONLY the two removed asserts + the docstring in `test_tool_call_style_is_accent_and_contrast_safe`.
- [ ] `uv run pytest tests/unit/test_tool_call_compact.py tests/unit/test_frontend_tool_states.py tests/unit/test_big_read_progress.py tests/unit/test_shared_page.py -v` all green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] No behavior change in completed work (E2E proof in task 03/04).
@@ -0,0 +1,41 @@
# Task 03 — The E2E story suite (fold / expand / deboxed-flow / restored-folded)
**Phase:** `117_tool_call_compact` · **Source:** owner visual-glitch report (2026-09-15).
**Story:** n/a (owner bug report)
## Objective
Prove the new behavior end-to-end against the deterministic mock: a completed 3-call tool turn folds to one "Tool calls (3)" line, expands on tap to the deboxed inline-flow lines, and renders folded on reload. Reuses the phase-37 admin login + the mock "use your tools" flow (exactly the 3 tool calls `test_agent_document_tools.py` drives: `ls` → `ls(scoped)` → `read`).
## Work
1. `tests/e2e/test_tool_call_compact.py` (NEW module) — header docstring cites phase 117 + the owner report; the conftest `app_url` + `page` fixtures; `from e2e.auth_helpers import login`. Constants:
- `MARKER_QUESTION = "Use your tools: what is the exact JSON shape of reeselink.json for my aws route53 hosted zone?"` (the SAME marker that drives the mock's 3-tool flow — mirror `test_agent_document_tools.py` so the flow is deterministic).
- `SUMMARY = "#messages .msg.brain .tool-calls-summary"` (the disclosure toggle).
- `DISCLOSURE = "#messages .msg.brain .tool-calls-disclosure"`.
- `LINES = "#messages .msg.brain .tool-call"`.
- A `_submit_tools_turn(page, app_url)` helper: `login(page, app_url, next="/")`; `page.fill("#message-input", MARKER_QUESTION)`; `page.click("#send-btn")`; wait for the answer bubble (`MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"`) — i.e. the turn is COMPLETE (the disclosure has already folded on the first delta).
2. **`test_tool_calls_fold_to_one_line_after_turn`** (pin 1): after `_submit_tools_turn`:
- `expect(page.locator(SUMMARY)).to_be_visible()` and `to_contain_text("Tool calls (3)")`.
- the disclosure is NOT open: `page.get_attribute(DISCLOSURE, "open")` is `None` (a closed `<details>` has no `open` attribute).
- the lines are present in the DOM but folded: `expect(page.locator(LINES)).to_have_count(3)` (DOM count) and the FIRST line is NOT visible (`expect(page.locator(LINES).first).not_to_be_visible()`).
- the answer is present: `expect(page.locator(".msg.brain .bubble").last).to_contain_text(MOCK_ANSWER_MARKER)`.
3. **`test_summary_click_expands_the_calls`** (pin 2): after `_submit_tools_turn`:
- `page.locator(SUMMARY).click()` (native `<summary>` — a real focusable toggle, AA).
- `page.get_attribute(DISCLOSURE, "open")` is now `"true"` (or the attribute present).
- `expect(page.locator(LINES)).to_have_count(3)` and every line is now visible; assert the text (mirror `test_agent_document_tools.py`): nth 0 `to_contain_text("Listing documents")`, nth 1 `to_contain_text("Listing documents in")`, nth 2 `to_contain_text("Reading ")`.
4. **`test_expanded_line_is_deboxed_inline_flow`** (pin 3): after expanding (reuse the click from pin 2 in a fresh turn):
- take the visible `.tool-call` "Reading" line; assert `page.evaluate` on its computed style: `getComputedStyle(el).display !== "flex"` (the label + path are one inline run, not two flex items — the debox).
- `expect(line).to_contain_text("Reading ")` (the label is immediately followed by the path in the same run — no separate indented code column).
5. **`test_restored_turn_renders_folded`** (pin 4): after `_submit_tools_turn`:
- `page.reload()` (same context — the phase-14/50 persisted conversation restores).
- wait for the restored brain message's answer; then `expect(page.locator(SUMMARY)).to_be_visible()` + `to_contain_text("Tool calls (3)")`; the disclosure is NOT open (`get_attribute(...) is None`); `expect(page.locator(LINES)).to_have_count(3)` in the DOM; the FIRST line is NOT visible (folded on load — no auto-expand).
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_tool_call_compact.py -v --no-cov` green **in isolation** (DB up: `podman compose up -d db`). Mock LLM by default (no slow proxy — the fold is about a completed turn, not a live gap).
- No `app/` change — the conftest app-server + mock flow are used as-is.
- Coverage: **>90%** on `app/` unaffected.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_tool_call_compact.py -v --no-cov` green in isolation (all four pins).
- [ ] The suite proves: folded-after-turn (pin 1), expand-on-tap (pin 2), deboxed inline-flow (pin 3), restored-folded (pin 4).
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] No behavior change in completed work (the adjacent-suite regression is task 04).
@@ -0,0 +1,33 @@
# Task 04 — Full gate + atomic commit
**Phase:** `117_tool_call_compact` · **Source:** owner visual-glitch report (2026-09-15).
**Story:** n/a (owner bug report)
## Objective
Run the complete phase gate, land the phase as one atomic commit, and move the phase directory to `complete/`.
## Work
1. **Full regression gate** (AGENTS.md rule 9):
- `uv run pytest` — unit + integration green (including `tests/unit/test_tool_call_compact.py`, the updated `tests/unit/test_frontend_tool_states.py`, and the untouched `tests/unit/test_big_read_progress.py` + `tests/unit/test_shared_page.py`).
- `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage **>90%** (no `app/` change this phase — confirm the floor is held).
- `uv run pytest tests/e2e/test_tool_call_compact.py -v --no-cov` — green **in isolation** (this phase's E2E story — fold / expand / deboxed-flow / restored-folded).
- `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov` — green in isolation (the tool-line count + text after a completed turn — the lines stay in the DOM inside the folded disclosure).
- `uv run pytest tests/e2e/test_big_read_progress.py -v --no-cov` — green in isolation (the FIRST `.tool-call` line + the `.tool-elapsed` suffix are visible DURING the live gap — the disclosure is still open then; this is the key live-behavior regression proof for D3).
- `uv run pytest tests/e2e/test_thinking_display.py -v --no-cov` — green in isolation (the Thinking block ordering vs. the new disclosure anchor).
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov` — green in isolation.
- `uv run ruff check . && uv run pyright` — clean.
2. **Manual live check** (keep the output in the session log): the live/dev app at a **mobile viewport (390×844)** — trigger a multi-call tool turn (e.g. "Generate a change log … last 5 phases"). Confirm: the 6+ stacked cards from the original report are now ONE collapsed "Tool calls (N)" line; tapping it reveals deboxed text rows whose paths wrap to the left edge with the label never breaking mid-word (`Reading` stays intact); during a live frameless gap the calls + any "(Ns)" suffix are visible (the disclosure is open); on reload the turn is folded.
3. **Commit** (AGENTS.md rule 8 — one atomic, Conventional-Commits commit, always `--no-gpg-sign`), staging `frontend/assets/app.js`, `frontend/assets/shared.js`, `frontend/assets/styles.css`, `tests/unit/test_tool_call_compact.py`, `tests/unit/test_frontend_tool_states.py`, `tests/e2e/test_tool_call_compact.py`, and the phase files:
`feat(ui): fold tool calls into a compact collapsible disclosure`
— body: owner visual-glitch report (2026-09-15) — each tool call rendered as a full-width bordered card, so a multi-call turn stacked N cards over the answer, and the flex label+code layout broke the label mid-word ("Reading" → "Rea/ding") with the path indented to 3 lines. Two changes, frontend-only: (1) the tool-call record now folds into a single native `<details>` "Tool calls (N)" line reusing the Thinking-block convention — open while the turn is live (phase 87's elapsed suffix stays visible), folded when the answer starts / on done / on stop / on restore, closed by default on the shared page; (2) the individual lines are deboxed (no card, no flex) and dechipped (no code background) so the label + path flow as one inline run that wraps to the left edge. The four pinned line literals, the `.tool-calls` list, the `<code>` textContent arguments, the emoji-guard set, the persistence format, and the server are byte-identical; only the ONE deboxed CSS assert in `test_frontend_tool_states.py` changes. New source-level unit pins + a dedicated fold/expand/debox/restore E2E story.
4. Move the phase directory: `mv .agents/phases/todo/117_tool_call_compact .agents/phases/complete/` and include the move in the same commit.
## Testing & Quality
- This task IS the phase-level gate — the commands above are the completion evidence.
- Coverage: >90% held.
## Completion Criteria
- [ ] All seven gate commands green (unit + integration, coverage >90%, the new E2E story + `test_agent_document_tools.py` + `test_big_read_progress.py` + `test_thinking_display.py` + smoke in isolation, ruff + pyright).
- [ ] The live mobile check shows the stacked cards replaced by one collapsed "Tool calls (N)" line that expands to deboxed inline-flow rows (path wraps to the left edge, label intact), live calls still visible during a gap, and restored turns folded (output kept in the session log).
- [ ] `git show --stat HEAD` lists the staged files above + the phase files (todo → complete move) — nothing else (in particular `app/`, `pyproject.toml` / `uv.lock`, `conftest.py`, `mock_llm.py`, `slow_llm.py` untouched).
- [ ] `.agents/phases/complete/117_tool_call_compact/` exists; `todo/` no longer contains it.
@@ -0,0 +1,58 @@
# Phase 118 — Summary seed context: top-5 "start here" suggestions (TODO L3)
**Source:** `TODO.md` L3 — "I want to change the way the initial chat context works. Right now we feed two entire documents into context before the LLM responds. That was a good idea before we had complex tools and document summaries, but now we have faster ways for the LLM to identify what information it needs without reading 2 files. Moving forward, after the question is embedded, I want the top 5 related documents to be suggested to the LLM for reading, with their summaries loaded into context rather than their entire content — basically a 'start here if these summaries seem right to you' approach. That way the LLM responds faster and doesn't begin by reading massive amounts of data that might be irrelevant because the similarity search had a bad day."
**Story:** n/a (owner-directed RAG behavior change; extends the phase-30 summaries, the phase-37/70/94 agent tools, and the phase-113 tiering assets).
**Context:** A grounded (HIGH) turn currently seeds `build_high_prompt` with the **full text** of the top-2 cited documents (`select_documents_tiered`'s cited tier — floor `BOR_SOURCE_USEFULNESS_FLOOR` 0.35, ceiling `BOR_TOP_N_DOCS` 2, `app/api/chat.py:plan_turn`), and `run_agent` receives them as `seed_docs` whose `read` is refused (`ALREADY_IN_CONTEXT` — the full text is already in the prompt). Summaries exist on `Document.summary` (phase-30 lite digest — currently **non-markdown only**; NULL for markdown docs, pre-30 rows and fail-soft misses) and are embedded as one `is_summary` chunk (position −1) per document. LOCKED A7 (full text of the top-2, never truncated, on the retrieval path) is **re-revised by the owner directive in this TODO**: full text enters the context only through the capped `read` tool (locked A6 below).
## Objective
Grounded turns seed the top 5 related documents as **summary** blocks — a "start here if these summaries seem right to you" starting point — and the LLM extends its context by `read`-ing only what it needs; every document (markdown included) has a stored summary (generated for all new/changed docs at import, backfilled on the next sync for existing NULL rows). Answers get faster and never start by ingesting massive possibly-irrelevant full texts.
## Dependencies
- `117_tool_call_compact` (complete) — the last completed phase; this phase rewrites the `read`-tool copy the battery gate pins, so the gate (task 07) runs against its final state.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Suggested tier (task 02):** new pure `select_suggested(chunks, n=None)` in `app/rag/retriever.py` — the top-N DISTINCT parent documents in fused rank order (the same stable score-descending walk as `select_documents_tiered` — a document's rank is fixed by its FIRST seen chunk), **no cosine floor** (suggestions are opt-in starting points, not citations — the floor stays for the `done`-event related tier only). `n` defaults to the new setting `suggested_docs` (`BOR_SUGGESTED_DOCS`, default **5**, validator `>= 1`).
- **Suggestion blocks (task 03):** the HIGH prompt keeps the `<documents>` section tag and the per-block `<document …>` markup (the E2E mock keys off both), but each block's BODY becomes the document's **summary**: the identity attributes (`source`/`path`/`title`/`date` — the phase-106 D5 `date` stays, always present) + `doc.summary` (stripped). Defensive fallback ONLY when `doc.summary` is None/whitespace (a fail-soft miss — locked A5): the first `suggestion_preview_chars` (`BOR_SUGGESTION_PREVIEW_CHARS`, default **400**) content characters + the shared `TRUNCATION_MARKER` — no LLM call at chat time. A new pinned intro line (a module constant in `app/rag/prompts.py`) precedes the blocks inside the section: the start-here framing — the blocks are summaries of the top-ranked documents; if one seems right, `read` that document (combined `source/path`) to add its full content; answer from what you have and what you read; cite the document(s) you used by path.
- **Agent `read` (task 04):** with summary seeds, `read` on a suggested document **adds its full text** — the existing mechanics are all unchanged (the `BOR_READ_MAX_CHARS` cap + `TRUNCATION_MARKER` + `READ_TRUNCATION_NOTICE` + `ToolResultPiece`/SSE `tool_result` frame, the phase-106 D5 `date:` line, `holder.read_docs` append, `tool_calls` increment). The dedupe set in `_execute_tool`'s read branch becomes **`holder.read_docs` only** (the seeds drop out — a suggested doc is a summary, not full text), so `ALREADY_IN_CONTEXT` fires only for a document ALREADY READ into full-text context (its copy stays accurate: "the full text is already in your prompt"). The `read` tool description in `AGENT_TOOLS` (`app/rag/agent.py`) and the `read` clause of `TOOLS_SECTION` (`app/rag/prompts.py`) are rewritten for the summary-seed mode — SINGLE mode (no kill switch, locked A1): the `<documents>` section shows summaries, `read` adds a document's full text, and re-reading an already-read document is refused. The `ls`/`grep` clauses and the discipline rules (one call per reply, never repeat) stay byte-identical. `tests/unit/test_prompt_lock.py` byte-pins `TOOLS_SECTION` (sha256 + length + prefix/suffix): re-cut its anchors in the same change; `PERSONA` and the LOW `DEFLECT_MODE` body MUST stay byte-identical (the lock pins those too — the E2E mock keys on the markers' presence).
- **Turn wiring (task 05):** `plan_turn` computes BOTH tiers once, for BOTH branches: `suggested = select_suggested(chunks)` and `related = select_related(chunks, excluded={suggested ids}, cap=settings.related_max_docs)` (new pure helper in `app/rag/retriever.py` — the same rank walk, skipping the excluded ids, no floor; with a 5-doc suggestion the related tier is rank order 6..n, the phase-113 UI "nearby docs" row). The HIGH branch: `build_high_prompt(suggested, …)` and `run_agent(..., seed_docs=suggested)`; the LOW branch: prompt BYTE-IDENTICAL (weak-hit titles only, `DEFLECT_MODE`, no tools, chips — A8 gate untouched) but the TurnPlan still carries suggested + related for the durable record. `TurnPlan.docs` is RENAMED `suggested_docs` (same dataclass position); `summary_hits` is redefined as the hit chunks with `is_summary` whose parent document is in the SUGGESTED set. Citations: `done.sources` = suggested + agent-read, deduped by `(source, path)` order-preserving (locked A4 — a grounded turn always shows chips; the deflected turn keeps `[]`). Durable record (LOCKED A3 — the log records retrieval, not citations): `query_log.sources` + the per-turn log line = suggested + related + read, deduped. The per-turn log line gains `suggested=N` after `summary_hits=N` (PLAN §9 extension). `source_usefulness_floor` stops being a seeding input (it may stay for other consumers — the executor checks; the SETTING itself is never removed).
- **Summaries for every document (task 01):** the importer's summary step covers **every** new/changed A9 document — the markdown exclusion (`_import_one_file`'s `if suffix in (".md", ".markdown"): return`) is removed. **Backfill:** on the UNCHANGED branch (same `content_hash`), a document whose `summary is None` gets the same best-effort `_store_summary` pass — summary text stored on `doc.summary` + one new embedded `is_summary` chunk — with a `backfill=True` flag that counts success in a NEW `ImportSummary.summary_backfilled` counter (instead of `summaries`). The backfill runs BEFORE the `created_at_manual` early-return and the date refresh (the manual flag protects the DATE only, phase 106 D1 — a manually-dated doc still gets its summary), is still fail-soft (`LLMError`/`EmbeddingError` → session rollback + `summary_errors += 1`, the doc row untouched, the date refresh still runs), and NEVER counts `added`/`updated`/`pruned` (so no `sources_meta` bump, no KB-overview/folder-summary regeneration — the content is unchanged, locked A2). Non-NULL summaries (owner-edited via phase 57 included) are never touched. The import log line (PLAN §9) gains `summary_backfilled=N` after `summary_errors=N`.
- **Old-contract retirement (task 06):** `tests/e2e/test_whole_document_context.py` (the phase-24 full-text sentinel E2E) is DELETED — its contract no longer exists (locked A1/A6). Every test that pins the retired behavior is updated in place to pin the new one: markers to audit across `tests/unit/`, `tests/integration/`, `tests/e2e/` — the `ALREADY_IN_CONTEXT` seed-read flows (`test_agent.py`, `test_ls_tree_drilldown.py`, `test_read_truncation_cap.py`), the `<documents>` full-text pins (`test_prompts.py`, `test_prompts_dates.py`, `test_chat_gate.py`), the markdown-summary pin (`test_importer.py::test_markdown_file_never_gets_summary` — inverted), the per-turn log-line pins (`test_chat_gate.py` — `summary_hits=`). A pin of UNCHANGED behavior stays untouched.
- **Copy gate (task 07):** the `read`-tool copy is the lever the gate telemetry proved matters (the phase-70/72/94 precedent) — the real-model fixture battery re-runs before the phase completes and its verdict is recorded in `TOOL_CALLING_TESTING.md` (gate semantics: locked A7 below).
## Tasks
1. `01_all_doc_summaries.md` — importer: every document gets a summary (markdown included) + the NULL-summary backfill on the unchanged path.
2. `02_suggested_tier_selection.md` — retriever: `select_suggested` (top-N, no floor) + `BOR_SUGGESTED_DOCS`.
3. `03_summary_document_blocks.md` — prompts: the "start here" suggestion blocks in the HIGH prompt (+ the preview fallback).
4. `04_read_suggested_docs.md` — agent: `read` on a suggested doc adds its full text; the `read` copy rewritten (both surfaces); the prompt-lock re-cut.
5. `05_chat_turn_wiring.md` — `plan_turn`: suggested seeding, `select_related`, citations, the durable record, `suggested=N`.
6. `06_e2e_summary_context.md` — new Playwright E2E; the phase-24 E2E retired; the old pins updated.
7. `07_fixture_battery_gate.md` — the real-model fixture battery re-run + the verdict recorded.
## Testing & Quality
- Unit: `tests/unit/test_importer.py` (markdown summarized; backfill success/failure; the manual-date case; the log-line counter), `tests/unit/test_retriever.py` (`select_suggested` / `select_related` rank order, cap, no floor, dedupe), `tests/unit/test_prompts.py` + `test_prompts_dates.py` (summary blocks, the `date` attribute, the preview fallback, the intro line; the LOW prompt byte-identical), `tests/unit/test_prompt_lock.py` (the re-cut `TOOLS_SECTION` anchors; `PERSONA` + LOW body untouched), `tests/unit/test_agent.py` (read-of-suggested success + re-read refusal + cap path), `tests/unit/test_chat_gate.py` (HIGH seeds 5 summaries — no full text in the prompt; related = rank 6+; both branches' durable record; `suggested=N`), `tests/unit/test_config.py` (the two new settings).
- E2E: `tests/e2e/test_summary_seed_context.py` (new — run in isolation: `uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov`).
- Regression: the remaining E2E suites green (the tool-flow suites re-targeted in task 06), and the battery gate (task 07).
- Coverage: **>90%** on `app/` (the validate.sh gate).
## Completion Criteria
- [ ] A grounded turn seeds the top-5 suggested documents as SUMMARY blocks (no full text in the prompt); the LLM `read`s what it needs and answers faster.
- [ ] Every imported document (markdown included) has a `summary`; an existing NULL-summary doc backfills on the next sync (no content re-embed, no `sources_meta` bump, fail-soft).
- [ ] `read` on a suggested doc adds its full text (cap/truncation/date line unchanged); a re-read of an already-read doc is refused.
- [ ] `done.sources` = suggested + read (deduped); `query_log` + the log line record suggested + related + read; the log line carries `suggested=N`.
- [ ] The deflection path and every SSE frame shape are byte-identical; `PERSONA` + the LOW body stay byte-locked.
- [ ] `uv run pytest` green; coverage >90%; the new E2E green in isolation; the fixture battery gate PASS (task 07); `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — no kill switch (owner-confirmed 2026-09-15, roadmap confirmation):** the summary-seed context IS the behavior; the legacy full-text top-2 seeding is gone and the phase-24 E2E is retired (not re-scoped).
- **A2 — every document gets a summary, no exceptions, including markdown (owner-confirmed 2026-09-15):** generated for all new/changed docs at import; existing NULL rows backfill on the next sync as a summary-only pass (no content re-chunk/re-embed, no `sources_meta` bump); owner-edited (non-NULL) summaries are never touched.
- **A3 — the suggestion set = the top 5 distinct documents in fused rank order, NO cosine floor** (`BOR_SUGGESTED_DOCS` default 5); the floor stays for the `done`-event related tier only (owner-confirmed 2026-09-15).
- **A4 — `done.sources` (the citation surface) = suggested + agent-read, deduped** — a grounded turn always shows chips (owner-confirmed 2026-09-15).
- **A5 — a NULL summary at prompt time (a fail-soft miss) falls back to a 400-char content preview** (`BOR_SUGGESTION_PREVIEW_CHARS`) — deterministic, no LLM call at chat time (owner-confirmed 2026-09-15).
- **A6 — A7 re-revised (owner directive, TODO L3, 2026-09-15):** the retrieval path never seeds full texts; full text enters the context only through the capped `read` tool. `PLAN.md` is untouched (Protocol B) — the owner folds it in at the next redo.
- **A7 — the battery-gate semantics under summary seeding (owner-confirmed 2026-09-15):** conditions 1 (every turn answers) and 2 (zero round-cap hits) still gate; condition 4 (fixture-mode contract accuracy ≥ 0.90) still gates; condition 3 (≥6/10 turns emit ≥1 tool call) is REPORTED, not gated — a summary-only answer is now the intended fast path, not tool abandonment. The full telemetry is recorded in `TOOL_CALLING_TESTING.md` regardless.
## Commit
```bash
git add app/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): seed grounded turns with top-5 document summaries instead of full texts — every document summarized, read extends the context"
```
@@ -0,0 +1,31 @@
# Task 01 — Summaries for every document + the NULL-summary backfill
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…now we have faster ways for the LLM to identify what information it needs… with their summaries loaded into context rather than their entire content…" (locked A2: every document gets a summary, no exceptions, including markdown.)
**Story:** n/a
## Objective
Every A9 document has a stored lite summary: the markdown exclusion is dropped (all new/changed docs get one), and an existing unchanged doc whose `summary` is NULL gets a summary-only backfill on the next sync.
## Work
1. `app/rag/importer.py` — `_import_one_file` (~L539–544): delete the markdown early-return (`if full_path.suffix.lower() in (".md", ".markdown"): return`) so EVERY new/changed document falls through to `_store_summary`. Update the adjacent comment (phase 30's "markdown is already natural language" rationale is retired — locked A2, 2026-09-15).
2. `app/rag/importer.py` — `_store_summary` (~L548–592): add a `backfill: bool = False` keyword; on success count `summary.summary_backfilled += 1` when `backfill` else `summary.summaries += 1` (the two counters are disjoint — the log line keeps both). Everything else (fail-soft rollback, the `is_summary` chunk replacement, the embed) is unchanged.
3. `app/rag/importer.py` — `_import_one_file`'s UNCHANGED branch (~L442–464): after `summary.unchanged += 1` and its log line, BEFORE the `created_at_manual` early-return and the date refresh, add: `if doc.summary is None: await _store_summary(session, doc=doc, source=source, rel=rel, content=content, llm=llm, summary=summary, backfill=True)`. Order matters: a backfill failure rolls back its own session work (nothing else is pending), and the date refresh still runs afterwards. Strict `is None` check — an empty-string summary is owner-set (phase 57) and is never overwritten. A backfill NEVER touches `content_hash`, the content chunks, `created_at` semantics, or the added/updated/pruned counters — so no `sources_meta` bump, no KB-overview/folder-summary regeneration (locked A2).
4. `app/rag/importer.py` — `ImportSummary` (~L95–117): new `summary_backfilled: int = 0` field (docstring: unchanged docs whose NULL summary was backfilled — one `is_summary` chunk per success); update the `summaries`/`summary_errors` docstrings ("Non-markdown files" → "Files" — every A9 format now, locked A2); `log()` gains `summary_backfilled=%d` placed after `summary_errors=%d` (PLAN §9 line extension).
5. `app/rag/summarizer.py` — NO copy change: `SUMMARY_INSTRUCTION` is already format-agnostic and the `SUMMARY_MODE` marker is the E2E mock's key (byte-stable).
6. ASSUMPTION: the backfill is NOT a KB change (locked A2) — no `sources_meta` bump even though a prompt-facing summary appears: the doc content is unchanged, and saved-chat staleness tracks content generations (the phase-53 invariant).
7. ASSUMPTION: `created_at_manual` rows still get the backfill — the manual flag protects the DATE only (phase 106 D1), not the summary.
## Testing & Quality
- `tests/unit/test_importer.py` — update/extend:
- `test_markdown_file_never_gets_summary` (~L574) → INVERT: a markdown file gets a stored summary + one embedded `is_summary` chunk (rename, e.g. `test_markdown_file_gets_stored_summary`).
- new: unchanged doc with NULL summary → backfilled (summary stored, one `is_summary` chunk with embedding, `unchanged` counter, `summary_backfilled=1`, `summaries` untouched, `added/updated/pruned` all 0).
- new: unchanged doc WITH a stored summary → no summary LLM call at all (the fake `chat` is never invoked) and no new chunk.
- new: unchanged doc with `created_at_manual=True` and NULL summary → summary backfilled, `created_at` byte-untouched.
- new: backfill failure (fake `chat` raises `LLMError`) → fail-soft: `summary_errors=1`, `summary_backfilled=0`, the doc row + date refresh unaffected (the date still refreshes when it differs).
- `test_import_summary_log_line_includes_summary_counters` (~L719) → the log line carries `summary_backfilled=N`.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] A sync of a KB containing markdown docs stores a summary on every doc (mock LLM `SUMMARY_MODE` path).
- [ ] A second sync (all unchanged) backfills exactly the NULL-summary docs; the third sync makes no summary calls at all.
- [ ] `uv run pytest tests/unit/test_importer.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,26 @@
# Task 02 — `select_suggested`: the top-N, no-floor suggestion tier
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…after the question is embedded, I want the top 5 related documents to be suggested to the LLM for reading…"
**Story:** n/a
## Objective
A pure `select_suggested(chunks, n)` in the retriever returns the top-N distinct parent documents in fused rank order, with NO cosine floor (locked A3), plus the `BOR_SUGGESTED_DOCS` setting (default 5).
## Work
1. `app/rag/retriever.py` — new pure function `select_suggested(chunks: Sequence[RetrievedChunk], n: int | None = None) -> list[Document]`: walk `chunks` in the SAME stable score-descending order as `select_documents_tiered` (a document's rank is fixed by its FIRST seen chunk; dedupe by `document.id`), return at most *n* documents — **no floor** (a lexical-only hit with cosine 0.0 is suggested when ranked, unlike the phase-113 cited tier). `n` defaults to `get_settings().suggested_docs`. Docstring: the phase-118 "start here" suggestion tier — opt-in starting points, not citations (locked A3); the returned rows carry the full document content, byte-identical (the content is what `read` serves later — never truncated, A6).
2. `app/config.py` — new `suggested_docs: int = 5` (env `BOR_SUGGESTED_DOCS`), validator `>= 1` (the `related_max_docs` non-negative style, `app/config.py` ~L381–387), docstring citing the phase-118 "start here" seeding; `.env.example` entry with the same comment (near the `BOR_TOP_N_DOCS` / `BOR_RELATED_MAX_DOCS` block).
3. `tests/unit/test_config.py` — the setting default + the validator (0 and negative rejected, 1 accepted).
4. `tests/unit/test_retriever.py` — `select_suggested`:
- rank order = best fused score per document, first-seen-chunk rank (the same walk — a doc whose best chunk appears later in the list still ranks by first-seen position, matching `select_documents_tiered`'s order semantics);
- dedupe (multiple chunks of one doc → one row);
- the cap (`n` and the settings default);
- NO floor: a lexical-only 0.0-cosine document is included when ranked (contrast pin against `select_documents_tiered`'s floored cited tier on the same input);
- deterministic tie-break inherited from the fused order.
## Testing & Quality
- Unit: the cases above; coverage **>90%** on `app/`.
## Completion Criteria
- [ ] `select_suggested` returns the top-5 distinct docs in fused rank order for a typical fused list; the floor never filters.
- [ ] `BOR_SUGGESTED_DOCS` is env-tunable (default 5) and validated.
- [ ] `uv run pytest tests/unit/test_retriever.py tests/unit/test_config.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,30 @@
# Task 03 — The "start here" suggestion blocks in the HIGH prompt
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…with their summaries loaded into context rather than their entire content — basically a 'start here if these summaries seem right to you' approach."
**Story:** n/a
## Objective
The HIGH prompt's `<documents>` section seeds each suggested document as an identity + SUMMARY block (never the full content), with a pinned intro line carrying the start-here framing; a NULL summary (fail-soft miss) falls back to a 400-char content preview (locked A5).
## Work
1. `app/rag/prompts.py` — `build_high_prompt`: keep the `<documents>` section tag and the per-block `<document source=… path=… title=… date=…>` markup BYTE-STABLE (the E2E mock's `_document_block` parser and the `read`-tool copy key off both) — only the block BODY changes: `doc.summary.strip()` instead of `doc.content`. The `date` attribute stays (phase 106 D5, always present).
2. `app/rag/prompts.py` — the preview fallback (locked A5): when `doc.summary` is None or whitespace-only, the body is `doc.content[:limit]` + the shared `TRUNCATION_MARKER` on its own line — `limit` from `get_settings().suggestion_preview_chars`, read ONLY on that fallback path (the house pattern: `build_steering_section`/`build_kb_section` read settings inside the function; `build_high_prompt`'s signature is unchanged, every existing caller works, and a prompt built from summary-bearing docs makes no settings call for the cap). New setting in `app/config.py`: `suggestion_preview_chars: int = 400` (env `BOR_SUGGESTION_PREVIEW_CHARS`, validator `> 0`) + `.env.example` entry.
3. `app/rag/prompts.py` — new module constant `SUGGEST_INTRO` (pinned copy, the phase-15 `_STEERING_INTRO`/phase-31 `_KB_INTRO` precedent) rendered inside the `<documents>` section, BEFORE the first block, when at least one block is present. Copy (adjust wording freely, keep the three contracts below, pin the final text in `test_prompt_lock.py` or `test_prompts.py`): the blocks are SUMMARIES of the top-ranked documents for the question — start here if one seems right; call `read` with the document's combined `source/path` to add its FULL content before answering specifics; a document's full text is NOT in the prompt until you read it; cite the document(s) you used by path. The empty-body fallback line (`"(no documents matched — do not invent specifics)"`) stays.
4. `app/rag/prompts.py` — docstrings: the module docstring's HIGH-mode description ("full top-document texts under `<documents>`") → the summary-seed contract (A6 re-revised); `build_high_prompt`'s docstring updated likewise; the phase-106 D5 note (the `date` attribute) stays.
5. `app/rag/retriever.py` — module docstring: the A7-revised paragraph ("the LLM receives the entire relevant document… the full text of the top-N documents is always fed through, never truncated") → the A6 re-revised contract: the retrieval path seeds SUMMARIES (the suggestion tier, locked A3); full text enters the context only through the capped `read` tool; `select_documents`/`select_documents_tiered` docstrings get the same one-line pointer (the functions themselves are untouched — they still serve the related tier + any remaining callers).
6. `build_deflect_prompt` is UNTOUCHED (the LOW prompt + `DEFLECT_MODE` body stay byte-identical — the prompt lock pins it).
7. ASSUMPTION: the section keeps the name `<documents>` (mock + copy contract) rather than a new `<suggested>` tag — the tag is a stability surface, the intro line carries the semantic change.
## Testing & Quality
- `tests/unit/test_prompts.py` — a HIGH prompt built from docs with summaries: the block bodies are the summaries (sentinel: a full-content string that appears NOWHERE in the prompt; the summary text appears inside the `<document>` block), the intro line is present before the first block; the identity + `date` attributes byte-identical to today's format.
- `tests/unit/test_prompts_dates.py` — the `date` attribute pins survive the body change (update bodies, keep the attribute assertions).
- new: the preview fallback — a doc with `summary=None` → body = first 400 chars + `TRUNCATION_MARKER`; `summary=" "` → same; a short content (< 400) → the whole content, no marker; `BOR_SUGGESTION_PREVIEW_CHARS` env override honored.
- new: the LOW prompt is byte-identical to pre-task on the same inputs (sha-pin via `build_deflect_prompt` output or the existing lock).
- `tests/unit/test_prompt_lock.py` — if `SUGGEST_INTRO` is added as a constant, pin it there (sha + prefix); `PERSONA`/LOW-body anchors untouched.
- `tests/unit/test_config.py` — the new setting.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] A grounded prompt built from 5 summary-bearing docs contains all 5 summaries + the intro, and ZERO full-content characters beyond the preview cap for NULL-summary docs.
- [ ] The LOW prompt output is byte-identical to pre-phase for identical inputs.
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_prompts_dates.py tests/unit/test_prompt_lock.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,30 @@
# Task 04 — `read` on a suggested doc adds its full text + the rewritten `read` copy
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…the top 5 related documents to be suggested to the LLM for reading, with their summaries loaded into context rather than their entire content…" (suggested = "for reading" — the model reads what its summaries point at.)
**Story:** n/a
## Objective
`read` on a suggested document is allowed and adds the document's full text (all existing read mechanics unchanged); the dedupe refusal fires only for an already-READ document; the `read` tool description and the `TOOLS_SECTION` `read` clause are rewritten for the summary-seed mode; the prompt-lock anchors are re-cut.
## Work
1. `app/rag/agent.py` — `_execute_tool`'s `read` branch (~L1131–1136): `known = {(doc.source, doc.path) for doc in holder.read_docs}` — the `seed_docs` fall out of the dedupe set (a suggested doc is a summary in the prompt, NOT its full text). A first `read` of a suggested doc now succeeds through the EXISTING path unchanged: `holder.read_docs.append(doc)`, `holder.tool_calls += 1`, the `BOR_READ_MAX_CHARS` cap + `TRUNCATION_MARKER` + `READ_TRUNCATION_NOTICE` + `holder.read_truncations` (→ the SSE `tool_result` frame), the `Document {source}/{path}:` header + phase-106 D5 `date:` line. A second `read` of the same (now full-text-in-context) document still gets `ALREADY_IN_CONTEXT` — its copy stays byte-identical (it already names the correct action: answer from the text in the prompt). The `seed_docs` parameter of `_execute_tool`/`run_agent` stays (callers unchanged) but its docstring + the `run_agent` docstring paragraph "``seed_docs`` are the documents the retrieval already put in context… re-reading one of them is rejected" are rewritten: the seeds are SUMMARY blocks in the prompt; reading one adds its full text; only an already-READ doc is refused.
2. `app/rag/agent.py` — `AGENT_TOOLS` `read` function description: rewrite the leading contract sentences — the `<documents>` section shows the SUMMARIES of the top-ranked documents (their full texts are NOT in the prompt); use `read` to add one of them (or any other document) by its combined `source/path` string, exactly as shown in the `ls` output / the `<documents>` blocks; do not re-read a document you have already read — its full text is already in your prompt. KEEP byte-identical: the combined-`source/path` identity contract, the bare-path-will-not-resolve sentence, the truncation-notice paragraph, the "call one tool at a time" sentence, and the `path` parameter description (minus its "Only pass a document NOT already shown in the `<documents>` section" clause → "Do not re-read a document you have already read").
3. `app/rag/prompts.py` — `TOOLS_SECTION`: rewrite ONLY the `read` clause the same way (today's "Do not call `read` for a document already shown in the `<documents>` section, even when the user asks you to open or read it — its full text is already in your prompt; answer directly from it." becomes: the `<documents>` section holds SUMMARIES — the full text of a suggested document is not in the prompt until you `read` it; do not re-read a document you have already read — its full text is already in your prompt; answer directly from it. If the user asks you to open or read a suggested document, `read` it — that is the point of the section.) KEEP byte-identical: the `ls` clause (the phase-94 drill-down contract), the `grep` clause, and the discipline rules (one call per reply, never repeat a refused/succeeded call, answer as soon as you have what you need).
4. `tests/unit/test_prompt_lock.py` — re-cut the `TOOLS_SECTION` anchors for the new copy: `TOOLS_SECTION_SHA256` + `TOOLS_SECTION_LEN` ALWAYS change; `TOOLS_SECTION_PREFIX`/`SUFFIX` only if the new copy touches those exact boundaries (the prefix is the `ls`-clause opening and the suffix is the discipline-rules ending — both stay, so they should survive; verify, don't assume). Update the "Pre-phase-112 anchors" comment to cite the phase-118 re-cut (A6, owner directive 2026-09-15). `PERSONA` and the LOW-body anchors stay untouched — if they fail, the change leaked into locked text and the task is wrong.
5. `app/rag/agent.py` — module docstring: the "A7 scope clarification" paragraph (~L120, "the top-2 seed documents stay whole — 'this should never happen'") → the A6 re-revised contract (seeds are summaries; the `read` path is the only full-text path and keeps its cap); point 3's seed-read refusal teaching → the already-read refusal.
6. ASSUMPTION: the `ALREADY_IN_CONTEXT` constant text is unchanged (it is accurate for the already-read case and the E2E mock/telemetry key on stable copy).
## Testing & Quality
- `tests/unit/test_agent.py` — update/extend:
- a `read` of a `seed_docs` document SUCCEEDS: the full-content result (header + date line), `holder.read_docs` length 1, `holder.tool_calls` 1;
- a second `read` of the same document → `ALREADY_IN_CONTEXT` (byte-identical line), counters untouched;
- the cap path on a suggested doc (content > `read_max_chars`) → truncated result + notice + `read_truncations` entry (existing pins re-targeted from "seed" to "suggested");
- the `AGENT_TOOLS` `read` description pins: the new summary contract present, the identity/truncation/one-call sentences byte-present;
- `ls` and `grep` behaviors byte-identical (their existing pins untouched).
- `tests/unit/test_prompts.py` — the `TOOLS_SECTION` inside a built HIGH prompt carries the new `read` clause; `ls`/`grep` clauses byte-identical (sha-pin the unchanged clauses if the file's style allows).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `read('source/path')` on a suggested doc adds its full text to the context and counts as an executed call; a repeat is refused.
- [ ] `PERSONA` + the LOW `DEFLECT_MODE` body byte-locks pass without re-cutting; only the `TOOLS_SECTION` anchors moved.
- [ ] `uv run pytest tests/unit/test_agent.py tests/unit/test_prompt_lock.py tests/unit/test_prompts.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,39 @@
# Task 05 — `plan_turn` wiring: suggested seeding, citations, the durable record
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…the top 5 related documents to be suggested to the LLM for reading, with their summaries loaded into context rather than their entire content… That way the LLM responds faster…"
**Story:** n/a
## Objective
`plan_turn` seeds the HIGH branch with the top-5 suggested documents (summary blocks) instead of the top-2 full texts; the related tier becomes rank 6+; citations (`done.sources`) = suggested + agent-read; the durable record (`query_log` + log line) = suggested + related + read with a new `suggested=N` field; the LOW/deflection path stays byte-identical.
## Work
1. `app/rag/retriever.py` — new pure helper `select_related(chunks, excluded_ids: set[uuid.UUID], cap: int) -> list[Document]`: the SAME stable score-descending walk as `select_documents_tiered` (first-seen-chunk rank, dedupe by id), skipping documents in *excluded_ids*, no floor, at most *cap* documents. (The phase-113 related tier was "ranked after the cited tier"; with a contiguous top-5 suggestion set, "excluding the suggested" is exactly "rank 6+".) Unit-test it here or in task 05's tests — either location, one set of tests.
2. `app/api/chat.py` — `plan_turn`: replace the `select_documents_tiered(chunks, n=settings.top_n_docs, floor=…, related_cap=…)` call with:
```python
suggested = select_suggested(chunks) # BOR_SUGGESTED_DOCS (5), no floor (A3)
related_docs = select_related(chunks, {d.id for d in suggested}, settings.related_max_docs)
```
Both branches then use `suggested`/`related_docs`: the HIGH branch → `build_high_prompt(suggested, notes=…, kb_overview=…)`; the LOW branch → `build_deflect_prompt(titles, …)` UNCHANGED (weak-hit titles only). `summary_hits` = the hit chunks with `is_summary` whose parent document id is in the SUGGESTED set (redefined from the cited set — the counting code shape is unchanged).
3. `app/api/chat.py` — `TurnPlan`: rename field `docs` → `suggested_docs` (SAME dataclass position; update every attribute access). Docstrings: the field holds the summary-seeded suggestion tier (A6); `related_docs` docstring updated (rank 6+ after the suggested set).
4. `app/api/chat.py` — the `chat` endpoint's post-stream step: `run_agent(..., seed_docs=plan.suggested_docs, …)`; `cited_docs` loop over `[*plan.suggested_docs, *holder.read_docs]` (LOCKED A4 — the dedupe pattern is byte-identical); `record_docs` over `[*plan.suggested_docs, *plan.related_docs, *holder.read_docs]` (LOCKED A3 — the durable record keeps the full retrieval); `source_paths` unchanged.
5. `app/api/chat.py` — the per-turn log line (PLAN §9): add `suggested=%d` (`len(plan.suggested_docs)`) after `summary_hits=%d`; the field order of every existing field is untouched (the phase-114 `retries=%d scaffold_stripped=%d` tail stays last).
6. `app/config.py` — `top_n_docs` and `source_usefulness_floor` lose their seeding role (plan_turn was their only `app/` consumer — verified). KEEP both settings (env back-compat) and keep `select_documents`/`select_documents_tiered` (dormant public helpers): one-line docstring note on each — "phase 118 retired the full-text seeding role (A6); the suggested tier (`select_suggested`) seeds the prompt now." No deletions.
7. ASSUMPTION: the `done.related` UI row (the phase-113 de-emphasized "nearby docs") keeps showing rank 6+ under the new tiering — the field name, shape and dedupe-vs-cited rule are unchanged (a read related doc stays a citation, never a "nearby doc").
## Testing & Quality
- `tests/unit/test_chat_gate.py` — update/extend:
- HIGH branch: the prompt contains the 5 suggested docs' SUMMARY text and NONE of their full content (sentinel); the related tier = rank 6+ (a 7-doc fixture: related = docs 6–7, capped by `related_max_docs`);
- the deflected branch: prompt byte-identical to pre-task on the same chunks (sha-pin the output), suggestions/chips unchanged, `deflected=True`;
- BOTH branches' TurnPlan carry suggested + related (the durable-record inputs);
- `summary_hits` redefinition (a summary chunk whose parent is in the suggested set counts; one whose parent is related-only does not);
- the log-line format pin (search the suite for the `summary_hits=` format string) gains `suggested=N` in the right slot;
- settings: `suggested_docs`/`related_max_docs` honored; `top_n_docs`/`source_usefulness_floor` no longer consulted by `plan_turn` (behavioral pin, not a removal pin).
- `tests/integration/test_chat_api.py` (and any sibling pinning the done-event sources) — `done.sources` = suggested + read (deduped) on a grounded turn with a mock `read` flow; `[]` on deflection; `done.related` = rank 6+ minus cited; `query_log.sources` = suggested + related + read.
- `tests/unit/test_retriever.py` — `select_related` (walk order, exclusion, cap, no floor, determinism).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] A grounded turn's system prompt seeds exactly the top-5 suggested summaries; `run_agent` gets them as `seed_docs`.
- [ ] `done.sources` = suggested + read (deduped, order-preserving); `query_log.sources` + the log line record suggested + related + read; `suggested=N` lands after `summary_hits=N`.
- [ ] A deflected turn is byte-identical in prompt and frames to pre-phase.
- [ ] `uv run pytest tests/unit/test_chat_gate.py tests/integration/test_chat_api.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,31 @@
# Task 06 — E2E: the new summary-seed suite, the phase-24 E2E retired, old pins updated
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…basically a 'start here if these summaries seem right to you' approach. That way the LLM responds faster and doesn't begin by reading massive amounts of data that might be irrelevant…"
**Story:** n/a
## Objective
Playwright coverage of the summary-seed contract (summaries in the prompt, full text only via `read`, markdown docs summarized, citation chips = suggested + read); the retired phase-24 full-text E2E deleted; every E2E/integration pin of the retired behavior re-targeted to the new contract.
## Work
1. NEW `tests/e2e/test_summary_seed_context.py` (run in isolation: `uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov`; DB up: `podman compose up -d db`). Pattern: the sibling suites' deterministic seeding (the mock LLM's bag-of-words `embed_text` + `import_sources` or the sync API — see `tests/e2e/test_read_truncation_cap.py`'s token-controlled seeding and `tests/e2e/test_whole_document_context.py`'s direct SQLAlchemy seeding; the mock's `SUMMARY_MODE` digest makes every stored summary deterministic, markdown included). Token-controlled bodies so the hybrid gate picks the intended suggested set deterministically. Cases:
- **(a) summaries seeded**: the `END_OF_NOTES_TRIGGER` tail-echo question → the answer quotes the last 160 chars of the seeded `<documents>` context, which now end in the LAST suggested document's SUMMARY — pin the mock's summary digest text (the phase-30 digest shape), proving summaries reached the prompt.
- **(b) full text NOT seeded**: the inverse of the phase-24 pin — a sentinel on a suggested document's LAST LINE is absent from the echoed context (the full content never reached the model pre-`read`).
- **(c) `read` adds the full text**: a scripted mock tool flow `read`s a suggested document (the existing `_tool_flow` read-echo pattern, cf. `test_ls_tree_drilldown.py`) → the answer quotes the full-text tail from the read RESULT (the sentinel now appears, sourced from the read, not the seed).
- **(d) markdown gets a summary**: a markdown document in the KB appears as a suggestion block carrying the mock `SUMMARY_MODE` digest (not a content preview) — the locked A2 end-to-end proof.
- **(e) citation chips**: after case (c), the UI chip row = suggested + the read document (deduped — the read doc is among the suggested, so chips = the suggested set), each a `done.sources` entry; `done.related` renders the de-emphasized row (rank 6+) when the seeded KB is large enough to have one.
2. `tests/e2e/test_whole_document_context.py` — DELETE (the retired full-text-seed contract; locked A1/A6 — no kill switch to re-scope it to).
3. `tests/e2e/test_ls_tree_drilldown.py` — the `READ_ANSWER_PREFIX` flow (~L61–67, L229–234, L660–666): today the scripted read targets a top-2 seed and is REFUSED (`ALREADY_IN_CONTEXT`), the mock answering from the `<documents>` full text. Re-target: the scripted read target stays, but the read now SUCCEEDS (seeds are summaries) and the mock answers from the READ RESULT with the same citation shape — update the prefixes/quotes to key off the read result (the full text, first 80 chars — the content is now delivered by the read, so the quote is byte-identical to today's) and the comments to the new contract. If the mock's answer-from-`<documents>` branch (its `ALREADY_IN_CONTEXT`-keyed flow, `mock_llm.py` ~L237) is no longer exercised by ANY suite, keep it (the already-read refusal is still real) but update its docstring (it now describes the already-read case, not the top-2-seed case).
4. `tests/e2e/test_read_truncation_cap.py` — the setup guard (~L364–379) that asserts the scripted read target is NOT a top-2 seed (its only reason was the seed-read refusal) — REMOVE the guard and its docstring; a seed-target read now executes, so the cap fires either way.
5. `tests/e2e/test_document_summaries.py` — the phase-30 pins: `summary.summaries == 1` for the yaml + md-control pair (~L169) → `== 2` (markdown summarized too, locked A2); the "md control has no summary" assertions → inverted to "the md control HAS the mock digest summary"; the summary-chunk count pins gain the md doc's chunk.
6. `tests/e2e/mock_llm.py` — extend ONLY if a case above needs a new trigger; the marker contracts stay (`<tools>` presence, `DEFLECT_MODE`, `_READ_RESULT_PREFIX` header, `END_OF_NOTES_TRIGGER`, `SUMMARY_MODE`). Update the docstrings that describe the top-2-seed/`ALREADY_IN_CONTEXT` scenario (~L235–241).
7. Audit sweep (markers, in `tests/e2e/`, `tests/integration/`, `tests/unit/`): `ALREADY_IN_CONTEXT`, "already shown in the `<documents>`", "top-2 seed", full-content-in-prompt sentinels, `summary.summaries` counts, the per-turn log-line format (`summary_hits=` — gains `suggested=N`). Every hit: pin of RETIRED behavior → re-target to the new contract; pin of UNCHANGED behavior → leave. `tests/unit/test_agent.py`'s seed-read pins were already updated in task 04 — do not duplicate.
## Testing & Quality
- The new suite green in isolation (the command above); every re-targeted suite green; full suite green.
- Coverage: **>90%** on `app/` (E2E is uncounted; keep the unit coverage from tasks 01–05 intact).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov` green (cases a–e).
- [ ] `tests/e2e/test_whole_document_context.py` no longer exists; `tests/e2e/test_ls_tree_drilldown.py`, `test_read_truncation_cap.py`, `test_document_summaries.py` green with the new-contract pins.
- [ ] The audit sweep left no test pinning the retired full-text-seed or seed-read-refusal behavior.
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,23 @@
# Task 07 — The fixture-battery gate re-run (real model) + verdict recorded
**Phase:** `118_summary_seed_context` · **Source:** `TODO.md:3` — "…That way the LLM responds faster…" (the summary-seed behavior must be proven against the real configured chat model before the phase completes — the phase-70/72/94 tool-copy gate precedent.)
**Story:** n/a
## Objective
The fixture KB dump is rebuilt so every fixture document (all eight are markdown) carries a stored summary — the phase-118 end state — and the real-model fixture battery passes against it under the locked gate semantics (locked A7), with the verdicts recorded in `TOOL_CALLING_TESTING.md`.
## Work
1. Rebuild the fixture dump — `uv run python -m scripts.load_test_kb` (the real pipeline: `import_sources` now summarizes every document, task 01 — the eight markdown fixture docs gain `doc.summary` + one embedded `is_summary` chunk each, so the dump's documents/chunks content changes; a rebuild is required, this is a legitimate re-run trigger — "the import pipeline's summary policy changed (phase 118)"). Preconditions: DB up, the aipi `embed` + `lite` endpoints reachable (`.env`). Verify from the build output: `summary_errors=0`, zero failed folder-summary batches, and the retrieval report's battery all-GROUNDED (the honesty gate is untouched by this phase, so grounded/deflected should be unchanged; if a question now deflects, it is a fixture-content regression — fix the fixture anchor per the build's own warning, never the gate). Commit the regenerated `tests/fixtures/test_kb.dump.sql` with the phase (it is tracked).
2. Run the gate — `uv run python -m scripts.agent_realmodel_check --restore --mode fixture` (the fast loop: inline restore + the curated battery against the live endpoint, the configured chat model). Use `--turns N` only for the copy-iteration micro-loop while refining task 04's `read` copy; the VERDICT run is always the full battery.
3. Interpret the verdict under locked A7 (owner-confirmed 2026-09-15): the phase gate = condition 1 (all turns answer) AND condition 2 (zero round-cap hits) AND condition 4 (fixture-mode contract accuracy ≥ 0.90) — the script prints each; condition 3 (≥6/10 turns emit ≥1 tool call) is REPORTED, not gated — a summary-only answer is the intended fast path now, not tool abandonment. If the script exits 1 on condition 3 alone with 1/2/4 green, the phase gate is PASS — record exactly that (the four metrics + the wall time + the `contract X/Y` line, the TOOL_CALLING_TESTING.md verdict format).
4. Copy iteration (only if condition 4 degrades or condition 2 is hit): refine the `read` description (`app/rag/agent.py` `AGENT_TOOLS`) / `TOOLS_SECTION` clause (`app/rag/prompts.py`) — re-cut the `test_prompt_lock.py` `TOOLS_SECTION` anchors again if the copy moves — micro-loop with `--turns`, then a full verdict run. `classify_call` needs NO re-baselining: a `read` of a suggested document is a well-formed call targeting a resolvable document — contract-correct under both old and new app semantics (its docstring's `ALREADY_IN_CONTEXT` note already excludes dedupe re-reads from the metric).
5. `TOOL_CALLING_TESTING.md` — append a dated section ("Phase 118 — summary-seed gate, 2026-09-15…"): the four conditions + metrics, the contract line, the wall time, the model name, the dump's rebuild note (summaries now on all eight docs), the locked-A7 semantics note (condition 3 reported, not gated), and the `read` copy's final shape (or a pointer to it). The controlled-methodology sections above stay untouched (append-only history).
## Testing & Quality
- The gate is the test: no unit suite here; the rebuild's own round-trip verification (dump restore + per-table checksum) must pass, and the full `uv run pytest` must stay green after the dump regeneration (E2E suites that restore the fixture dump pick up the summary rows — if any suite pins the old 9-chunk fixture state, that pin moves to the new 17-chunk state: 9 content + 8 summary chunks).
## Completion Criteria
- [ ] `tests/fixtures/test_kb.dump.sql` regenerated with all eight fixture docs summarized (round-trip checksum verified by the build).
- [ ] Full-battery verdict run recorded: conditions 1, 2, 4 GREEN (condition 3 reported); the telemetry + semantics note appended to `TOOL_CALLING_TESTING.md`.
- [ ] `uv run pytest` green with the new dump (including any fixture-chunk-count pins updated).
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,79 @@
# Phase 119 — Name-signal retrieval + suggested-folder context; chips cite read docs only
**Source:** owner live report, 2026-09-16 — brain.reeseapps.com, turn "How do I deploy gitea?". Three findings, all validated live and reproduced against the 1,000-document live replica (restored 2026-09-16 from `data/bor_live_kb_replica.dump.sql`):
1. **Irrelevant documents in the chip cards** — the turn's 7 chips were 5 suggested docs + 2 agent-read docs (phase-118 locked A4); 2 of the 5 suggestions were off-topic (`homepage/src/index.html`, `deploy/Deployments/reeseapps/gateway/README.md`) and 3 were never read. The agent's one *failed* read (`docker-compose.yaml` — verified absent from the live KB, teaching refusal) correctly earned no chip, so the chip surface itself had no construction bug — it faithfully showed a noisy suggestion tier.
2. **The agent lists instead of reading the proposed files** — the 5 summaries seeded into the prompt did not include the canonical `deploy/Deployments/reeseapps/gitea/README.md`, so the model rationally explored: 3 `ls` drill-downs (`deploy` → `reeseapps` → `gitea`), one wrong-guess `read` (refused), one more `ls`, one `read` — 7 tool rounds. It never read a single suggested file, and its answer's "Docs used:" line cited `gitea-web.env.j2`, a file it never read (confabulation from the suggestion blocks being in context).
3. **Suggested documents off** — reproduced on the replica (which lacks the phase-118 markdown summary chunks; live was *worse*: `index.html` ranked #1 there):
| Question (replica, pre-phase) | Suggested top-5 | Canonical doc |
|---|---|---|
| "How do I deploy gitea?" | gitea playbook, **gateway README**, k8s gitea-values ×2, container_gitea.md | ❌ rank 11 |
| "How do I deploy forgejo?" | foobar.md, AGENTS.md, reeseapps README, inventory.yaml, bifrost.md | ❌ **zero forgejo docs** |
| "How do I set up the gateway?" | shelly.md, forgejo.network, framework_16.md, firewall playbook, home_assistant.md | ❌ not in top-5 |
**Root causes.** (a) The lexical side never sees document *names*: `chunks.tsv` is `to_tsvector('english', content)` — body only (migration 0002) — and the name-hit mechanism (the 2026-09-05 versioned-name incident fix, `app/rag/retriever.py::_name_hit_chunks`) fires **only for digit-bearing** question tokens, so `gitea`/`forgejo`/`gateway` (no digits) get no name signal; "forgejo" appears in only 26 chunk bodies vs 136 for "gitea", so an OR-tsquery dominated by the common token "deploy" buries the name. (b) Phase 118 widened the blast radius: suggestions = top-5 **no floor** (118-A3) and chips = suggested + read (118-A4), so every noisy suggestion became a visible junk chip; the new per-document summary chunks also created hub attractors (site-wide summaries vector-match almost anything). (c) The prompt teaches "start here **if one seems right**" + "answer as soon as you have what you need" — with a suggestion set missing the canonical doc, exploration (and its latency) is the rational model behavior.
**Story:** n/a (owner-directed RAG behavior change; extends the phase-118 summary seeding, the 2026-09-05 name-hit mechanism, and the phase 94/113 tier assets).
## Objective
Product-name questions surface the product's own documents in the suggestion tier (component name-hits + a bounded, kill-switchable name-hit bonus); the HIGH prompt pre-seeds each suggested document's parent-folder contents (collapsing the `ls` drill-downs the agent currently pays); the citation chips carry **only documents the agent actually read** (owner decision 2026-09-16 — phase-118 A4 retired); and the model is taught to cite only what it read or used.
## Dependencies
- `118_summary_seed_context` (complete) — the summary-seed prompt, `select_suggested`/`select_related`, the `read`-extends-context contract, and the done-frame tiering this phase re-locks.
- `117_tool_call_compact` (complete) — the current `read`/`ls`/`grep` copy this phase leaves byte-identical.
## Design (shared by all tasks — the executor reads this, not the chat)
**D1 — name-hit rule (task 01, LOCKED A2).** `name_hit_tokens` returns **all** normalized whitespace tokens of length ≥ 4 (dotted tokens kept whole, as today) *plus* the versioned-name join candidates — the digit filter moves from the token list to the match side. `_name_hit_chunks` matches a token against the document's **path** (never the title — titles are prose: `deploy/Deployments/reeseapps/README.md` is titled "Deployments", and the common token "deploy" must not name-match it or the 9 other deployment-titled docs; the owner-verified failure mode of the naive relaxation):
- **digit-bearing token** → PREFIX of a normalized path part or file stem (preserves the 2026-09-05 incident fix: `qwen38` → `qwen3.8-27b-juggernaut-vulkan.container`);
- **digitless token** → EQUALS a normalized path part, a file stem, or a stem sub-component (the stem split on non-alphanumeric runs: `kubernetes_gitea` → `kubernetes`/`gitea`, `gitea-values` → `gitea`/`values`, `test-gateway` → `test`/`gateway`).
Measured effect (replica, 2026-09-16, replacing the digit-only rule with this one): "forgejo" → all 5 suggested slots are forgejo-tree docs (README #1); "gitea" → 4 of 5 slots gitea-tree incl. the canonical README; "gateway" → 4 of 5 slots gateway-tree. Ranking of name hits: (distinct matched token count desc, then catalog order `(source, path)`) — the total-matched-length tie-break is RETIRED (it systematically outranked 5-char product names by 6-char common tokens). `NAME_HIT_LIMIT` stays 10. Name-hit rows keep the existing `fts_hit=True`, `cosine=0.0` convention (A8 already treats them as a lexical signal needing vector corroboration). `RetrievedChunk` gains `name_hit: bool = False` (dataclass field, default keeps every existing construction valid) so the selection tier can tell them apart.
**D2 — name-hit bonus (task 02, LOCKED A3).** A bounded, kill-switchable **selection-time** re-rank — the phase-106 recency-boost pattern (additive, single apply site, `0` = off → byte-identical ordering, negative fails startup loudly). New setting `name_hit_bonus` (`BOR_NAME_HIT_BONUS`, default **0.005**). In the document-selection walks only (`select_suggested`, `select_related`, `weak_hit_titles`), a document's effective selection score = its best fused chunk score **+ bonus if any of its chunks is a name hit**. Chunk scores, `fuse()`, `retrieve()`, the A8 honesty gate (reads chunk cosine/fts), and `query_log.top_score` are untouched. 0.005 ≈ a 2–4 rank head start on the RRF scale (rank 1 vs 5 in one list ≈ 0.0010); measured, it is the difference that puts the canonical doc inside the top-5 on all three failing questions (post-phase suggested top-5: gitea → playbook, gitea-postgres.env.j2, gateway README, **gitea README #4**, gitea-web.env.j2; forgejo → 5/5 forgejo; gateway → shelly.md, istio test-gateway.yaml, **gateway README #3**, gateway certbot/ddns playbooks). The 0.005 default is an owner-tunable starting point, not a calibrated constant — the battery (task 06) records the realized margins. 118-A3 is **re-revised** by this lock: the suggestion set is still top-5 distinct docs with NO cosine floor, now walked with the name-hit bonus. `scripts/eval_retrieval.py` gains a `suggested:` line (the `select_suggested` walk over the fused list, bonus included) so the tuning tool reports the tier the prompt actually seeds.
**D3 — suggested-folder context (task 03, LOCKED A4).** The HIGH prompt's `<documents>` section carries, **after `SUGGEST_INTRO` and before the first `<document>` block**, one line per DISTINCT parent folder of the suggested documents (in suggested-doc order, deduped by `(source, parent prefix)`, at most 5 lines): `<source>/<parent>/: <entry, …>` — subfolders as `name/ (N docs)`, files by relative filename, entry order = the existing `ls` folder-level rendering order (`group_folder_listing` in `app/rag/agent.py` — the line must read the same as the model's own `ls` output), capped at 8 entries per line + ` +N more`; the source root renders as `<source>/: …`. The suggested document itself is excluded from its folder's entries (its identity is already in the block). Source: a new pure helper `suggested_folder_lines(db, suggested, …)` next to the `ls` helpers in `app/rag/agent.py`, reusing `_source_document_rows` + `group_folder_listing`; computed in the chat endpoint (which holds the session) and passed through `plan_turn` (new parameter, default empty) → `build_high_prompt` (new parameter, default empty). The `<document>` block markup AND body stay **byte-identical** (the E2E mock's tail echo keys off the LAST block's tail — the folder lines must never land after a summary); the LOW prompt stays byte-identical; with an empty `folder_lines` the HIGH prompt is byte-identical to the phase-118 output. The live-turn payoff: the README sits in the parent folder of suggested `gitea/app/gitea-web.env.j2`, so its name appears in the prompt — the model reads it in round 1 instead of walking three `ls` levels.
**D4 — cite discipline (task 04, LOCKED A5).** `SUGGEST_INTRO`'s final sentence ("Cite the document(s) you used, by path.") is REPLACED with: `Cite only the document(s) you read — or, if you answered from a suggested summary without reading it, cite that suggested document — never a document you neither read nor used.` `PERSONA`, the LOW/`DEFLECT_MODE` body, and `TOOLS_SECTION` stay byte-identical (the phase-117 copy is untouched — no tool-copy gate re-run, D6); the byte-pinned anchors re-cut in the same task.
**D5 — chips cite read docs only (task 05, LOCKED A1 — owner decision 2026-09-16, phase-118 A4 retired).** `done.sources` = `holder.read_docs` only, deduped by `(source, path)`, read order preserved. A grounded turn on which the agent reads nothing (the phase-118 summary-only fast path) shows **no chips** — an accepted, owner-directed consequence (the answer prose still names the doc it used, per D4). The DURABLE record is untouched (118-A3 stands: `query_log.sources` + the per-turn log line = suggested + related + read); the `related` tier (the de-emphasized "nearby docs" row) is untouched except its existing dedupe against the cited set now keys on read docs only (a read related doc is a chip, never a nearby doc — unchanged intent). No frontend change: `appendSources`/`addSources` already render zero entries as no DOM.
**D6 — verification (task 06).** New dedicated Playwright suite `tests/e2e/test_name_signal_read_chips.py` (run in isolation; the `test_summary_seed_context.py` local-directory-source fixture pattern), a mock-LLM trigger echo making the folder lines observable, the retired-A4 pins re-targeted, and the empirical battery: the 1,000-doc live replica (the `restore-test-db` skill; the dump is a local gitignored artifact — if missing, STOP and ask the owner, never fabricate numbers) + `scripts/eval_retrieval --from-file` over a tracked question battery, + the real-model fixture run recorded as TELEMETRY in `TOOL_CALLING_TESTING.md` (the gate is NOT re-triggered — no `AGENT_TOOLS`/`TOOLS_SECTION` copy change; conditions, were they read, follow the phase-118 A7 semantics).
**Invariants (all tasks).** The A8 honesty gate, every SSE frame shape, the LOW prompt, `PERSONA`, `TOOLS_SECTION`, the `<document>` block markup/body, the `Document {source}/{path}:` read-result header, and the per-turn log line format (fields unchanged; `sources=`/`suggested=` values move with the data) are byte-identical. `PLAN.md` untouched (Protocol B — the owner folds the 118-A4 retirement + 118-A3 re-revision in at the next redo; both are owner-confirmed below).
## Tasks
1. `01_name_hit_component_match.md` — retriever: the component name-hit rule (D1) + the `name_hit` flag.
2. `02_name_hit_bonus.md` — the bounded selection-time bonus (D2) + `eval_retrieval` suggested line.
3. `03_suggested_folder_context.md` — the prompt's suggested-folder lines (D3).
4. `04_cite_discipline_copy.md` — `SUGGEST_INTRO` cite discipline + the re-cut pins (D4).
5. `05_chips_read_only.md` — `done.sources` = read docs only (D5).
6. `06_e2e_retarget_and_battery.md` — the dedicated E2E, the retired pins, the mock echo, the replica battery + real-model telemetry (D6).
## Testing & Quality
- Unit: `tests/unit/test_retriever.py` (D1 rule incl. the "deploy"-title precision case and the versioned prefix; D2 bonus lift/kill-switch/validator), `tests/unit/test_config.py` (the new setting), `tests/unit/test_agent.py` or a new `tests/unit/test_folder_context.py` (D3 helper: root/nested/exclusion/dedupe/caps), `tests/unit/test_prompts.py` + `tests/unit/test_prompt_lock.py` (D3 line placement + the D4 re-cut; LOW + PERSONA byte-identical), `tests/unit/test_chat_gate.py` (D5 done-frame + durable record).
- E2E: `tests/e2e/test_name_signal_read_chips.py`, **in isolation** (`uv run pytest tests/e2e/test_name_signal_read_chips.py -v --no-cov`); the regression suites green (the re-targeted phase-113/118 pins included).
- Battery (task 06): the three name questions put their canonical doc in the printed suggested top-5; the out-of-KB question's gate verdict is unchanged; full output recorded in `06_…md`; real-model fixture telemetry recorded in `TOOL_CALLING_TESTING.md`.
- Coverage: **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] "How do I deploy gitea/forgejo?" and "How do I set up the gateway?" (1,000-doc replica) put the canonical doc in the suggested top-5 (eval battery, recorded); "Qwen 3.8 27b" incident question and the out-of-KB deflection sanity are unchanged.
- [ ] A grounded prompt's `<documents>` section carries the suggested-folder lines (block markup/body and the LOW prompt byte-identical).
- [ ] `done.sources` = read docs only (a zero-read grounded turn chips nothing; the related row and the durable record are untouched); the unit + E2E pins agree.
- [ ] The new E2E green in isolation; the re-targeted old pins green; `uv run pytest` green; coverage >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — chips (done.sources) = agent-read documents only** (deduped by `(source, path)`, read order); phase-118 A4 (suggested + read) is RETIRED; a zero-read grounded turn shows no chips (owner decision, 2026-09-16, stated consequence).
- **A2 — the name-hit rule**: length-≥4 normalized question token; digit-bearing → prefix of a normalized path part/stem; digitless → exact equality with a path part, stem, or stem sub-component (split on non-alnum); **titles are never name-matched**; ranking (matched-token count, catalog order) — the length tie-break retired; `NAME_HIT_LIMIT` stays 10 (owner-confirmed 2026-09-16, from the measured D1/D2 evidence above).
- **A3 — the name-hit bonus**: `BOR_NAME_HIT_BONUS` default 0.005, additive at selection time only, `0` = byte-identical kill switch, negative fails startup loudly; 118-A3 re-revised (top-5, NO floor, bonus walk); the A8 gate / chunk scores / `query_log.top_score` untouched (owner-confirmed 2026-09-16).
- **A4 — the suggested-folder context**: lines after `SUGGEST_INTRO`, before the first block; ≤5 lines, ≤8 entries + ` +N more` per line; `<document>` block markup/body and the LOW prompt byte-identical (owner-confirmed 2026-09-16).
- **A5 — the cite-discipline sentence** replaces `SUGGEST_INTRO`'s final sentence verbatim as written in D4; `PERSONA` + LOW body + `TOOLS_SECTION` byte-identical (owner-confirmed 2026-09-16).
- **A6 — PLAN.md untouched (Protocol B)**: the 118-A4 retirement and 118-A3 re-revision are recorded here (owner-confirmed) and folded into the anchor table at the next redo (the phase-118 A6 precedent).
## Commit
```bash
git add app/ tests/ scripts/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): component name-hits + bounded name-hit bonus for product-name questions, suggested-folder context in the grounded prompt, and chips that cite read documents only"
```
@@ -0,0 +1,33 @@
# Task 01 — The component name-hit rule (phase-119 D1, LOCKED A2)
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D1 — the 2026-09-16 live finding: product-name questions ("How do I deploy forgejo?") get no lexical name signal because the name-hit mechanism fires only for digit-bearing tokens and `chunks.tsv` indexes body text only.
**Story:** n/a
## Objective
`_name_hit_chunks` in `app/rag/retriever.py` matches question tokens against document **paths** under the new two-class rule (digit-bearing → prefix; digitless → exact component/stem/sub-component), titles never, ranked by (matched-token count, catalog order); `RetrievedChunk` gains a `name_hit` flag.
## Work
1. `app/rag/retriever.py` — `name_hit_tokens(question)`: return ALL normalized whitespace tokens of length ≥ 4 (the existing `_TOKEN_RE` dotted-token handling, `_normalize_name`) **plus** the existing versioned-name join candidates (a purely-numeric second token), order of first appearance, de-duplicated — the digit filter is REMOVED from this function (the digit distinction moves to the match side). Update its docstring: the token list is class-agnostic; matching applies the A2 rule.
2. `app/rag/retriever.py` — `_name_hit_chunks`: for each document (catalog order, the existing single projection query over `(id, source, path, title)` — **title is selected but never matched**), compute the name parts: for every path part — the normalized part, the normalized file stem, and the stem's sub-components (the stem lowercased, split on `[^a-z0-9]+`, each piece normalized, empties dropped). A token matches when:
- it contains a digit → it is a PREFIX of a normalized part or stem (`qwen38` → `qwen3.8-27b-juggernaut-vulkan`);
- it has no digit → it EQUALS a normalized part, stem, or sub-component (`gitea` → the `gitea` folder, `gitea.md`, `kubernetes_gitea`, `gitea-values`).
A document is a name hit on ≥1 matched token. Rank hits by (distinct matched token count DESC, then `(source, path)` catalog order) — the old total-matched-length tie-break is RETIRED (record the reason in the docstring: it outranked 5-char product names by 6-char common tokens). Cap at `NAME_HIT_LIMIT` (10, unchanged). The representative-chunk fetch (`_NAME_HIT_SQL`), the `fts_hit=True`/`cosine=0.0` convention, and the deterministic re-ordering are unchanged; each returned `RetrievedChunk` is constructed with `name_hit=True`.
3. `app/rag/retriever.py` — `RetrievedChunk` dataclass: new field `name_hit: bool = False` (after `is_summary`; the default keeps every existing construction valid — `fuse()`'s `replace()` copies it automatically). Docstring: True for the name-hit representative row (phase 119) — the selection tier's bonus input (task 02).
4. `app/rag/retriever.py` — module docstring: the lexical-list paragraph now describes the two-class component rule (the 2026-09-05 incident fix, extended 2026-09-16 for product names without digits; the title-exclusion precision guard and its measured failure case: `deploy/Deployments/reeseapps/README.md` titled "Deployments" must NOT match the common token "deploy").
5. `tests/unit/test_retriever.py` — the rule, on synthetic `RetrievedChunk`/document inputs where possible and the real query path where it needs the DB (follow the file's existing fixture pattern):
- digitless exact: a token equals a path part (`gitea/` folder), a file stem (`gitea.md`), and a stem sub-component (`kubernetes_gitea`, `gitea-values`, `test-gateway`) — all hit;
- digitless precision: a doc under a `Deployments/` folder with title "Deployments" does NOT hit for token `deploy` (the part normalizes to `deployments` ≠ `deploy`, and titles are never matched);
- digit-bearing prefix: `qwen38` hits `qwen3.8-27b-epic-vulkan.container` and does NOT hit a doc whose stem merely CONTAINS the token mid-word without a prefix (`xqwen38y…`);
- the versioned join still produces `qwen38` from "Qwen 3.8" (existing behavior, now matchable as a prefix token);
- short tokens (<4 normalized) never hit;
- ranking: two-token matches beat one-token matches; catalog order breaks the remainder (the length tie-break is GONE — pin a case where the old ordering would differ);
- the cap (10) and the representative-chunk choice (summary preferred, else chunk 0) are unchanged;
- `name_hit=True` on name-hit rows, `False` on ordinary FTS/vector rows (including after `fuse()`).
## Testing & Quality
- Unit: the cases above; coverage **>90%** on `app/`.
## Completion Criteria
- [ ] "forgejo"-class (digitless) and "qwen38"-class (digit) questions both produce name-hit lexical rows; "deploy" against deployment-titled docs produces none.
- [ ] `name_hit` survives `fuse()` and is False by default everywhere else.
- [ ] `uv run pytest tests/unit/test_retriever.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,31 @@
# Task 02 — The bounded name-hit bonus (phase-119 D2, LOCKED A3)
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D2 — name hits LEADING the lexical list is not enough to lift a weak-vector document into the top-5 (measured: the gateway README stayed out); a bounded selection-time bonus (the phase-106 recency-boost pattern) is the surgical lever.
**Story:** n/a
## Objective
A new setting `name_hit_bonus` (`BOR_NAME_HIT_BONUS`, default **0.005**) and a selection-time re-rank: in `select_suggested`, `select_related`, and `weak_hit_titles` only, a document's effective selection score = best fused chunk score + bonus if any of its chunks carries `name_hit=True`. Chunk scores, `fuse()`, `retrieve()`, the A8 gate, and `query_log.top_score` are untouched; `0` reproduces the pre-phase ordering byte-identically; a negative value fails startup loudly.
## Work
1. `app/config.py` — new `name_hit_bonus: float = 0.005` (env `BOR_NAME_HIT_BONUS`), docstring citing the phase-119 selection re-rank + the phase-106 recency-boost precedent (additive, bounded, single apply site, `0` = off). Validator: `v < 0` → `ValueError` naming the field (the house fail-loud pattern for negative kill switches, e.g. the `agent_max_rounds` precedent); `0` is legal (the kill switch). `.env.example`: an entry with a comment, next to `BOR_RECENCY_BOOST`.
2. `app/rag/retriever.py` — one private helper the three selection walks share, e.g. `_selection_order(chunks, bonus) -> list[tuple[Document, float, float, int]]` returning `(document, effective_score, best_cosine, first_seen_index)` for each distinct document, ordered by `(-effective, -best_cosine, document.path, first_seen_index)`:
- best fused chunk score and best cosine per document = the EXISTING walk semantics (first-seen-chunk rank order is what `select_documents_tiered`/`select_suggested` already use — keep the semantics, not the loop);
- `effective = best + (bonus if any chunk of the doc has name_hit else 0)`;
- `bonus=0.0` (or no name-hit chunks present) → the order is IDENTICAL to the pre-phase walk (pin it, below).
`select_suggested`, `select_related`, `weak_hit_titles` gain a `bonus: float | None = None` parameter (default `get_settings().name_hit_bonus`, the existing settings-read pattern of the `n` parameter) and consume the helper. The `select_documents`/`select_documents_tiered` dormant helpers are UNCHANGED (they pre-date the bonus and are not the selection path).
3. `scripts/eval_retrieval.py` — after the existing per-question top-N document lines, print the seeded suggestion tier: ` suggested: 1. src/path 2. src/path …` from `select_suggested(chunks)` (bonus included, default settings) — the tuning tool must report the tier the prompt actually seeds.
4. `tests/unit/test_config.py` — default 0.005, env override, `0` accepted, negative rejected (names the field).
5. `tests/unit/test_retriever.py`:
- the bonus lifts a name-hit document past a non-name-hit document whose best fused score is higher by less than the bonus (and does NOT lift it when the gap exceeds the bonus);
- the bonus is applied ONCE per document regardless of how many of its chunks are name hits;
- `bonus=0` (and the no-name-hit case) → ordering byte-identical to the pre-phase walk: build a mixed fused list and assert the helper's document order equals the golden order the old loop produced (pin the golden list in the test);
- `select_related` skips excluded ids under the bonus exactly as before; `weak_hit_titles` orders titles by the bonus-adjusted walk;
- the A8 inputs are untouched: chunk `cosine`/`fts_hit`/`score` values are unchanged by the bonus (assert on a fused list before/after — the bonus lives in the selection layer only).
## Testing & Quality
- Unit: the cases above; coverage **>90%** on `app/`.
## Completion Criteria
- [ ] `BOR_NAME_HIT_BONUS` is env-tunable (default 0.005), `0` = byte-identical ordering, negative fails startup naming the field.
- [ ] `eval_retrieval` prints the `suggested:` tier for each question.
- [ ] `uv run pytest tests/unit/test_retriever.py tests/unit/test_config.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,32 @@
# Task 03 — Suggested-folder context in the grounded prompt (phase-119 D3, LOCKED A4)
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D3 — the live turn's 3 `ls` drill-downs walked `deploy → reeseapps → gitea` to find files the prompt's suggestions already pointed next to (the canonical README is the parent-folder sibling of suggested `gitea/app/gitea-web.env.j2`); pre-seed the parent-folder contents so the model reads the right file in round 1.
**Story:** n/a
## Objective
The HIGH prompt's `<documents>` section carries, after `SUGGEST_INTRO` and before the first `<document>` block, one line per distinct parent folder of the suggested documents — the folder's direct children (subfolders as `name/ (N docs)`, files by relative filename, `ls` entry order), capped 8 entries + ` +N more`, at most 5 lines. `<document>` block markup AND body stay byte-identical; the LOW prompt stays byte-identical; empty `folder_lines` ⇒ the phase-118 HIGH prompt byte-identical.
## Work
1. `app/rag/agent.py` — new pure helper `suggested_folder_lines(db: Session, suggested: Sequence[Document], max_lines: int = 5, max_entries: int = 8) -> list[str]` next to the `ls` helpers:
- for each suggested document in order: parent prefix = the path up to (excluding) the last `/` (`""` = source root);
- dedupe by `(source, prefix)` (first suggested doc wins the slot); stop at `max_lines`;
- fetch the source's document rows with `_source_document_rows(db, source)` filtered to the prefix (files: `path == prefix + name`; subfolders: the next segment of `path` after the prefix), and group with the EXISTING `group_folder_listing` semantics (subfolder names + recursive doc counts, file names; entry order = the `ls` folder-level rendering order — the line must read the same as the model's own `ls` output);
- exclude the suggested document itself from its folder's entries;
- render: `<source>/<prefix>/: e1, e2, …` (source root: `<source>/: …`); subfolder entries `name/ (N docs)` (singular `(1 doc)`), file entries by relative filename; at most `max_entries` entries then ` +N more` (N = the remaining count).
2. `app/rag/prompts.py` — `build_high_prompt(documents, notes=None, kb_overview=None, folder_lines: Sequence[str] = ())`: when blocks are present, the `<documents>` body is `SUGGEST_INTRO`, then the folder lines (each on its own line, immediately after the intro line), a blank line, then the first `<document>` block. NO new markup/tag — plain lines (the E2E mock keys off the `<documents>` marker and the LAST block's tail; plain lines before the blocks are inert). `folder_lines` empty ⇒ output byte-identical to the phase-118 shape (pin it). The LOW prompt and `build_deflect_prompt` are untouched.
3. `app/api/chat.py` — `plan_turn` gains `folder_lines: Sequence[str] = ()` (passed through to `build_high_prompt` in the HIGH branch; the LOW branch ignores it — the deflected prompt stays byte-identical). In the chat endpoint's HIGH branch (where the DB session is in scope, ~the `plan_turn` call site): before `plan_turn`, compute `suggested = select_suggested(chunks, n=settings.suggested_docs)` (the same deterministic walk `plan_turn` performs internally — one extra cheap walk, no shared state) and `folder_lines = suggested_folder_lines(db, suggested)`; pass `folder_lines` into `plan_turn`. The `TurnPlan`, the durable record, and the per-turn log line are unchanged.
4. `tests/unit/test_agent.py` (or a new `tests/unit/test_folder_context.py` — follow the suite that already imports the `ls` helpers) — `suggested_folder_lines`:
- source-root doc → `<source>/: …` line with the source's top level;
- nested doc → the parent folder line; children correct (subfolder counts recursive, file names relative);
- the suggested doc is excluded from its own folder's entries;
- dedupe: two suggested docs in the same folder ⇒ one line (first doc's position); `max_lines` caps the list;
- `max_entries` cap + ` +N more` (exact remainder); the `(1 doc)` singular;
- `ls` entry order (pin a mixed folder/file case against the `group_folder_listing` order).
5. `tests/unit/test_prompts.py` — the folder-line placement (after `SUGGEST_INTRO`, before the first block, exact line shape); empty `folder_lines` ⇒ byte-identical to the phase-118 prompt (the existing pin keeps passing); the LOW prompt pin is untouched and still passes; the `<document>` block markup/body pins still pass unchanged.
## Testing & Quality
- Unit: the cases above; coverage **>90%** on `app/`.
## Completion Criteria
- [ ] A HIGH prompt for 5 suggested docs spanning 3 folders carries exactly 3 folder lines, correctly shaped; a zero-folder-lines build is byte-identical to phase 118.
- [ ] `uv run pytest tests/unit/test_agent.py tests/unit/test_prompts.py tests/unit/test_chat_gate.py -v` green (or the equivalent split); full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,25 @@
# Task 04 — Cite-discipline copy in the suggestion intro (phase-119 D4, LOCKED A5)
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D4 — the live turn's answer cited `gitea-web.env.j2` in its "Docs used:" line although the agent never read it: the model confabulates usage from the suggestion blocks that sit in context. The intro's loose "cite what you used" is replaced with a discipline sentence.
**Story:** n/a
## Objective
`SUGGEST_INTRO`'s final sentence is replaced verbatim with the cite-discipline sentence (LOCKED A5); every byte pin of the old sentence re-cuts in the same change; `PERSONA`, the LOW/`DEFLECT_MODE` body, and `TOOLS_SECTION` stay byte-identical.
## Work
1. `app/rag/prompts.py` — `SUGGEST_INTRO` constant: replace the final sentence `Cite the document(s) you used, by path.` with:
`Cite only the document(s) you read — or, if you answered from a suggested summary without reading it, cite that suggested document — never a document you neither read nor used.`
The rest of the constant (the start-here framing, the `read` pointer, the "its full text is not in the prompt until you read it" clause) is untouched. Update the constant's docstring: the phase-119 cite discipline (the live confabulation it closes; the phase-118 sentence retired).
2. Re-cut the pins (audit with `grep -rn "Cite the document(s) you used" tests/`):
- `tests/unit/test_prompts.py` — any assertion that pins the old final sentence (exact-string or sha/length) re-points at the new sentence; the intro's PREFIX (start-here framing) pin, if separate, is untouched;
- `tests/unit/test_prompt_lock.py` — if `SUGGEST_INTRO` (or a prompt built from it) is byte-pinned (sha256/length/prefix/suffix anchors), re-cut those anchors in this same change (the phase-118 TOOLS_SECTION re-cut precedent); `TOOLS_SECTION`, `PERSONA`, and the LOW body anchors MUST stay byte-identical — assert they are untouched;
- any E2E that greps the old sentence in the rendered stream or mock output (audit `tests/e2e/`) — re-target or confirm none exists (the mock keys off markers, not this sentence — expected: none).
3. `tests/unit/test_prompts.py` — new pin: the HIGH prompt carries the discipline sentence exactly once, inside `<documents>` after the intro; the sentence is absent from the LOW prompt (byte-identical pin passes).
## Testing & Quality
- Unit: the pins above; coverage **>90%** on `app/`.
## Completion Criteria
- [ ] The HIGH prompt carries the LOCKED A5 sentence verbatim; `PERSONA`, LOW body, and `TOOLS_SECTION` byte-pins pass unchanged.
- [ ] No test in `tests/` references the retired sentence.
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_prompt_lock.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,30 @@
# Task 05 — Chips cite read documents only (phase-119 D5, LOCKED A1)
**Phase:** `119_name_signal_read_chips` · **Source:** owner decision 2026-09-16 ("I'd like the chips to be read/used docs only") — phase-118 A4 (`done.sources` = suggested + read) is retired; the live turn's 7 chips (5 noisy suggestions + 2 reads) are the symptom.
**Story:** n/a
## Objective
`done.sources` on grounded turns = `holder.read_docs` only (deduped by `(source, path)`, read order preserved); a zero-read grounded turn chips nothing. The durable record (`query_log.sources` + the per-turn log line = suggested + related + read), the `related` tier, and the deflected turn's `[]` are untouched. No frontend change.
## Work
1. `app/api/chat.py` — the done-frame section (the "4. Durable record" block):
- `cited_docs` = `holder.read_docs` deduped by `(source, path)`, order preserved — `plan.suggested_docs` is REMOVED from the citation walk (the `cited_seen` set keeps its role: it now collects read docs only);
- `record_docs` (the durable record) and `source_paths` are UNCHANGED (suggested + related + read, deduped — 118-A3 stands);
- `related_refs` dedupe stays `if (d.source, d.path) not in cited_seen` — a read related doc is a chip, never a nearby doc (unchanged intent, now keyed on read docs);
- `ChatDoneEvent` construction and every SSE frame shape are unchanged; the deflected branch still yields `sources=[]`.
2. `app/api/chat.py` — module docstring + the done-section comment: the citation surface is now READ DOCS ONLY (phase 119, LOCKED A1, owner decision 2026-09-16; phase-118 A4 retired; the zero-read grounded turn chips nothing — an accepted consequence, the answer prose names the doc per the phase-119 cite discipline; the durable record still records retrieval, not citations).
3. `tests/unit/test_chat_gate.py` — re-target the done-frame pins (audit the file's `done`/`sources` assertions):
- grounded turn, agent read two docs ⇒ `done.sources` == exactly those two, read order, deduped;
- grounded turn, agent read NOTHING (a summary-only answer) ⇒ `done.sources == []` (the new A1 pin — replaces the phase-118 "always chips" pin if present);
- suggested docs that were never read appear NOWHERE in `done.sources` (explicit contrast pin against the retired A4 union);
- a related doc that was read ⇒ in `done.sources`, NOT in `done.related`;
- the durable record pins (suggested + related + read, deduped) and the `suggested=N` log-line pin are untouched and still pass;
- the deflected turn's `sources == []` pin is untouched.
4. Frontend: NO change (audit only) — confirm `appendSources` (`frontend/assets/app.js`) and `addSources` (`frontend/assets/shared.js`) both no-op on an empty `sources` array (they do: `if (!sources || !sources.length) return;`) and that no view asserts ≥1 chip on a grounded turn (e.g. no `aria`/label assumption that breaks on an absent meta row — the related row and action buttons occupy `.msg-meta` independently, per the phase-113 layout pins).
## Testing & Quality
- Unit: the pins above; coverage **>90%** on `app/`.
## Completion Criteria
- [ ] A grounded done frame carries exactly the agent-read docs (a zero-read turn carries none); the durable record and the related tier behave as pinned.
- [ ] `uv run pytest tests/unit/test_chat_gate.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,40 @@
# Task 06 — Dedicated E2E, retired pins, mock echo, and the verification battery (phase-119 D6)
**Phase:** `119_name_signal_read_chips` · **Source:** phase overview D6 — the phase is complete only when the new behavior is pinned E2E (in isolation), the retired-A4 pins are re-targeted, the 1,000-doc replica battery passes, and the real-model telemetry is recorded.
**Story:** n/a
## Objective
`tests/e2e/test_name_signal_read_chips.py` (green in isolation) pins the four behaviors — name-signal suggestions, chips = read docs only, the zero-read no-chip consequence, and the suggested-folder lines (via a new mock trigger echo); every E2E pin of the retired phase-118 A4 chip contract is re-targeted; the replica battery + real-model run are recorded.
## Work
1. `tests/e2e/mock_llm.py` — new trigger (follow the existing trigger conventions exactly: lowercase phrase, checked where its specificity warrants, no existing E2E question or fixture file contains it — verify with a repo-wide grep before choosing):
- user message containing **`repeat your folder map`** AND system prompt containing `<documents>` → the composed answer ends with ` (folders: <line 1>; <line 2>; …)` echoing the suggested-folder lines VERBATIM (the plain lines between `SUGGEST_INTRO` and the first `<document ` in the `<documents>` section; joined with `; `). If no folder lines are present, the echo is omitted (the answer is otherwise unchanged). This is the same prompt-injection-echo convention as the `<tuning>` / `<knowledge_base>` triggers.
- Header docstring: a section for the new trigger (what it keys on, what it echoes, why — phase 119 D3 observability). No existing trigger or marker changes; the tail-echo (`show the end of your notes`), the tool flows, and all other behaviors stay byte-identical.
2. Fixture design (in the new suite, following `test_summary_seed_context.py`'s local-directory-source pattern — seed dirs + `/api/git-sources` registration + sync; per-module truncate/re-import so the suite runs in ISOLATION):
- ~6 markdown files across 2–3 folders of one local source; one file under a DISTINCTIVE digitless path component (e.g. `forgejo/forgejo-install.md`) whose BODY has minimal overlap with the battery question words (its name is its main signal); the other suggested-tier files' bodies overlap the question so they out-rank it on vector/FTS.
- Invariant to engineer (assert, don't assume): for the question `<distinctive-token> … show the end of your notes`, the name-hit document is the **LAST** of the top-5 suggested (its `Source: <source>/<path>` tail line is the tail echo). Tune file contents until the tail-echo assertion passes; verify determinism by running the suite twice.
3. `tests/e2e/test_name_signal_read_chips.py` — the four assertion groups (admin session, SSE `done` frame via the existing chat helpers, chip DOM `.source-chip`):
- **(a) name signal → suggestion:** the distinctive-token question + `show the end of your notes` ⇒ the rendered answer contains `Source: <source>/<name-hit path>` (the name-hit document was the last suggested — the digitless component rule + bonus put it in the tier);
- **(b) chips = read docs only:** the `use your tools` flow (the mock's single-read: ls → drill → read first file line ⇒ answer `Read <sp>. …`) ⇒ the bubble carries exactly ONE `.source-chip`, `<source>/<read path>`; none of the five suggested docs chips; the de-emphasized related row (`NEARBY DOCS, IN CASE:`) still renders when the fixture's rank-6+ tier is non-empty (phase-113 behavior untouched);
- **(c) zero-read grounded turn chips nothing:** a plain question (no tool trigger) ⇒ `done` frame `sources == []` and ZERO `.source-chip` elements in the bubble (LOCKED A1's visible consequence); the answer still renders;
- **(d) suggested-folder lines:** the distinctive-token question + `repeat your folder map` ⇒ the answer contains the exact folder line for the name-hit document's folder (`<source>/<folder>/: …` with its real entries — assert the line prefix and one known sibling entry).
4. Retarget the retired-A4 pins (audit rule for every hit of `grep -rn "done\[.sources.\]\|source-chip" tests/e2e/`: asserted set == the docs the flow actually READ ⇒ keep; asserted set contains a suggested/cited-but-unread doc ⇒ re-target to the read set with a phase-119 A1 comment; related-row assertions ⇒ keep):
- `tests/e2e/test_summary_seed_context.py` — the `done["sources"] == [(SOURCE, p) for p in SUGGESTED_TAIL]` pin (the no-read flow) ⇒ `[]`; the `… == [(SOURCE, p) for p in SUGGESTED_READ]` pin (the read flow) ⇒ the read set only; the durable-record pins (`row.sources == … SUGGESTED_TAIL + RELATED_PATHS`) STAY (118-A3 unchanged);
- `tests/e2e/test_source_chip_quality.py` (phase 113) — the cited-tier chip pins re-target per the audit rule (the related-row pins stay);
- `tests/e2e/test_agent_document_tools.py`, `test_agent_unlimited_tools.py`, `test_read_truncation_cap.py`, `test_search_tool.py`, `test_retrieval_quality.py`, `test_share_chat.py` — audit per the rule (read-flow chip assertions should mostly pass unchanged; the share page re-renders `done` sources, so its pins follow the same rule).
- `tests/unit/test_chat_gate.py` was already re-targeted in task 05 — confirm no E2E-side twin remains.
5. Battery (record in THIS file — the phase-118 gate-record precedent):
- Precondition: the 1,000-doc live replica in the app DB (the `restore-test-db` skill; verify the fingerprint `docs=1000, chunks=8866`; the dump `data/bor_live_kb_replica.dump.sql` is a local gitignored artifact — if missing, STOP and ask the owner; NEVER fabricate or reuse stale numbers). Note: the replica predates the phase-118 markdown-summary backfill (markdown `summary` NULL, no markdown `is_summary` chunks) — the overview's measured numbers are the same-state baseline; report that fact with the numbers.
- `tests/fixtures/retrieval_battery.txt` (new, tracked): one question per line, `#` comment lines allowed (the reader skips them — NO inline comments, `scripts/eval_retrieval.py` reads the whole line): the three phase-119 name questions (canonical docs noted in comment lines), the 2026-09-05 incident regression (`Do you have notes on the qwen 3.8 27b model?` — a `qwen3.8-27b` quadlet stays in the suggested top-5), and the deflection sanity (`What is the capital of Mongolia?` — zero name hits in the KB; the gate verdict and fts count must match the pre-phase replica run recorded in the phase overview: `HIGH`, fts=5 — the name-hit change must not move it).
- Run `uv run python -m scripts/eval_retrieval --from-file tests/fixtures/retrieval_battery.txt`; **gate:** each of the three name questions' `suggested:` top-5 contains its canonical doc (the overview's post-phase numbers), the incident doc stays top-5, and the Mongolia verdict/fts is unchanged. Paste the full output into this file under "Battery run 2026-09-…" with the verdict line.
- Real-model telemetry (NOT a gate — no `AGENT_TOOLS`/`TOOLS_SECTION` copy changed, so the tool-copy gate is not re-triggered; record for the owner): `uv run python -m scripts.agent_realmodel_check --restore --mode fixture`; append a phase-119 TELEMETRY section to `TOOL_CALLING_TESTING.md` (the phase-118 format: the run output, the four conditions read under the phase-118 A7 semantics, per-turn reading — highlight any change in `ls`-drill-down frequency on the read turns, the phase's intended latency effect).
6. Housekeeping: `TOOL_CALLING_TESTING.md` section header notes it is telemetry-only; the phase dir is otherwise untouched by this task.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_name_signal_read_chips.py -v --no-cov` green **in isolation** (DB up: `podman compose up -d db`); the re-targeted suites green (run each re-targeted file in isolation — the suites truncate/re-import and must not share state); full `uv run pytest` green; coverage **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] The new suite green in isolation: (a) the name-hit doc is suggested (tail-echo sentinel), (b) chips == read docs only with the related row intact, (c) a zero-read grounded turn chips nothing, (d) the folder line is echoed verbatim.
- [ ] No E2E pin asserts a suggested/cited-but-unread doc as a chip; the durable-record pins are untouched.
- [ ] The battery gate passes and the full output + the real-model telemetry are recorded (this file + `TOOL_CALLING_TESTING.md`).
- [ ] `uv run pytest` green; coverage >90%; lint + types clean.
@@ -0,0 +1,51 @@
# Phase 120 — Failed-turn retry: network errors and refresh survive a failed turn
**Source:** `TODO.md` L3–4 — "Retry doesn't seem to work on network error" + "Refreshing the page after an error shows only the chat message you sent and no options to retry the message, forcing the user to click 'new chat' or be stuck."
**Story:** n/a (bug-fix follow-up; extends the phase-49/53 redo-in-place retry, phase-67 LLM retry, and phase-111 banner Retry assets).
**Context:** `frontend/assets/app.js` — `showErrorBanner(detail, opts)` (L2210) reveals `#banner-retry` only when `opts.retryable && lastBrainWrap` (L2222); `retryLastTurn(wrap)` (L2276) pops the LAST brain record and re-asks the user question immediately before it (the invariant `every brain record follows its user record`); `rememberBrainTurn(rawText, meta, replaceIndex)` (L2108) pushes/replaces the brain record in `conversation` + `saveConversation()` + `persistConversation()` (the phase-55 auto-save rides the same call). `lastBrainWrap` is assigned only on three paths: the `done` settle (L2609), the zero-frame fallback bubble (L2671), and the user-stop finalize (L2699) — **never on a turn error**. The error catch (the `else` branch at ~L2690) calls `setUiState(UI_STATE.error, detail, { hint })` with NO brain record persisted, whether or not a partial `wrap` exists. The stream-drop guard (~L2651, `!sawDone && !aborted && (acc || thinkingAcc)`) also lands in the error state with nothing persisted. Restore: `renderStoredMessage(m)` (L1642) renders `m.stopped` via `appendStoppedNote` (L486); the restore loop sets `lastBrainWrap` on the last restored brain bubble (L1694) and calls `markLastRetryable()` (L1708, L560 — removes all `.retry-btn`, re-adds on the LAST `.brain-wrap`). `app/schemas.py` — `ChatMessage` (L742) is `extra="forbid"` with fields `who`, `text` (≤32 000), `sources?`, `related?`, `deflected?`, `suggestions?`, `thinking?` (≤32 000), `tools?`, `stopped?: bool | None` (L786); `SavedChatCreate/Update` messages are non-empty, ≤200 (phase 83). `tests/e2e/mock_llm.py` + `tests/e2e/test_llm_retry.py` hold the existing LLM-failure mock pattern for the E2E.
## Objective
A failed chat turn — network error (zero frames), SSE `error` frame, or mid-stream drop — leaves a **retryable error state** both live (the banner Retry and an in-bubble Retry both work) and after a page refresh (the failed turn restores as an error bubble with a working Retry button). No failed turn strands the user with a bare question and no recovery.
## Dependencies
- `119_name_signal_read_chips` (complete) — pipeline predecessor (execution order) only.
- Code dependencies (all complete): phase 49/53 `retryLastTurn` redo-in-place, phase 111 `#banner-retry`, phase 48 `stopped` persistence + `appendStoppedNote` pattern, phase 55 auto-save riding `rememberBrainTurn`.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Failed record (task 01, server side):** two new OPTIONAL fields on `ChatMessage`, the phase-48 `stopped` precedent (L786): `failed: bool | None = None` and `error: str | None = Field(default=None, max_length=500)` (the persisted error detail; 500 caps a hostile detail string in the phase-83 style). `extra="forbid"` stays — the keys are now declared, unknown keys still 422. No change in `app/api/chats.py` logic (the schema flows through `SavedChatCreate/Update`); the shared-chat shape (`SharedChatOut.messages`) carries failed records verbatim (text renders as-is on the shared page — no change needed there).
- **Failed turn = a brain record (LOCKED A1):** a failed turn persists `{ who: "brain", text: <detail or fallback>, failed: true, error: <detail> }` via `rememberBrainTurn` — so it lands in localStorage AND the server-side saved chat through the existing phase-55 auto-save ride. There is no separate error table and no new API: `retryLastTurn`'s pop-the-last-brain-record-then-re-ask-the-preceding-question logic works on a failed record UNCHANGED (the invariant holds — the question's user record immediately precedes it).
- **Live error paths (task 01, frontend):** the error catch's `else` branch (non-abort, non-stop) and the stream-drop guard BOTH funnel into one new helper `finalizeFailedTurn(detail, { acc, thinking, tools })`:
- **Partial exists** (`wrap` with streamed text): close the thinking block + tool calls (the stop-finalize pattern), `appendFailedNote(wrap, detail)` (new, mirrors `appendStoppedNote` L486 — an in-bubble error line with the detail), persist via `rememberBrainTurn(acc, { thinking, tools, failed: true, error: detail }, leavePartialIndex)`, `lastBrainWrap = wrap`.
- **No wrap** (network error, zero frames): create a brain bubble with a fixed fallback text (a short honest "my answer didn't make it" line — NOT the `EMPTY_ANSWER_FALLBACK` answer text; the `appendFailedNote` carries the real detail), persist the same record shape, `lastBrainWrap = fwrap`.
- Then `markLastRetryable()` — the in-bubble Retry button appears, and `showErrorBanner`'s existing `opts.retryable && lastBrainWrap` condition (L2222) now holds on a turn error, so the phase-111 banner Retry appears too — **no change to `showErrorBanner`** (it binds `() => retryLastTurn(lastBrainWrap)` at reveal; `lastBrainWrap` is set before `setUiState(UI_STATE.error, …)` runs).
- The zero-frame-but-stream-completed case keeps its existing fallback bubble (L2663–2673) — now ALSO marked `failed: true` + error note (it is a failed turn; the bubble text stays `EMPTY_ANSWER_FALLBACK` so the record keeps a meaningful `text`).
- **Restore (task 02):** `renderStoredMessage(m)` gains the failed branch — a `m.failed` record renders as a brain bubble (the persisted `text`), gets `appendFailedNote(wrap, m.error)`, and gets NO Save-as-doc button (a note, not an answer — the `m.stopped` exclusion at L1687 precedent: `if (!m.stopped && !m.failed) appendSaveAsDocButton(…)`). No other restore change is required: the restore loop's `lastBrainWrap = wrap` (L1694) + `markLastRetryable()` (L1708) already target the last `.brain-wrap`, which is now the failed bubble → the in-bubble Retry button renders on refresh. `retryLastTurn` needs no change (the failed record is the last brain record; its preceding user record is the question).
- **Interaction with `stopped`:** a turn is either stopped (user engaged, partial kept, `stopped: true`) or failed (`failed: true`) — mutually exclusive by construction (the stop path is the catch's `stoppedByUser`/`AbortError` branch, which this phase does not touch).
- **NOT touched:** `retryLastTurn` itself, the stop path, the done path, the server save/restore API logic (schema fields only), the shared page rendering, and every non-chat `showErrorBanner` caller.
## Tasks
1. `01_persist_failed_turn.md` — `ChatMessage.failed`/`error` fields + the live error paths persist a failed brain record with a rendered error bubble (banner Retry works on network errors).
2. `02_restore_failed_turn.md` — restore renders a `failed` record as an error bubble with a working Retry button (the refresh case).
3. `03_failed_turn_tests.md` — unit + integration + isolated E2E `test_failed_turn_retry.py`.
## Testing & Quality
- Unit: `tests/unit/test_chat_message_failed.py` (new, task 03) — `ChatMessage` accepts `failed`/`error`, `error` >500 chars 422s, unknown keys still 422, omitted keys round-trip `None`; `tests/unit/test_frontend_failed_turn.py` (new, task 03) — house-style source assertions: the error catch + stream-drop guard route through the failed-turn finalize (persist `failed: true`, call `markLastRetryable`), `appendFailedNote` exists and mirrors the stopped-note structure, the restore branch renders the note and excludes Save-as-doc, `showErrorBanner` is byte-unchanged (the `lastBrainWrap` condition untouched).
- Integration: `tests/integration/test_chats_api.py` (extend) — `POST`/`PUT /api/chats` with a `failed: true` + `error` record round-trips byte-identically (the phase-50 contract); a shared chat carrying a failed record still serves (public shape unchanged).
- E2E: `tests/e2e/test_failed_turn_retry.py` (new, task 03) — run in isolation per AGENTS.md §4. Scenarios (the `mock_llm.py` failure pattern from `test_llm_retry.py`): (A) network-class failure (zero frames) → banner with a visible Retry → click re-asks without re-typing; (B) SSE `error` frame after partial deltas → partial bubble keeps its text + error note + Retry → click re-asks; (C) reload the page after a failed turn → the failed bubble restores with a working Retry button → click re-asks.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] A network-error turn shows a Retry (banner and/or in-bubble); clicking it re-asks the last question without re-typing.
- [ ] Reloading the page after a failed turn shows the failed bubble (with the error detail) and a working Retry — no "new chat" required.
- [ ] Stopped turns (phase 48) and successful turns behave byte-identically to before.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **A1 — a failed turn persists as a brain record with a `failed` marker (+ capped `error` detail), the phase-48 `stopped` precedent; no separate error table, no new API, `retryLastTurn` reused unchanged (owner-confirmed 2026-09-24, roadmap confirmation).**
- **Banner Retry stays as-is** — the phase-111 `opts.retryable && lastBrainWrap` condition is kept; this phase makes `lastBrainWrap` exist on the error paths so the existing button finally appears (owner-confirmed: same mechanism, no `showErrorBanner` change).
## Commit
```bash
git add app/ frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(chat): persist failed turns so retry works on network errors and survives a refresh"
```
@@ -0,0 +1,37 @@
# Task 01 — Persist the failed turn: schema fields + live error paths
**Phase:** `120_failed_turn_retry` · **Source:** `TODO.md:3–4` — "Retry doesn't seem to work on network error" + "Refreshing the page after an error shows only the chat message you sent and no options to retry the message, forcing the user to click 'new chat' or be stuck."
## Objective
Every failed chat turn (network error, SSE `error` frame, stream drop) persists a `failed: true` brain record and renders a retryable error bubble live — so the phase-111 banner Retry finally appears on network errors (its `lastBrainWrap` precondition now holds).
## Work
1. `app/schemas.py` — `ChatMessage` (L742, `extra="forbid"`): add, next to `stopped` (L786),
```python
failed: bool | None = None
error: str | None = Field(default=None, max_length=500)
```
Extend the docstring: the phase-48 `stopped` precedent — a FAILED turn (network/SSE error/stream drop) stores `{who: "brain", text: <detail or fallback>, failed: true, error: <detail>}`; `error` is the persisted banner detail, capped at 500 (phase-83 value-bounds style). No serializer change — `None` values flow as absent/None exactly like `stopped` today (the phase-50 byte-identical round-trip contract covers the new keys automatically).
2. `frontend/assets/app.js`:
- New `appendFailedNote(wrap, detail)` — mirror of `appendStoppedNote` (L486): one `.failed-note` per bubble (guard query), a visible error line inside the brain bubble carrying `detail` (the banner keeps its role="alert" summary; the note is the in-bubble, refresh-surviving copy).
- New `finalizeFailedTurn(detail, { acc, thinking, tools, wrap, leavePartialIndex })` — the single funnel for every non-stop, non-abort turn failure:
- `wrap` exists (partial streamed): `closeThinkingBlock(wrap)` + `closeToolCalls(wrap)` (the stop-finalize pattern, ~L2685), `appendFailedNote(wrap, detail)`, `rememberBrainTurn(acc, { thinking: thinking || undefined, tools: tools.length ? tools : undefined, failed: true, error: detail }, leavePartialIndex)`, `lastBrainWrap = wrap`.
- no `wrap` (network error, zero frames): `fwrap = addMessage("brain", FAILED_TURN_TEXT)` where `FAILED_TURN_TEXT` is a new short honest constant ("My answer didn't make it — the connection dropped. Use Retry to ask again.") — NOT `EMPTY_ANSWER_FALLBACK` (that constant stays for the zero-frame-but-completed case); `appendFailedNote(fwrap, detail)`, `rememberBrainTurn(FAILED_TURN_TEXT, { failed: true, error: detail }, leavePartialIndex)`, `lastBrainWrap = fwrap`.
- end with `markLastRetryable()`.
- `detail` is the trimmed error string, truncated to 500 chars before persistence (the schema cap is the backstop).
- The error catch's `else` branch (~L2690, currently `setUiState(UI_STATE.error, detail, { hint })` with no persistence): call `finalizeFailedTurn(detail, {…})` BEFORE `setUiState(UI_STATE.error, detail, err.hint ? { hint: err.hint } : {})` (the banner stays — now with its Retry revealed because `lastBrainWrap` is set).
- The stream-drop guard (~L2651, `!sawDone && !aborted && (acc || thinkingAcc)` → currently a bare `setUiState(UI_STATE.error, "The stream ended before my answer finished — try again?")`): route through the same `finalizeFailedTurn` with that detail (the partial persists as failed — a half-answer is a failed answer, and Refresh must restore what the user saw + a Retry).
- The zero-frame-but-completed fallback bubble (L2663–2673): add `failed: true` + `error: "The model answered with nothing."` to its `rememberBrainTurn` call and `appendFailedNote(fwrap, …)` — the bubble text stays `EMPTY_ANSWER_FALLBACK`.
- Do NOT touch: `showErrorBanner` (L2210), `retryLastTurn` (L2276), the stop branch, the done settle (L2609), `markLastRetryable` (L560).
3. `frontend/assets/styles.css` — `.failed-note`: the in-bubble error line treatment (the `.stopped-note` family, error-colored per the current theme's error token — contrast ≥4.5:1, PLAN §7).
4. ASSUMPTION: the zero-frame-but-completed case (stream returns, no events, no throw) is also marked failed — it is a failed turn, and its record previously persisted with no marker (inconsistent with the refresh case this phase fixes).
## Testing & Quality
- Unit: `tests/unit/test_chat_message_failed.py` (shipped with task 03's test task — this task ships the code): `ChatMessage` accepts `failed: true` + `error`; `error` >500 chars → 422; an unknown key still → 422; a record without the new keys is byte-identical to before. `tests/unit/test_frontend_failed_turn.py` (task 03): the catch `else` branch + stream-drop guard + zero-frame fallback all route through the failed finalize (persist `failed: true`, call `markLastRetryable`); `appendFailedNote` exists; `showErrorBanner` and `retryLastTurn` sources are untouched (byte-pinned).
- Coverage: **>90%** on `app/` for the schema change (the frontend JS is pinned by source-assertion unit tests).
## Completion Criteria
- [ ] `ChatMessage` round-trips `failed`/`error` (unit tests green).
- [ ] A zero-frame network error (E2E scenario A, task 03) shows the banner WITH a visible Retry button and a failed bubble with the error detail.
- [ ] No call site outside the three failed paths persists `failed: true` (grep).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,25 @@
# Task 02 — Restore the failed turn: refresh keeps the Retry
**Phase:** `120_failed_turn_retry` · **Source:** `TODO.md:4` — "Refreshing the page after an error shows only the chat message you sent and no options to retry the message, forcing the user to click 'new chat' or be stuck."
## Objective
A `failed` record restores as an error bubble (persisted text + error note) that carries a working Retry button — the refreshed page is the same retryable state the live error was, and Retry re-asks the question through the unchanged `retryLastTurn`.
## Work
1. `frontend/assets/app.js` — `renderStoredMessage(m)` (L1642): add the failed branch, mirroring the `m.stopped` handling (L1685–1688):
- a `m.failed` brain record renders its `text` (the persisted detail or fallback line) as the bubble content, then `appendFailedNote(wrap, m.error)` (the note is absent when `m.error` is null — the record's `text` already carries it), and
- `if (!m.stopped && !m.failed) appendSaveAsDocButton(wrap, m.text);` — a failed turn is a note, not an answer (the stopped exclusion precedent at L1687).
- No Tune button for failed records either (a note, not an answer — same scope as the `m.stopped` exclusion around L584–588 if the restore call site applies it there).
- No other restore change: the restore loop already sets `lastBrainWrap = wrap` on the last restored brain bubble (L1694) and calls `markLastRetryable()` (L1708), which removes every `.retry-btn` and re-adds it on the LAST `.brain-wrap` — the failed bubble. `retryLastTurn` works unchanged: the failed record is the last brain record, the user record immediately before it is the question (the invariant holds by construction — task 01 persists the brain record right after the user record), so the redo-in-place pops the failed record and re-asks.
2. `frontend/assets/app.js` — the shared-chat restore (`frontend/assets/shared.js` / the shared page): failed records render their `text` as a plain brain bubble (no note, no Retry — the shared view is read-only and text-only by design; no change beyond confirming the `renderStoredMessage`-equivalent there does not choke on the unknown-looking `failed`/`error` keys — it renders `text` only).
3. ASSUMPTION: a failed bubble restored at the END of the conversation gets the Retry; a failed bubble in the MIDDLE of a longer conversation does not (the phase-49 last-bubble-only rule, unchanged).
## Testing & Quality
- Unit: `tests/unit/test_frontend_failed_turn.py` (task 03): the restore branch renders the failed note, excludes Save-as-doc (and Tune, where applicable), and the restore path is the only place `m.failed` is read for rendering; the shared page renders failed records text-only.
- E2E: scenario C of `tests/e2e/test_failed_turn_retry.py` (task 03) pins this task end-to-end.
- Coverage: n/a (frontend) — the validate.sh `app/` gate must stay green.
## Completion Criteria
- [ ] Reload after a failed turn (E2E scenario C): the failed bubble shows with its error detail and a Retry button; clicking Retry re-asks the preceding question without re-typing and the failed record is replaced by the new answer.
- [ ] A mid-conversation failed record restores with NO Retry button (last-bubble-only rule intact).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,34 @@
# Task 03 — Failed-turn tests: unit + integration + isolated E2E
**Phase:** `120_failed_turn_retry` · **Source:** `TODO.md:3–4` — "Retry doesn't seem to work on network error" + "Refreshing the page after an error shows only the chat message you sent and no options to retry the message …"
## Objective
Pin the whole failed-turn contract: the schema boundary, the save/restore round-trip, the three live failure paths, and the two user scenarios (retry on network error; retry after refresh) as an isolated Playwright suite.
## Work
1. `tests/unit/test_chat_message_failed.py` (new):
- `ChatMessage` accepts `{who: "brain", text: "…", failed: true, error: "detail"}`; `error` of 501 chars → 422; an unknown key (e.g. `"foo": 1`) → 422 (`extra="forbid"` intact); a record WITHOUT the new keys serializes byte-identically to a pre-phase record (the phase-50 contract).
2. `tests/unit/test_frontend_failed_turn.py` (new) — house-style source assertions (the phase-111 `test_frontend_banner_retry.py` pattern):
- the error catch `else` branch, the stream-drop guard, and the zero-frame fallback bubble all persist `failed: true` (grep the three sites for the `failed: true` persist) and each funnel path ends with `markLastRetryable`;
- `appendFailedNote` exists and guards against duplicates (one `.failed-note` per bubble);
- `FAILED_TURN_TEXT` is a distinct constant (not `EMPTY_ANSWER_FALLBACK`);
- `showErrorBanner` and `retryLastTurn` are byte-unchanged (pin their source — the phase's explicit "NOT touched" contract);
- the restore branch renders `m.failed` (note + Save-as-doc exclusion).
3. `tests/integration/test_chats_api.py` (extend):
- `POST /api/chats` with a brain record `{text, failed: true, error: "…"}` returns it byte-identically; `PUT` re-Save round-trips it; `error` >500 chars → 422;
- a shared chat (the phase-51 `share` path) carrying a failed record still serves `GET /api/shared/{token}` (public shape — `title` + `messages` — unchanged).
4. `tests/e2e/test_failed_turn_retry.py` (new — ONE file, run in isolation per AGENTS.md §4: `uv run pytest tests/e2e/test_failed_turn_retry.py -v --no-cov`), reusing `tests/e2e/mock_llm.py`'s failure pattern from `tests/e2e/test_llm_retry.py`:
- **A — network error:** mock the chat endpoint to fail with ZERO frames (connection reset / immediate close — the same failure `test_llm_retry.py` exercises past its retry budget, or a hard 500 if the mock supports it) → assert: the banner is visible with a working Retry button AND a failed bubble with the error detail exists → click banner Retry → the question is re-asked (mock now succeeds) → a grounded answer renders and the failed bubble is gone.
- **B — SSE error frame with partial:** mock streams some `delta` frames then an `error` frame → assert: the partial bubble KEEPS its streamed text + shows the error note + carries the in-bubble Retry → click it → re-asked in place (redo-in-place: the failed record is replaced by the new answer).
- **C — refresh:** fail a turn (as in A) → `page.reload()` → assert: the question + the failed bubble restore (error detail visible) + the Retry button is present on the failed bubble → click it → re-asked → answer renders.
- Negative: a STOPPED turn (user Stop) still restores with the "Answer stopped." note and NOT a failed note (phase 48 unchanged).
5. Run the full gate: `uv run pytest` (unit + integration), `uv run pytest --cov=app --cov-report=term-missing` (TOTAL >90%), the isolated E2E file, `uv run ruff check . && uv run pyright`.
## Testing & Quality
- This task IS the phase's test suite (see Work).
- Coverage: **>90%** on `app/` — the only `app/` code in this phase is the `ChatMessage` schema (100% by the unit cases).
## Completion Criteria
- [ ] All four test artifacts exist and pass; the isolated E2E file passes standalone.
- [ ] `uv run pytest --cov=app` TOTAL >90%; lint + types clean.
- [ ] No test asserts the old (broken) behavior — grep for any assertion that a network error shows NO Retry (must not exist).
@@ -0,0 +1,48 @@
# Phase 121 — Private git sources: a token that never reaches the UI or the API
**Source:** `TODO.md` L5 — "Need a way to add private repos without exposing the token in the UI (like when adding an https repo `https://myuser:ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@github.com/myuser/my-private-repo.git`)"
**Story:** n/a (feature request; extends the phase-28/35/38 git/local sources and phase-89 per-source settings assets).
**Context:** `app/models.py:245` — `GitSource`: `url: Text UNIQUE NOT NULL`, `kind` ("git"/"local"), `path`, `ignore_paths` (JSONB), `include_hidden`, `added_at` — no token column. `app/schemas.py` — `GitSourceIn` (L561: `kind`, `url` min 1/max 500, `path`, `ignore_paths`, `include_hidden`; `_trim_url` before-validator L594), `GitSourceOut` (L605: `url: str`), `GitSourceRow` (L626: `url`), `GitSourcePatchIn` (L651). `app/api/git_sources.py` — `URL_RE = ^(?:https?://|ssh://|git@)` (L195; a prefix match, so `user:token@` URLs pass); POST validates the prefix (L344–345) and duplicates via `GitSource.url == url` (L349); GET list / GET single return `row.url` RAW (L235, L250, L297, L423) — an embedded token is echoed to any browser (the leak); the local-source upload endpoint (L432). `app/api/sync.py:296` — `clone_or_pull(row.url, sources_root / repo_name(row.url))`; `scripts/git_sync.py` — `clone_or_pull`, `repo_name`; `scripts/import_docs.py::_resolve_sources` — the CLI's second clone caller (both consume `app.rag.git_sources.effective_sources`, L27). `frontend/assets/git-sources.js` — add form `#git-source-url` (L241; submit `body: (url) => ({ url })` L945); every display site renders `s.url` raw (list cell L395–406 incl. the `title` attr, delete row L533, edit modal L697, ignore-list context L763/L830). `alembic/` — migrations.
## Objective
Private repos are added with a bare URL plus an optional MASKED token field. The token lives in a dedicated DB column, is injected only into the clone URL at sync/clone time, and is absent from every API response and UI surface — including legacy rows that already embed the token in `url` (those are sanitized on output but keep working).
## Dependencies
- `120_failed_turn_retry` (todo) — pipeline predecessor (execution order) only; no code dependency.
- Code dependencies (all complete): phase 35/38 `GitSource` kinds + `effective_sources`, phase 89 per-source settings (the PATCH-field precedent), phase 28 `clone_or_pull`/`repo_name`.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Storage (task 01, LOCKED A2):** `GitSource.token` — `Text NULL` (NULL = public/no credential; **plaintext by necessity** — the repo must remain cloneable, so the raw credential must be recoverable; the Postgres DB is the trusted store and is never served to the UI; there is deliberately no external secrets backend). `GitSourceIn.token: str | None = None` (max 500, trimmed); `GitSourcePatchIn.token: str | None = None` — PATCH semantics: **absent/None = no change, non-empty = replace, empty string = clear** (the UI offers replace; clear exists for API completeness). `GitSourceOut`/`GitSourceRow` gain NO token field — no response shape ever carries it (LOCKED A2).
- **Normalization (task 02):** on POST (and PATCH when a new url/token arrives), the server normalizes: if the incoming URL contains userinfo (`user:pass@host`, only for `https?://` URLs — ssh/`git@` carry no userinfo), the **userinfo is stripped** for storage and the embedded credential is moved into `token` — UNLESS the caller also sent an explicit `token` field, which WINS (explicit beats embedded). Pasting the old-style `https://user:ghp_…@github.com/x/y.git` URL still works and ends up token-column-clean. The duplicate check (L349) runs on the NORMALIZED bare URL, so the same repo with a different token is still the same source (409, not a second row).
- **Clone-time credential (task 02):** `clone_url_for(row) -> str` in `app/rag/git_sources.py` (next to `effective_sources`): `row.url` unchanged when `token` is NULL; otherwise inject `https://x-access-token:<token>@<host>/<path>` (https rows only — a token on a non-https row is a no-op with a warning log). `repo_name` keeps operating on the bare `row.url`. Callers switch from `row.url` to `clone_url_for(row)`: `app/api/sync.py:296` and `scripts/import_docs.py::_resolve_sources` (both already import from `app.rag.git_sources`).
- **Output sanitization (task 02, LOCKED A2):** every API surface that returns a git URL runs it through `sanitize_url(url)` (new, in `app/rag/git_sources.py`): strips the userinfo component (`https://…@host/…` → `https://host/…`), leaves ssh/`git@`/local paths untouched. Applied to `GitSourceOut.url` / `GitSourceRow.url` construction (GET list L235/L297, GET single, the `BOR_GIT_SOURCES` env fallback rows L250 — env rows can embed tokens too) and to any sync-status field echoing a repo URL (grep for `url=` in the sync responses). Belt-and-braces for legacy embedded-token rows whose credential is NOT in the `token` column: their DB value is untouched (the clone still authenticates from the stored URL) but no API/UI output ever shows the credential.
- **UI (task 03):** the add form gains a second field — a masked `<input type="password" id="git-source-token">`, optional, labelled "Token (private repos)" with a visible "optional" hint; submit sends `{ url, token }` (token omitted when blank). The edit modal mirrors it with placeholder "leave blank to keep the current token" (blank → omit from PATCH = no change). Every display site keeps rendering `s.url` — now bare by server sanitization, so list cells, `title` attributes, the delete row, and the ignore-list context become token-free with no per-site change. No new CSS beyond reusing the existing form-field styles (the theme's input treatment).
- **NOT touched:** local-kind sources (no URL credential), the `BOR_GIT_SOURCES` env parsing (its rows are sanitized on OUTPUT only), the upload endpoint, sync scheduling, and the Sources page layout.
## Tasks
1. `01_token_storage.md` — migration + `GitSource.token` + input schemas (`GitSourceIn`/`GitSourcePatchIn`); no token in any output shape.
2. `02_clone_url_and_sanitization.md` — URL/token normalization on write, `clone_url_for` at clone time, `sanitize_url` on every output.
3. `03_ui_token_field.md` — masked token field in the add form + edit modal; display stays `s.url` (now bare).
4. `04_token_tests.md` — unit + integration + isolated E2E `test_git_source_tokens.py`.
## Testing & Quality
- Unit: `tests/unit/test_git_source_token.py` (new, task 04) — `sanitize_url` (https userinfo stripped, ssh/git@/local untouched, no-userinfo unchanged), `clone_url_for` (NULL token → bare URL; token → injected; non-https token → bare + no crash), normalization (embedded token moved to the column when no explicit token; explicit token wins; duplicate on bare URL).
- Integration: `tests/integration/test_git_sources_api.py` (extend) — POST with `token` → GET list/single responses contain the token NOWHERE (assert on the raw JSON text) and show the bare URL; POST with an old-style embedded-token URL → stored bare + token column populated, responses clean; PATCH token replace/clear semantics; the sync flow builds the clone URL with the injected token (mock `clone_or_pull`).
- E2E: `tests/e2e/test_git_source_tokens.py` (new, task 04) — isolated run per AGENTS.md §4: add a private repo through the Sources UI (bare URL + token) → the list row shows the bare URL, the token is absent from the page text, the `title` attribute, and `GET /api/git-sources` JSON; edit the row (blank token) → no 4xx, token kept.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] A private repo added via the UI (or a pasted embedded-token URL) syncs/clones fine, and its token appears in NO API response, NO page text, and NO attribute.
- [ ] Legacy embedded-token rows (pre-phase) still clone, and their API/UI output is token-free.
- [ ] Public repos and local sources behave byte-identically to before.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **A2 — the token is stored PLAINTEXT in a dedicated `GitSource.token` column (cloneability requires the raw credential; no external secrets backend), is NEVER returned by any API shape, and legacy embedded-token URLs are sanitized on output while keeping their stored value for clones (owner-confirmed 2026-09-24, roadmap confirmation).**
- **A6 — pasting an old-style embedded-token URL is accepted and normalized (userinfo → `token` column); an explicit `token` field wins over an embedded one (owner-confirmed: same confirmation — the proposed design).**
## Commit
```bash
git add app/ alembic/ frontend/ scripts/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): add private git repos with a masked token that never reaches the UI or API"
```
@@ -0,0 +1,38 @@
# Task 01 — Token storage: model, migration, input schemas
**Phase:** `121_git_source_tokens` · **Source:** `TODO.md:5` — "Need a way to add private repos without exposing the token in the UI (like when adding an https repo `https://myuser:ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@github.com/myuser/my-private-repo.git`)"
## Objective
The `git_sources` table can hold a per-row token, and the add/patch input shapes can carry one — while NO output shape (`GitSourceOut`, `GitSourceRow`, list, single, env-fallback rows) ever can.
## Work
1. `app/models.py` — `GitSource` (L245): add
```python
#: Private-repo credential (phase 121, LOCKED A2): the PAT the owner
#: types into the masked Sources-page field. NULL = public repo (or a
#: legacy row whose credential is still embedded in ``url``). Stored
#: plaintext BY NECESSITY — the repo must remain cloneable, so the
#: raw credential must be recoverable at sync time; the DB is the
#: trusted store and is never served to the UI. Injected into the
#: clone URL ONLY at clone time
#: (:func:`app.rag.git_sources.clone_url_for`); NEVER returned by
#: any API shape (the output models gain no token field).
token: Mapped[str | None] = mapped_column(Text, default=None)
```
(docstring first, then the column — the phase-89/105 field-docstring house style).
2. `alembic/versions/` — new revision (head of the current chain): `op.add_column("git_sources", sa.Column("token", sa.Text(), nullable=True))` + downgrade `op.drop_column`. Follow the existing migration file conventions (check the latest revision for the revision/down_revision pattern).
3. `app/schemas.py`:
- `GitSourceIn` (L561): add `token: str | None = Field(default=None, max_length=500)` with a `before`-mode trim validator (the `_trim_url` L594 precedent) — plus the docstring note: masked input from the Sources page; absent/None = no credential.
- `GitSourcePatchIn` (L651): add `token: str | None = Field(default=None, max_length=500)` — docstring the PATCH tri-state: **absent/None = no change, non-empty = replace, empty string = clear**.
- `GitSourceOut` (L605) / `GitSourceRow` (L626): add NO field; extend their docstrings with the explicit "no token — never a response field (phase 121)" note so the omission is a documented contract, not an accident.
4. No endpoint changes in this task (acceptance of `token` and normalization are task 02) — the extra field on the input models is inert until then (Pydantic would currently just pass it through unused; task 02 consumes it).
## Testing & Quality
- Unit: `tests/unit/test_git_source_token.py` (task 04 extends): `GitSourceIn`/`GitSourcePatchIn` accept/trim `token`; `GitSourceOut`/`GitSourceRow` reject a `token` key (they are output models built from rows — assert constructing them with a token kwarg raises).
- Integration: `tests/integration/test_git_sources_api.py` (task 04) — after `alembic upgrade head`, `git_sources.token` exists (a `SELECT` sanity check in the existing test fixtures).
- Coverage: **>90%** on `app/` for the touched modules.
## Completion Criteria
- [ ] `uv run alembic upgrade head` applies the new revision on a clean DB and the downgrade removes the column.
- [ ] The ORM round-trips a `token` value; output models have no token field (grep + unit assertion).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,29 @@
# Task 02 — Clone-time credential + output sanitization
**Phase:** `121_git_source_tokens` · **Source:** `TODO.md:5` — "Need a way to add private repos without exposing the token in the UI …"
## Objective
The token is injected into the clone URL only at clone time, old-style embedded-token URLs are normalized into the column on write, and every URL that leaves the API is token-free — including legacy rows and env-fallback rows.
## Work
1. `app/rag/git_sources.py` — add two pure helpers (unit-testable, no DB):
- `sanitize_url(url: str) -> str` — strip the userinfo component of `https?://` URLs (`https://user:pass@host/path` → `https://host/path`); leave `ssh://`, `git@`, and local paths untouched; idempotent. Use a small regex (`^(https?://)([^/@]+)@` → `\1`) — never a URL parser that re-serializes (byte-identical output for clean URLs is a requirement: the phase-50/35 contract is that stored URLs surface verbatim when they carry no credential).
- `clone_url_for(row) -> str` — `row.url` when `row.token` is falsy; for an `https://` row with a token, inject `https://x-access-token:<token>@<host>/<path>` (replace any existing userinfo with the column credential); for a non-https row with a token, log a warning and return `row.url` unchanged (a token cannot authenticate ssh — the owner must use a deploy key/agent there).
- Also `normalize_credential(url, token) -> (bare_url, effective_token)` — the write-path normalizer: if `url` (https only) contains userinfo, strip it → bare URL, and the embedded credential becomes `effective_token` UNLESS `token` is non-None (explicit wins, LOCKED A6). Returns the input untouched for clean URLs.
2. `app/api/git_sources.py`:
- POST git source (L342–355): run `normalize_credential(payload.url, payload.token)`; store the BARE url + `effective_token`; the duplicate check (L349) runs on the bare URL.
- PATCH (the url/token branch): when `payload.url` or `payload.token` is present, re-normalize the (current or new) pair with the same rules — PATCH token tri-state from task 01 (None = no change, `""` = clear → store NULL, non-empty = replace).
- Every output construction runs `sanitize_url` on the URL before it enters the response: the DB-row list path (L235/L297), the GET single path (L423), and the `BOR_GIT_SOURCES` env-fallback rows (L250 — an env URL can embed a token; the ENV VALUE itself is untouched, only the response is masked). Grep the router for any other `url=` response field (including sync-status echoes — `app/api/sync.py` responses that surface a repo URL get the same treatment) and sanitize those too.
3. `app/api/sync.py` (L296) + `scripts/import_docs.py::_resolve_sources` — swap `row.url` → `clone_url_for(row)` at the clone call site (`clone_or_pull(clone_url_for(row), sources_root / repo_name(row.url))` — `repo_name` stays on the bare URL so the checkout directory name is credential-free).
4. ASSUMPTION: `x-access-token` as the injected userinfo username (GitHub-agnostic — any git host that accepts `https://user:token@` treats the first component opaquely; `oauth2:` is also common, but `x-access-token` works on GitHub and GitLab and reads as non-identifying).
## Testing & Quality
- Unit: `tests/unit/test_git_source_token.py` (task 04 finalizes) — `sanitize_url` (strip/no-op/idempotent/ssh/git@/local), `clone_url_for` (NULL token; token injected; non-https token no-op), `normalize_credential` (embedded→column, explicit wins, clean URL untouched).
- Integration: `tests/integration/test_git_sources_api.py` (task 04) — POST embedded-token URL → row.url bare + row.token populated; GET list JSON (raw text) contains the token NOWHERE; sync with a token row (mock `clone_or_pull`) receives the injected URL and a credential-free checkout path.
- Coverage: **>90%** on the touched modules.
## Completion Criteria
- [ ] No token string in ANY API response for a token-bearing row (integration assertion on raw JSON text).
- [ ] A legacy row (token embedded in the stored `url`, `token` NULL) still produces the ORIGINAL stored URL at clone time (the credential keeps working) but its API output is masked.
- [ ] `repo_name` / checkout paths are credential-free.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,30 @@
# Task 03 — UI: masked token field on add + edit
**Phase:** `121_git_source_tokens` · **Source:** `TODO.md:5` — "Need a way to add private repos without exposing the token in the UI …"
## Objective
The Sources-page git-source form takes a separate masked token field (add and edit); every display surface shows the bare URL (server-sanitized) and the token is nowhere in the DOM.
## Work
1. `frontend/assets/git-sources.js`:
- Add form (the `#git-source-url` field at L241, submit wiring at L942–945): add a second labelled field
```html
<label for="git-source-token">Token <span class="field-hint">optional — private repos</span></label>
<input type="password" id="git-source-token" autocomplete="off" placeholder="ghp_… or another PAT">
```
(reuse the existing form-field markup/CSS classes from the url field — the theme's input treatment, no new CSS needed beyond the existing `.field-hint` or an equivalent inline span). Submit body becomes `(url, token) => ({ url, ...(token ? { token } : {}) })` — blank token = key omitted (None = no credential).
- Edit modal (the url display/edit at L697): the same masked token field, placeholder "leave blank to keep the current token"; PATCH body includes `token` ONLY when non-blank (blank → omitted → no change — the task-01 tri-state).
- Display sites (L395–406 list cell incl. the `title` attribute, L533 delete row, L763/L830 ignore-list context): keep rendering `s.url` UNCHANGED — the server now returns bare URLs, so nothing to do per site. Add a source-comment note (one line) that URLs arrive sanitized server-side (phase 121) and the UI must never re-embed a credential.
2. `frontend/assets/styles.css` — only if the "optional" hint span has no existing class to reuse: a minimal `.field-hint` (muted color, contrast ≥4.5:1 per PLAN §7, small).
3. ASSUMPTION: the password field is `type="password"` with `autocomplete="off"` (a PAT is not a site credential; browsers must not offer to save it).
## Testing & Quality
- Unit: `tests/unit/test_git_source_token.py` (task 04) — house-style source assertions: `#git-source-token` is `type="password"` and `autocomplete="off"`; the submit body omits a blank token; the edit PATCH omits a blank token; no display site concatenates a token.
- E2E: `tests/e2e/test_git_source_tokens.py` (task 04) — the UI scenarios.
- Coverage: n/a (frontend) — the validate.sh `app/` gate must stay green.
## Completion Criteria
- [ ] Adding a private repo through the UI with a token succeeds; the list row shows the bare URL.
- [ ] The token is absent from the rendered page text, the `title` attribute, and the Sources-page DOM (E2E assertion).
- [ ] Editing with a blank token keeps the existing credential (integration: the PATCH tri-state).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,36 @@
# Task 04 — Token tests: unit + integration + isolated E2E
**Phase:** `121_git_source_tokens` · **Source:** `TODO.md:5` — "Need a way to add private repos without exposing the token in the UI …"
## Objective
Pin the whole credential contract: helpers are pure and correct, no token ever crosses the API boundary (raw-JSON assertion), legacy rows stay cloneable and masked, and the UI never renders a credential.
## Work
1. `tests/unit/test_git_source_token.py` (new):
- `sanitize_url` — https userinfo stripped (`https://myuser:ghp_x@github.com/x/y.git` → `https://github.com/x/y.git`), clean https unchanged byte-identically, `ssh://git@host/x.git` untouched, `git@github.com:x/y.git` untouched, a local path untouched, idempotent on already-clean URLs.
- `clone_url_for` — `token` NULL → `row.url` verbatim; https + token → `https://x-access-token:<token>@host/path` (existing userinfo REPLACED); non-https + token → `row.url` (no crash, warning logged).
- `normalize_credential` — embedded userinfo → bare URL + token populated; explicit token wins over embedded; clean URL + None token → unchanged.
- Output models: `GitSourceOut`/`GitSourceRow` reject a `token` kwarg (no response field can ever carry it).
- Frontend source assertions (house style): `#git-source-token` is `type="password"` + `autocomplete="off"`; submit/PATCH omit a blank token.
2. `tests/integration/test_git_sources_api.py` (extend):
- POST `{url: "https://github.com/acme/private.git", token: "ghp_test123"}` → 201; `GET /api/git-sources` raw response TEXT does not contain `ghp_test123`; the row's `url` is the bare URL; `GET` single likewise.
- POST an old-style `https://myuser:ghp_legacy@github.com/acme/legacy.git` (no token field) → stored `url` bare, `token` = `ghp_legacy`; responses token-free.
- POST the same repo a second time (different token) → 409 (duplicate on the bare URL).
- PATCH token tri-state: absent → kept; non-empty → replaced (clone URL uses the new one); `""` → cleared (clone URL bare again).
- Legacy-row simulation (insert a row directly with the embedded URL, `token` NULL): `GET` output masked; the sync path (mock `clone_or_pull`) still receives the ORIGINAL stored URL (clone works).
- Sync flow: a token row → `clone_or_pull` called with the injected URL; the checkout path is credential-free.
3. `tests/e2e/test_git_source_tokens.py` (new — isolated run per AGENTS.md §4: `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov`):
- Open the Sources page (admin), add a git source with a bare URL + a distinctive fake token (`ghp_e2esecret…`);
- assert: the list row renders the BARE URL; the token string is absent from `document.body.innerText`, from every `title` attribute, and from the `GET /api/git-sources` JSON (via a `page.request.get` inside the test);
- open the edit modal: the token field is blank (never pre-filled — a password must not be echoed back, so it is simply empty by design); save with it blank → 200, row intact;
- remove the source (cleanup) — the list is empty again.
4. Run the full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (TOTAL >90%), the isolated E2E file, `uv run ruff check . && uv run pyright`.
## Testing & Quality
- This task IS the phase's test suite (see Work).
- Coverage: **>90%** on `app/` — the phase's `app/` code (helpers + router + schema + model) is fully exercised by the unit/integration cases.
## Completion Criteria
- [ ] All test artifacts exist and pass; the isolated E2E file passes standalone.
- [ ] The raw-JSON "token nowhere" assertion covers list AND single AND sync-status surfaces.
- [ ] `uv run pytest --cov=app` TOTAL >90%; lint + types clean.
@@ -0,0 +1,54 @@
# Phase 122 — Image documents: standalone images become first-class, retrievable documents
**Source:** `TODO.md` L6 — "Need to support images. Images uploaded as part of documents or as standalone images should be read, summarized, and retrieved like any other document. Note that the embedding model won't support images, so the only embedded part of an image will be the summary generated by the model. The user should be able to turn on and off image support in their .env depending on whether their model supports it. Images retrieved by the RAG should be shown in the chat nicely and users should be able to submit images as part of their question in brain of reese."
**Story:** n/a (feature request; extends the phase-28/30/38 import pipeline, phase-90 no-scan uploads, and the RAG/chat assets).
**Context:** `app/config.py` — `Settings` (`BOR_` prefix; `upload_dir` L361, `sources_dir` L355, `upload_max_mb` L370 — raw-string/`expanduser` house convention). `app/rag/importer.py` — `iter_importable_files` (L235, extension filter via `llm.settings.import_extension_set` + `match_extension` L217), `import_sources` (L279, `prune` L283), `_index_file` (L435 — `read_text` L453, sha256 over text, Document upsert, the unchanged/hash path with the phase-118 summary backfill), `_store_summary` (L582 — lite-model summary + the position −1 `is_summary` chunk), `_prune` (L649 — deletes docs of the imported sources not in `seen`). `app/models.py` — `Document` (L102: `content`, `content_hash`, `summary` L140, `created_at`/`created_at_manual`), `Chunk` (L148: `is_summary` L161). `app/rag/summarizer.py` + `app/rag/llm.py` — the lite summary path + the chat-model client (`Settings` model names, `check_models`). `app/rag/archive_upload.py` — archive unpack into `upload_dir` (image members land on disk today, then get filtered out by the extension walk). `app/api/git_sources.py:432` — the upload endpoint. `app/api/docs.py` — the document content endpoint (document viewer). `app/rag/retriever.py` / `app/rag/agent.py` (read tool, message build L1446–1448) / `app/api/chat.py` — RAG + the SSE sources frames. `app/api/config.py:30` — `GET /api/config` public flags dict (task 01 of phase 123 extends it). Frontend: `frontend/assets/app.js` (chat source chips), `frontend/assets/document.js` + `frontend/document.html` (`#doc-content` L144), `frontend/assets/sources.js` (Sources page). `alembic/` — migrations.
## Objective
With `BOR_IMAGES=true`, a standalone image file — arriving as a direct upload, inside an uploaded archive, or as a file in a git/local source — becomes a first-class document: the vision model (the chat model) describes it, the description is the document's content AND summary, only the description is embedded (the embedding model never sees pixels), the image bytes persist and are served, and the image shows up in the Sources page, the document viewer, and the chat — with retrieved image docs rendered inline in the answer's sources.
## Dependencies
- `121_git_source_tokens` (todo) — pipeline predecessor (execution order) only; no code dependency.
- Code dependencies (all complete): phase 30 summary pipeline (`_store_summary`, `is_summary` chunk), phase 90 no-scan upload, phase 89/105 per-source walk options, the RAG agent + SSE sources frames.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Toggle (task 01, LOCKED A3):** three new `Settings` fields — `images: bool = False` (`BOR_IMAGES`, `0`/`false` off — the phase-67 `llm_retries` bool style), `image_extensions: str = "png,jpg,jpeg,webp,gif,bmp"` (`BOR_IMAGE_EXTENSIONS`, comma-separated, lowercased into a frozenset by the importer — the `import_extension_set` property precedent), and `image_dir: str = "~/bor-sources/images"` (`BOR_IMAGE_DIR`, raw-string/`expanduser` convention — the persistent home for image bytes, deliberately separate from `sources_dir`/`upload_dir`). `.env.example` gets all three with a comment: **off by default — enable only when your chat model supports vision, because image descriptions are generated by the chat model.** `GET /api/config` (task 01) gains `images: bool` so the UI can gate affordances (consumed by phase 123; the Sources page can show an "images off" hint — optional, not required).
- **Why the bytes are copied (task 02):** upload dirs are REPLACED on every upload (`archive_upload.swap_in`), git checkouts are re-cloned, and local dirs are user-edited — a served image must outlive its source file. The importer copies each ingested image to `image_dir/<doc-uuid>.<ext>` (created on demand) and stores that path in `Document.image_path`. The copy happens ONLY when the doc is new or its hash changes; a replaced image deletes the stale copy; pruned docs delete their copy.
- **Storage (task 02):** `Document.is_image: bool` (server default `false` — every pre-phase-122 row is a text doc) + `Document.image_path: str | None` (NULL for text docs). One migration, one downgrade.
- **Ingest (task 02):** the walk: `iter_importable_files`/`import_sources` accept the image frozenset IN ADDITION to `import_extension_set`, ONLY when `settings.images` is true (images are never user-configurable via `BOR_IMPORT_EXTENSIONS` — the toggle is the single knob, LOCKED A3/A4). `_index_file` branches on image extension: read BYTES (not `read_text`), sha256 over the bytes (the digest rule is unchanged — content identity), copy to `image_dir`, set `is_image` + `image_path`, and `content` = the vision description (task 03). The normal chunk pipeline then embeds the content (= the description) — that is exactly the TODO's "the only embedded part of an image will be the summary generated by the model"; the phase-30 summary chunk (`is_summary`, position −1) mirrors `Document.summary`, which equals the description too. Title = the file stem (the non-markdown rule at L505–509). The unchanged/hash path works unmodified (byte digest → "unchanged" skips re-describing; the phase-118 backfill path re-describes a NULL-summary image doc on its next sync — same fail-soft). **Prune guard:** `_prune` (L649) must NOT delete `is_image` docs while `settings.images` is false (an image doc is invisible to an images-off walk, not a deleted file — otherwise turning the toggle off and syncing would silently destroy the image documents). Toggle ON → normal prune semantics (a deleted image file prunes its doc + copy).
- **Description (task 03, LOCKED A3):** `describe_image` in `app/rag/summarizer.py` (one function, the summarizer module owns model-text generation): a SINGLE chat-model call (`Settings.llm_chat_model` — the vision model; the lite summary model is NOT assumed vision-capable, LOCKED A3) with a multimodal user message `[{type: "text", text: <fixed describe prompt>}, {type: "image_url", image_url: {url: <data URL from the bytes + mime>}}]`; the prompt asks for a faithful, retrieval-oriented description (what is shown, any text/labels/diagram content, salient details — the description is the ONLY thing retrievable, so it must carry the image's meaning). Output capped at `settings.summary_max_chars` (the description IS the doc's summary; the phase-30 cap keeps it uniform). Stored: `Document.summary = Document.content = description`. **Fail-soft:** a failed/empty description → the doc is SKIPPED (no row, `ImportSummary` counts it in a new `images_failed` counter + a `logger.warning` with source/path) — an undescribed image is unsearchable noise; the sync continues (the importer's existing fail-soft convention).
- **Serve + display (task 04):** `GET /api/documents/{doc_id}/image` (new route in `app/api/docs.py`) — 404 for missing docs and non-image docs; serves `image_path` bytes with the correct `Content-Type` (ext → mime map: png/jpeg/webp/gif/bmp) — PUBLIC like the document content itself (this app's document content is already anonymous-readable; the image is part of that content). The document content endpoint (the one `frontend/assets/document.js` boots against) gains `is_image: bool` + `image_url` (the new route's path, absent for text docs) so `document.html` renders `<img src>` (max-width 100%, the theme's image treatment) with the summary/description text below it instead of the markdown content; the Sources page row for an image doc shows a small thumbnail (lazy-loaded, `loading="lazy"`, aspect-ratio box) or the existing doc icon when the fetch is not yet possible offline — the thumbnail is a progressive enhancement (a fetch failure falls back to the icon).
- **RAG display (task 05):** the SSE sources frames (and any sources-list shape the chat bubble renders from) carry an OPTIONAL `image_url` on image docs (the retriever/agent know the `Document` row — add the field where `SourceRef`-shaped frames are built in `app/api/chat.py`/`app/rag/retriever.py`); the chat's sources block renders a compact inline `<img>` (capped height, the summary as caption/alt) for image docs — "shown in the chat nicely" (TODO L6). The agent's `read` tool on an image doc returns its description prefixed with a one-line marker (e.g. `Image document — description generated from the image:`) so the model knows what it is reading. `alt` text = the summary everywhere (WCAG).
- **NOT touched (this phase):** chat-side image submission (phase 123), the lite summary path for TEXT docs, archive unpacking (image members already land on disk — only the walk filter changes), and git/local sync scheduling.
- **Locked assumptions:** **A3** — descriptions use the CHAT model (`BOR_LLM_CHAT_MODEL`, must be vision-capable); `BOR_IMAGES` defaults to **false**; generation failure → doc skipped + logged. **A4** — "images uploaded as part of documents" = standalone image files arriving via direct upload / uploaded archives / source walks — NOT embedded-image extraction from PDFs/DOCX.
## Tasks
1. `01_image_toggle.md` — `BOR_IMAGES` / `BOR_IMAGE_EXTENSIONS` / `BOR_IMAGE_DIR` settings + `.env.example` + `GET /api/config` flag; off = byte-identical behavior.
2. `02_image_ingest.md` — `Document.is_image`/`image_path` + migration; walk accepts image extensions when on; `_index_file` binary branch + persistent copy; prune guard when off.
3. `03_image_description.md` — `describe_image` (chat-model vision), content = summary = description, fail-soft skip + counter.
4. `04_serve_and_display.md` — `GET /api/documents/{id}/image`; document viewer + Sources page rendering.
5. `05_rag_display.md` — `image_url` on chat source frames + inline image in the chat sources block + the agent `read` marker.
6. `06_image_tests.md` — unit + integration + isolated E2E `test_image_documents.py`.
## Testing & Quality
- Unit: `tests/unit/test_image_documents.py` (new, task 06) — settings parsing (toggle off by default, extensions frozenset, mime map), the `_index_file` image branch (bytes digest, copy to `image_dir`, `is_image`/`image_path` set, text `content` never read for an image), the prune guard (toggle off → image docs survive; toggle on → deleted image prunes), `describe_image` prompt shape (multimodal content list, chat model, cap) with a mock client, and the fail-soft skip path.
- Integration: `tests/integration/test_docs_api.py` (extend) — the image route (200 + correct Content-Type for a seeded image doc; 404 for text docs and missing ids); the content endpoint exposes `is_image`/`image_url` for image docs and omits them for text docs (byte-identical text-doc responses); `import_sources` end-to-end with `images=True` and a mock vision client (a fixture PNG → doc row with description content + `is_summary` chunk embedding; `images=False` → the file is ignored, pre-existing image doc survives prune).
- E2E: `tests/e2e/test_image_documents.py` (new, task 06) — isolated run per AGENTS.md §4, `BOR_IMAGES=true` for this suite's app instance: upload a small fixture PNG (via the existing upload endpoint's UI or `page.request`) → sync → the Sources page lists it (thumbnail or icon) → open the document viewer → the image renders with its description → ask a question the mock LLM grounds on the image doc → the chat's sources block shows the inline image.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] With `BOR_IMAGES=true`: an uploaded standalone image (direct or in an archive) and an image file in a git/local source become documents whose content/summary is the vision description and whose ONLY embedded text is that description.
- [ ] With `BOR_IMAGES=false` (the default): every request, walk, and response is byte-identical to pre-phase; existing image docs (if any) survive a sync.
- [ ] The image renders in the document viewer and in the chat's sources block (inline, with alt text); a failed description skips the doc and logs — the sync completes.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **A3 — image descriptions are generated by the CHAT model (`BOR_LLM_CHAT_MODEL`, vision-capable); `BOR_IMAGES` defaults to false; a failed/empty description skips the doc and logs (owner-confirmed 2026-09-24, roadmap confirmation).**
- **A4 — "images uploaded as part of documents" = standalone image files via direct upload / uploaded archives / source walks — no embedded-image extraction from PDFs/DOCX (owner-confirmed 2026-09-24).**
- **Prune guard (derived from A3/A4, same confirmation):** images-off syncs never prune `is_image` docs — turning the toggle off must not destroy image documents.
## Commit
```bash
git add app/ alembic/ frontend/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): index standalone images as documents — described, embedded, and displayed via the vision model"
```
@@ -0,0 +1,25 @@
# Task 01 — Image toggle: BOR_IMAGES + extensions + dir, off by default
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "The user should be able to turn on and off image support in their .env depending on whether their model supports it."
## Objective
The single env knob for image support exists and is surfaced — `BOR_IMAGES` (default **false**), `BOR_IMAGE_EXTENSIONS`, `BOR_IMAGE_DIR` — with `GET /api/config` exposing the flag for UI gating. Toggle off = byte-identical behavior to pre-phase.
## Work
1. `app/config.py` — three new `Settings` fields (house docstring style, the `upload_dir`/`llm_retries` precedents):
- `images: bool = False` — `BOR_IMAGES`, `0`/`false` off (LOCKED A3 default). Docstring: master switch for image-document indexing (phase 122) — off by default, enable only when the chat model supports vision (descriptions come from it).
- `image_extensions: str = "png,jpg,jpeg,webp,gif,bmp"` — `BOR_IMAGE_EXTENSIONS`, comma-separated, case-insensitive; a property/parse into a lowercased-dotted frozenset (the `import_extension_set` precedent) — the image set is SEPARATE from `import_extension_set` (images are never user-added via `BOR_IMPORT_EXTENSIONS`).
- `image_dir: str = "~/bor-sources/images"` — `BOR_IMAGE_DIR`, raw string, `Path.expanduser()` applied by the importer (the `sources_dir`/`upload_dir` convention) — the persistent home for image bytes (uploads are replaced, checkouts re-cloned — the copy must outlive the source file).
2. `.env.example` — the three entries with the comment block: off by default + the vision-model dependency note (LOCKED A3).
3. `app/api/config.py:30` — the `app_config` dict gains `"images": settings.images` (the dict is `str | bool`-valued — bools already allowed). Extend the docstring: consumed by the chat composer (phase 123) to show/hide the attach control, optionally by the Sources page.
4. ASSUMPTION: `GET /api/config` is already anonymous-readable (the UI gates on it pre-login in phase 123 — no auth change here).
## Testing & Quality
- Unit: `tests/unit/test_image_documents.py` (task 06 finalizes) — defaults (`images` False, extensions frozenset `{".png", …}` with the dotted form the matchers expect, dir default), env overrides, the frozenset parse is case-insensitive and trims spaces.
- Integration: the existing `GET /api/config` test asserts the new `images` key (default false in the test env).
- Coverage: **>90%** on the touched modules.
## Completion Criteria
- [ ] `Settings()` with no env: `images is False`, the extension set is the six defaults, `image_dir` is the default path.
- [ ] `GET /api/config` returns `images: false` in the default test env (byte-check the other keys unchanged).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,30 @@
# Task 02 — Image ingest: model fields, walk filter, binary index path, prune guard
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "Images uploaded as part of documents or as standalone images should be read, summarized, and retrieved like any other document."
## Objective
When `BOR_IMAGES=true`, standalone image files in ANY ingest path (direct upload, uploaded archive, git/local source walk) become `Document` rows — bytes persisted to `image_dir`, `is_image`/`image_path` set, `content` = the vision description (task 03) — and an images-off sync never prunes existing image docs.
## Work
1. `app/models.py` — `Document` (L102): add, with house docstrings (the `summary` L140 / `created_at_manual` precedent):
- `is_image: Mapped[bool] = mapped_column(Boolean, default=False, server_default=text("false"), nullable=False)` — True iff the doc's content is a vision description of an image (phase 122); the image bytes live at `image_path`.
- `image_path: Mapped[str | None] = mapped_column(Text, default=None)` — absolute path of the persistent copy in `settings.image_dir`; NULL for text docs.
2. `alembic/versions/` — new revision: both columns (`is_image` NOT NULL server_default 'false'; `image_path` nullable) + downgrade.
3. `app/rag/importer.py`:
- `iter_importable_files` (L235) / the walk in `import_sources` (L279): when `llm.settings.images`, accept a file iff its extension matches `import_extension_set` OR the image frozenset (task 01) — pass the image set in (the function takes explicit extension sets; the image set is NOT merged into `import_extension_set`).
- `_index_file` (L435): image branch FIRST (before the `read_text` at L453) — if the path's extension is in the image set: `data = full_path.read_bytes()`, `digest = sha256(data)`, and on new/changed: copy `data` to `image_dir/<doc-id or uuid4>.<ext>` (dir created with `mkdir(parents=True, exist_ok=True)`), set `is_image=True` + `image_path` on the `Document` row, `content` = the description (task 03's `describe_image` — this task wires the call; the function lands in task 03, so for THIS task store `content = ""` placeholder ONLY if task 03 is not yet merged — the phases run task-ordered, so in practice task 03's function exists; wire it directly and let task 03 implement it. If implementing strictly per task: this task stores `content` via a `_describe_or_skip` hook that task 03 fills — keep the seam single and commented).
- A CHANGED image (hash differs) deletes the stale `image_path` copy before replacing it.
- The unchanged/hash path (L470+) works unmodified for images (byte digest); the phase-118 summary-backfill branch (L480) re-describes an image doc whose `summary` is NULL on the next sync (same fail-soft).
- `_prune` (L649): the prune guard (LOCKED derived decision) — when `settings.images` is FALSE, skip every `is_image` doc (invisible to the walk ≠ deleted); toggle TRUE → normal prune + delete the `image_path` copy of each pruned image doc (also on the normal prune path when the file is gone).
- `ImportSummary` (L100): new `images_failed: int = 0` counter + its slot in `format_counts`/`log` (L135–151) — task 03 increments it; add it now so the log shape is stable.
4. ASSUMPTION (A4 re-stated): only standalone image FILES are ingested — no archive-of-documents extraction, no PDF/DOCX embedded-image pulls (the archive unpacker already places image members on disk; the walk now just accepts them).
## Testing & Quality
- Unit: `tests/unit/test_image_documents.py` (task 06) — the walk accepts `.png` only when `images=True` (off → ignored, the byte-identical default), the binary branch (digest over bytes, copy made, fields set, `read_text` never called for an image), the changed-image stale-copy delete, the prune guard (off → image doc survives; on + file gone → pruned + copy deleted), `images_failed` in the log line.
- Integration: `tests/integration/test_docs_api.py` (task 06) — the `import_sources` end-to-end cases.
- Coverage: **>90%** on the touched modules.
## Completion Criteria
- [ ] `alembic upgrade head` applies; a seeded image walk with `images=True` creates the doc row + `image_dir` copy; `images=False` ignores the file entirely.
- [ ] A sync with `images=False` leaves a pre-existing image doc untouched (prune guard).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,28 @@
# Task 03 — Image description: the vision model writes the only embedded text
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "the only embedded part of an image will be the summary generated by the model."
## Objective
`describe_image` generates the image's description with the CHAT model (vision), the description becomes BOTH `Document.content` and `Document.summary` (so the chunk pipeline embeds exactly that text — and only that text), and a failed description fails soft (skip + count + log, sync continues).
## Work
1. `app/rag/summarizer.py` — new `async def describe_image(llm, data: bytes, mime: str, settings=None) -> str | None` (the summarizer module owns model-text generation; follow the existing summary-call conventions — client, model, timeout, the `summary_max_chars` cap):
- ONE chat-model call (`settings.llm_chat_model` — LOCKED A3; the lite summary model is not assumed vision-capable) with messages `[{role: "user", content: [{type: "text", text: <DESCRIBE_PROMPT>}, {type: "image_url", image_url: {url: f"data:{mime};base64,{b64}"}}]}]` — the multimodal content-list shape the OpenAI-compatible API expects.
- `DESCRIBE_PROMPT` (a module constant, pinned by a unit test): a faithful, retrieval-oriented description — what is depicted, any visible text/labels/titles, diagram/table structure, salient details; 2–4 sentences of substance (the description is the ONLY retrievable text of the doc, so it must carry the image's meaning).
- Return the stripped text capped at `settings.summary_max_chars` (the phase-30 cap — the description IS the summary); return `None` on any client error, empty response, or non-2xx (the caller fails soft). No retries beyond the SDK's own — a description failure must not stall a sync.
2. `app/rag/importer.py` — wire the task-02 seam: the image branch's `content`/`summary` come from `describe_image` —
- description `None` → **skip the doc entirely** (no row, no `image_dir` copy kept — delete the copy if it was made, or make the copy AFTER a successful description so a failure never leaves an orphan), `summary.images_failed += 1`, `logger.warning("import: image description failed source=%s path=%s", source, rel)` — the fail-soft skip (LOCKED A3).
- success → `content = description`, then the existing `_store_summary` path (L582) runs with the description as the summary (the `is_summary` position −1 chunk mirrors it — phase-30 behavior, unchanged), and the normal content chunks embed the description (for a short description that is typically ONE content chunk + the summary chunk — the chunker's existing behavior, no special case).
- the phase-118 backfill branch (unchanged image doc, `summary is None`) calls the SAME path — a description failure there keeps the doc as-is and logs (no row mutation).
3. `app/rag/llm.py` — no new client: `describe_image` reuses the existing `llm.chat`-equivalent client the summarizer already uses for text summaries (verify the exact client method name in `app/rag/summarizer.py` and match it — the multimodal payload is a plain `list[dict]` message, so no client change is needed; IF the existing client hard-codes text-only `content: str` typing, extend its signature to accept `content: str | list` — pyright-clean).
4. ASSUMPTION (A3 re-stated): the CHAT model describes; if the owner's chat model lacks vision, `describe_image` returns `None` (the SDK errors) and every image doc is skipped + logged — honest, visible failure (the `images_failed` counter in the sync log is the signal).
## Testing & Quality
- Unit: `tests/unit/test_image_documents.py` (task 06) — with a MOCK client: the prompt shape (text part + `image_url` data-URL part, correct model), the cap is applied, whitespace stripped; `None` on mock error / empty string / client exception; the importer's skip path (no row, `images_failed == 1`, warning logged, no orphan copy) and the success path (content == summary == description, `is_summary` chunk present, embedding called with the description text — the ONLY text embedded).
- Integration: the mock-vision `import_sources` end-to-end (task 06).
- Coverage: **>90%** on the touched modules.
## Completion Criteria
- [ ] A fixture PNG through the mock vision client yields a doc whose `content` == `summary` == the description, with its embedding(s) derived from that text only.
- [ ] A failing mock client skips the doc, bumps `images_failed`, logs, and the sync completes with the other docs indexed.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,31 @@
# Task 04 — Serve the image + render it in the document viewer and Sources page
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "Images … should be read, summarized, and retrieved like any other document."
## Objective
Image bytes are served through a dedicated document route, the document viewer renders the image with its description, and the Sources page shows an image affordance — image docs read like first-class documents in every existing surface.
## Work
1. `app/api/docs.py` — new route `GET /api/documents/{doc_id}/image`:
- 404 (the router's existing "unknown document" shape) for a missing doc and for a doc with `is_image` false / `image_path` NULL;
- 404 if the file is missing on disk (defensive — the row exists but the copy was lost);
- otherwise `FileResponse` (or a `Response` with the bytes) with `Content-Type` from an ext→mime map (`png`→`image/png`, `jpg`/`jpeg`→`image/jpeg`, `webp`→`image/webp`, `gif`→`image/gif`, `bmp`→`image/bmp` — the map lives in `app/rag/importer.py` or a small shared spot the unit tests can import; default `application/octet-stream` for an unexpected ext) and `Cache-Control: private, max-age=3600` (the image bytes are content-hashed — long enough, bustable by re-upload).
- PUBLIC, like the document content endpoint (this app serves document content to anonymous visitors — the image is part of that content).
- The document CONTENT endpoint the viewer boots against (same module): response gains `is_image: bool` (always present) + `image_url` (the `/api/documents/{id}/image` path — ABSENT for text docs, the `_drop_absent_share_url` omission precedent; never `null`). Text-doc responses gain only `is_image: false` — one new key, documented in the response schema's docstring.
2. `frontend/assets/document.js` + `frontend/document.html`:
- boot reads `is_image`; when true, `#doc-content` renders `<img src="{image_url}" alt="{summary}">` (block, `max-width: 100%`, the theme's surface treatment) with the description/summary text in the normal content slot below it (the document's readable content IS the description — no markdown render of a non-markdown string is needed; render it as the existing plain-content path).
- an `<img>` error fallback: on `onerror` the image area shows a small "image unavailable" note (the 404-on-missing-file case) — the page still shows the description.
3. `frontend/assets/sources.js` — the Sources page row for an image doc: a small thumbnail (48px box, `object-fit: cover`, `loading="lazy"`, `alt = summary`) where the doc icon sits; the thumbnail is a PROGRESSIVE enhancement — a failed fetch (or the row rendered before the fetch resolves) falls back to the existing icon (no layout shift beyond the fixed box). The doc title/path columns are unchanged.
4. `frontend/assets/styles.css` — the viewer image block + the Sources thumbnail box (theme tokens; WCAG: alt text everywhere, no contrast concerns for decorative images).
5. ASSUMPTION: the thumbnail uses the SAME full-size route (no separate thumb route) — a KB-scale image set makes a thumb pipeline unjustified; lazy loading keeps the Sources page fast.
## Testing & Quality
- Integration: `tests/integration/test_docs_api.py` (task 06) — the image route (200 + exact `Content-Type` per ext for a seeded doc; 404 for a text doc; 404 for a missing id; 404 for a row whose file is deleted); the content endpoint: `is_image` present in ALL responses, `image_url` absent for text docs and present for image docs.
- Unit: `tests/unit/test_image_documents.py` (task 06) — the ext→mime map (all six + the octet-stream default); house-style source assertions: the viewer renders the `img` from `image_url` with `alt = summary`, the sources row falls back to the icon on image error, no `null` in the text-doc content response.
- E2E: `test_image_documents.py` scenarios (task 06) cover viewer + Sources rendering.
- Coverage: **>90%** on the touched modules.
## Completion Criteria
- [ ] `GET /api/documents/{id}/image` serves the exact uploaded bytes with the right Content-Type; text docs 404.
- [ ] The document viewer shows the image + its description; the Sources page shows the thumbnail (or the icon fallback).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,24 @@
# Task 05 — RAG display: image docs in the chat sources + the agent read marker
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "Images retrieved by the RAG should be shown in the chat nicely."
## Objective
When a retrieved/agent-read document is an image, the chat shows it: the sources block renders a compact inline image with its summary as caption/alt, and the agent's `read` tool tells the model it is reading a generated image description.
## Work
1. `app/rag/retriever.py` / `app/api/chat.py` — the sources frames the chat bubble renders (the SSE `sources`/related-doc frames and the agent-sourced doc list): add an OPTIONAL `image_url` field to the per-doc ref shape — populated (the `/api/documents/{id}/image` path) iff the doc row has `is_image`, absent otherwise (the omission rule — text-doc frames stay byte-identical). The retriever already has the `Document` row; the agent's doc refs (the read-tool results / source list) do too — set it at the frame-build sites (grep for the source-ref construction in both modules; one shared helper `source_ref_with_image(doc, …)` keeps the two sites in lockstep).
2. `frontend/assets/app.js` — the chat's sources block renderer: when a source ref carries `image_url`, render a compact inline `<img>` (max-height ~96px, `object-fit: contain`, the theme's surface, `alt` + visible caption = the doc summary — the "shown nicely" requirement) in place of / beside the existing doc chip text (keep the title + the existing chip affordance — the image is additive, not a replacement). A failed image load collapses to the plain chip (never a broken-image icon).
3. `app/rag/agent.py` — the `read` tool's result for an image doc: prefix the description with the marker line `Image document — the text below is a description generated from the image:` (a module constant) so the model reasons about what it is reading; non-image docs' results are byte-identical.
4. ASSUMPTION: the chat QUESTION side (users submitting images) is phase 123 — this task only covers RETRIEVED images in the answer's sources.
5. ASSUMPTION: the sources-frame `image_url` is the only new frame field — no doc-id leak beyond what the frame already carries (the path encodes the doc id, same as the content endpoint).
## Testing & Quality
- Integration: `tests/integration/test_chat_api.py` (extend, task 06) — a mocked grounded answer that includes an image doc in its sources → the SSE frame carries `image_url` for that ref only; a text-only grounding has NO `image_url` key anywhere (byte check).
- Unit: `tests/unit/test_image_documents.py` (task 06) — the frame-helper (present/absent), the agent marker (image vs non-image result), house-style source assertions: the sources renderer reads `image_url`, sets `alt`, and falls back on image error.
- E2E: `test_image_documents.py` scenario (task 06) — ask a question the mock LLM grounds on the fixture image doc → the chat sources block shows the inline image with its caption.
- Coverage: **>90%** on the touched modules.
## Completion Criteria
- [ ] A chat answer grounded on an image doc shows the inline image + caption in its sources block; text-doc answers render byte-identically to before.
- [ ] The agent `read` result for an image doc carries the marker; the model sees the description, not raw bytes.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,33 @@
# Task 06 — Image tests: unit + integration + isolated E2E
**Phase:** `122_image_documents` · **Source:** `TODO.md:6` — "Need to support images …"
## Objective
Pin the whole image-document contract: the off-by-default byte-identity, the ingest/description/serve pipeline, the RAG display, and the end-to-end user path (upload → Sources → viewer → chat) as an isolated Playwright suite.
## Work
1. `tests/unit/test_image_documents.py` (new) — consolidates the per-task unit cases (the tasks ship code; this task ships the full pin):
- settings: defaults (`images` False, six-extension frozenset, dir default), env overrides, case-insensitive parse (task 01);
- the walk: image accepted iff `images=True`; off → the file is ignored (the default byte-identity);
- `_index_file` image branch: digest over BYTES, copy to `image_dir`, `is_image`/`image_path` set, changed-image stale-copy delete, prune guard (off → survives; on + gone → pruned + copy deleted), `images_failed` in the log line (task 02);
- `describe_image`: mock-client prompt shape (multimodal parts, chat model), cap, `None` on error/empty, the importer skip path (no row, no orphan, counter, warning) and the success path (content == summary == description; embedding called with the description only) (task 03);
- the ext→mime map (task 04);
- the source-frame `image_url` helper (present/absent) + the agent `read` marker + house-style frontend assertions (viewer `img` + alt + fallback; sources thumbnail fallback; chat sources inline image + alt) (task 05).
2. `tests/integration/test_docs_api.py` (extend, task 04's cases) — the image route (200 + Content-Type per ext; 404 text doc / missing id / missing file), the content endpoint's `is_image`/`image_url` omission rules; `tests/integration/test_chat_api.py` (extend, task 05's case) — the SSE `image_url` frame; `tests/integration/` (new file `test_image_import.py` or the existing import test file — follow whichever exists) — `import_sources` end-to-end: `images=True` + mock vision → the fixture PNG becomes a doc (description content, `is_summary` chunk, one content chunk); `images=False` → ignored + a pre-seeded image doc survives prune; a failing mock → `images_failed == 1`, no row, other docs indexed.
- Fixtures: a tiny valid PNG (a few bytes, generated in-test or a committed fixture under `tests/` — check the existing fixture conventions), a mock vision client (the existing mock-LLM test patterns in `tests/`).
3. `tests/e2e/test_image_documents.py` (new — isolated run per AGENTS.md §4: `uv run pytest tests/e2e/test_image_documents.py -v --no-cov`). The suite's app instance runs with `BOR_IMAGES=true` (env override in the E2E fixture — the `conftest.py` pattern for per-suite app env):
- upload a fixture PNG (the Sources-page upload flow or `page.request` against the upload endpoint, then trigger the sync through the UI as the Sources page does);
- the Sources page lists the image doc (thumbnail or icon fallback);
- open the document viewer → the image renders + the description text below it;
- ask a question the mock LLM grounds on the image doc (the existing mock-LLM grounding pattern) → the chat's sources block shows the inline image with its caption;
- negative: with the DEFAULT env (`BOR_IMAGES` unset/false), the same upload produces NO image doc (the default-off contract).
4. Run the full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (TOTAL >90%), the isolated E2E file, `uv run ruff check . && uv run pyright`.
## Testing & Quality
- This task IS the phase's test suite (see Work).
- Coverage: **>90%** on `app/` — the phase's `app/` surface (config, importer, summarizer, docs API, chat frames, agent marker) is fully exercised.
## Completion Criteria
- [ ] All test artifacts exist and pass; the isolated E2E file passes standalone.
- [ ] The default-off byte-identity is asserted (unit + integration + the E2E negative case).
- [ ] `uv run pytest --cov=app` TOTAL >90%; lint + types clean.
@@ -0,0 +1,50 @@
# Phase 123 — Chat image questions: attach an image to a question
**Source:** `TODO.md` L6 — "…users should be able to submit images as part of their question in brain of reese."
**Story:** n/a (feature request; completes the phase-122 image capability on the question side).
**Context:** Phase 122 (todo, this pipeline) — `BOR_IMAGES` toggle + `GET /api/config` `images` flag (task 01), the ext→mime map, `image_dir` storage convention. `app/schemas.py:60` — `ChatRequest` (`message` min 1/max 4000, `history` ≤100 — `HistoryTurn` is text-only), `ChatMessage` (L742, `extra="forbid"`, phase-83 value bounds). `app/api/chat.py` — the turn pipeline: the user message is built at L645 (`{"role": "user", "content": request.message}`; a grounded turn runs `run_agent`, a deflected turn a direct `chat_stream` on the same `messages`), and `app/rag/agent.py:1370/1448` — `run_agent(..., user_message: str)` builds its own `[system, user]` (verify the data flow — if `run_agent` receives the already-built `messages`, the single edit site is chat.py). The phase-114 SSE error-frame-with-hint pattern (the "question too long" frame — `ChatErrorEvent.detail` + optional `hint`, consumed by the banner at app.js L2210). `frontend/index.html` — the composer (label L287, `#message-input` L292, `#send-btn` L325). `frontend/assets/app.js` — `handleSend` (L2306), `runTurn` (the turn driver + the user append/save-point-1 at send), `addMessage("user", …)` (user bubble), `rememberBrainTurn` (L2108, the brain save point), `renderStoredMessage` (L1642, user branch). `frontend/assets/shared.js` — the shared page's message render (text-only today). `app/api/config.py:30` — the public flags dict (phase 122 task 01 added `images`).
## Objective
The user attaches one image to a question: a masked-by-server upload stores the bytes, the vision model (the chat model) receives a multimodal message, the user's bubble renders the image, the record persists the image path (not base64) so refresh and shared chats render it, and `BOR_IMAGES=false` rejects the request with a helpful hint.
## Dependencies
- `122_image_documents` (todo) — CODE dependency: the `BOR_IMAGES`/`images` config flag (the toggle gates this feature), the ext→mime map, and the `image_dir` storage convention (this phase's `chat_image_dir` follows it).
- Code dependencies (all complete): phase 14/50/55 conversation persistence, phase 74 history mapping, phase 114 SSE error-hint frames, phase 51 shared chats.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Storage (task 01, LOCKED A5):** user question-images are server-stored, NOT base64-in-saved-chats: `Settings.chat_image_dir: str = "~/bor-sources/chat-images"` (`BOR_CHAT_IMAGE_DIR`, the `image_dir` convention — a sibling of phase 122's `image_dir`, separate because question-images are per-conversation, not per-source) + `Settings.chat_image_max_mb: int = 10` (`BOR_CHAT_IMAGE_MAX_MB`, the ~10 MB cap of A5; `upload_max_mb`'s fail-loud validator precedent for `<= 0`). `POST /api/chat-images` (multipart, in `app/api/chat.py` or a small new `app/api/chat_images.py` router — the executor's call, following the repo's one-concern-per-module style): accepts an image file, validates the mime/ext against the SAME six-extension set as phase 122 (reuse the frozenset; the Content-Type header is a hint — the EXTENSION is the source of truth, the archive-uploader precedent), rejects oversize with a 413 (the fixed-detail style), stores `chat_image_dir/<uuid4().hex>.<ext>`, returns `{ "path": "/api/chat-images/<uuid-hex>.<ext>" }`. `GET /api/chat-images/{filename}` serves the bytes (404 on missing/unknown — the filename is a uuid, no enumeration value) with the phase-122 mime map; PUBLIC like saved-chat content (a saved chat's id is already its credential — phase 55 A1 — the image is part of that content).
- **Request (task 01):** `ChatRequest.image: str | None = Field(default=None, max_length=500)` — a STORED PATH, pattern-validated (`^/api/chat-images/[0-9a-fA-F]{32}\.(png|jpe?g|webp|gif|bmp)$` — the stored filename is `uuid4().hex.<ext>`) — never a raw data URL (the upload endpoint already did the size/mime enforcement; re-validating a 10 MB base64 string in the schema would be the anti-pattern). Toggle OFF (`settings.images` false) with `image` set → the turn settles with the phase-114 SSE error frame: `detail` "Image support is turned off on this server." + `hint` "Enable BOR_IMAGES in the server's .env (and restart) to ask with an image." (the question itself is NOT persisted — a rejected turn saves nothing, the existing error-path convention). `image` set but file missing → the same frame shape with a "that image is no longer available" detail (a stale-path edge: the stored file was deleted out-of-band).
- **Multimodal (task 01):** the user message becomes `{"role": "user", "content": [{"type": "text", "text": request.message}, {"type": "image_url", "image_url": {"url": <data URL from the stored file>}}]}` at BOTH construction sites (chat.py:645 and agent.py:1448 if it builds independently — verify the flow; when `request.image` is None the content stays the plain string, byte-identical to today). The data URL is built server-side from the stored bytes + mime map (the phase-122 `describe_image` data-URL construction — reuse it). `HistoryTurn`/`history_to_messages` are UNCHANGED (LOCKED A7): prior turns' images are never replayed into the model's history — the history budget is text, and a 10 MB image per past turn would blow every budget; the model simply sees the text of a prior turn that had an image.
- **Persistence (tasks 01+02):** `ChatMessage.image: str | None = Field(default=None, max_length=500)` — the stored path, on the USER record (the image belongs to the question). The user record is saved at save-point-1 (send), BEFORE the turn resolves — so the client uploads FIRST (`POST /api/chat-images`) and stores the returned path in the user record, then POSTs `/api/chat` with `image=<path>`. Saved chats, shared chats, and the localStorage shape all carry the path (≤500 chars — no phase-83 cap pressure). A brain record never carries `image` (the answer may reference the image's sources, but the attachment is the user's).
- **Composer (task 02):** the attach control appears ONLY when `GET /api/config` says `images: true` (phase 122's flag; fetched once at boot like the other config — the composer reads the existing cached config if present). A paperclip button (SVG, the icon style of the other composer glyphs, `aria-label="Attach an image"`) before the input → hidden `<input type="file" accept="image/png,image/jpeg,image/webp,image/gif,image/bmp">` → on select: a preview strip above the input (thumbnail ≤48px, the filename, a remove ✕) + the file's data URL kept client-side until send; on send with an attachment: `POST /api/chat-images` (the file) → the returned path goes into the user record + the `/api/chat` body → the preview clears. Upload failure (oversize, non-image, server down) → the phase-114-style out-of-turn banner ("Couldn't attach the image — …") and the send is BLOCKED (no question without the image the user attached — ASSUMPTION A8, locked below). The user bubble renders the image (from the data URL live, from the stored path after restore) with `alt = filename`, capped height, above/beside the text (the theme's bubble treatment; the image is part of the question, visible in both the live bubble and the restore).
- **Restore + shared (task 03):** `renderStoredMessage`'s user branch: `m.image` present → the user bubble includes `<img src="{m.image}" alt="…">` (a load failure collapses to a small "image unavailable" line — never a broken icon). The shared page (`shared.js`) renders the user image the same way (the image route is public — the shared view is faithful; no new shared-shape field beyond `ChatMessage.image`, which the public `messages` shape already carries).
- **NOT touched:** the history budget/trimming, the honesty gate, the suggestion chips, phase-122's document-image pipeline (a QUESTION image is a separate concern — it is NOT indexed as a document), and the stop/failed-turn paths (they persist whatever records exist, including the new `image` key, unmodified).
## Tasks
1. `01_vision_request.md` — `POST/GET /api/chat-images`, `ChatRequest.image` + `ChatMessage.image`, the toggle-off/stale error frames, the multimodal user message at both construction sites.
2. `02_composer_attach.md` — the config-gated attach control, preview, upload-then-send, the user bubble's image.
3. `03_restore_and_shared.md` — `ChatMessage.image` on restore (chat page) and on the shared page.
4. `04_chat_image_tests.md` — unit + integration + isolated E2E `test_chat_image_questions.py`.
## Testing & Quality
- Unit: `tests/unit/test_chat_image_questions.py` (new, task 04) — the path pattern validator (accepts well-formed, rejects data URLs / wrong ext / traversal), the multimodal message build (both sites; `image=None` → byte-identical plain string), the toggle-off + stale-file error frames (detail + hint shapes), the upload endpoint's mime/size/ext rules (tmp-dir settings), the serve route (200/404), `ChatMessage.image` bounds + omission.
- Integration: `tests/integration/test_chat_api.py` (extend, task 04) — upload → `POST /api/chat` with `image=<path>` → the mock client RECEIVES the multimodal content list (text part + image_url data URL); `image` with `images=false` → the SSE error frame with the hint and NO model call, no persisted record; `image=None` requests are byte-identical to pre-phase; a saved chat round-trips a user record with `image`; a shared chat serves it.
- E2E: `tests/e2e/test_chat_image_questions.py` (new, task 04) — isolated run per AGENTS.md §4, `BOR_IMAGES=true`: attach a fixture PNG in the composer → preview + remove works → send → the user bubble shows the image → the (mock) answer streams → reload → the user bubble restores WITH its image → open the shared link → the shared page shows the image. Plus the default-off negative: with `BOR_IMAGES` unset, the attach control is ABSENT from the DOM.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] With `BOR_IMAGES=true`: attach → send → the vision model gets text+image; the user bubble, the refreshed page, and the shared chat all show the image; the saved chat stores the PATH (assert no base64 in the stored payload).
- [ ] With `BOR_IMAGES=false`: the attach control is absent, an API request with `image` gets the hinted error frame, and no model call / record happens.
- [ ] Text-only questions behave byte-identically to pre-phase (the multimodal branch is inert).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **A5 — one image per question; the ~10 MB cap (`BOR_CHAT_IMAGE_MAX_MB`); server-stored bytes under `chat_image_dir`; the saved/shared record carries the path, never base64 (owner-confirmed 2026-09-24, roadmap confirmation).**
- **A7 — a question's image applies to the CURRENT turn only; prior turns' images are never replayed into the model's history (the text of a prior turn stands alone) (owner-confirmed: same confirmation — the proposed design).**
- **A8 — if the image upload fails, the send is blocked with a banner (the question is never sent without the image the user attached) (owner-confirmed: same confirmation).**
## Commit
```bash
git add app/ frontend/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "feat(chat): attach an image to a question — vision input, in-bubble render, persisted and shared"
```
@@ -0,0 +1,31 @@
# Task 01 — Vision request: upload/serve endpoints, request + message schemas, multimodal build
**Phase:** `123_chat_image_questions` · **Source:** `TODO.md:6` — "…users should be able to submit images as part of their question in brain of reese."
## Objective
The server side of the image question: a uuid-named upload/serve pair for question images, `ChatRequest.image` (stored path) + `ChatMessage.image` (persistence), the toggle-off/stale error frames, and the multimodal user message at both construction sites — with text-only requests byte-identical to pre-phase.
## Work
1. `app/config.py` — `chat_image_dir: str = "~/bor-sources/chat-images"` (`BOR_CHAT_IMAGE_DIR`, the phase-122 `image_dir` convention) + `chat_image_max_mb: int = 10` (`BOR_CHAT_IMAGE_MAX_MB`, the A5 cap; the `upload_max_mb` fail-loud `<= 0` validator precedent). `.env.example` entries.
2. New router (a small `app/api/chat_images.py`, registered in `app/main.py` next to the chat router — the one-concern-per-module house style):
- `POST /api/chat-images` — `UploadFile` (the git-sources upload endpoint L432 pattern): extension must be in the phase-122 image frozenset (the EXTENSION is the source of truth — a Content-Type header is a hint); total bytes capped at `chat_image_max_mb` (stream-count the bytes — reject with 413 + a fixed detail that names the cap, never echoing the filename); store `chat_image_dir/<uuid4().hex>.<ext>` (dir created on demand); response `{"path": "/api/chat-images/<uuid>.<ext>"}`.
- `GET /api/chat-images/{filename}` — filename must be `<uuid-hex>.<ext>` (the regex guard → 404 otherwise, no path traversal by construction); 404 on missing file; serve the bytes with the phase-122 ext→mime map + `Cache-Control: private, max-age=3600` (public, like saved-chat content — phase 55 A1).
3. `app/schemas.py`:
- `ChatRequest` (L60): `image: str | None = Field(default=None, max_length=500)` + a `field_validator` — when set, it must match `^/api/chat-images/[0-9a-fA-F]{32}\.(png|jpe?g|webp|gif|bmp)$` (the uuid4().hex shape — 32 hex chars; adjust if the uuid format differs) with a fixed 422 detail ("image must be an uploaded chat image path" — no echo). Docstring: the path from `POST /api/chat-images` (task 01) — never a data URL; the upload endpoint owns size/mime enforcement.
- `ChatMessage` (L742, `extra="forbid"`): `image: str | None = Field(default=None, max_length=500)` — on the USER record only (the question's attachment); the docstring notes brain records never carry it and the saved/shared shape therefore gains one optional key (omitted when None — the phase-50 byte-identical contract for text-only chats holds).
4. `app/api/chat.py` — the turn pipeline:
- pre-stream validation (BEFORE any model call, at the top of the turn handler): `request.image` set → `settings.images` false → yield the phase-114 error frame (`ChatErrorEvent(detail="Image support is turned off on this server.", hint="Enable BOR_IMAGES in the server's .env (and restart) to ask with an image.")`) and return (NO model call, NO record — the existing error-path convention); file missing on disk → the same frame shape, `detail="That image is no longer available."` + a generic reachability-free hint (or no hint — the banner's default is fine).
- the user message (L645): when `request.image` is set, `{"role": "user", "content": [{"type": "text", "text": request.message}, {"type": "image_url", "image_url": {"url": <data URL>}}]}` — the data URL built from the stored bytes + the phase-122 mime map (REUSE the data-URL construction from `describe_image` — factor it to a shared helper if it is buried in `app/rag/summarizer.py`); `request.image` None → the plain-string content, byte-identical.
- `app/rag/agent.py` — verify the data flow: if `run_agent` (L1370) receives the already-built `messages` from chat.py, NO change here (the L1448 build is for a different entry); if it builds its own user message from `user_message`, extend `run_agent`'s signature (`user_message: str | list | None` — pyright-clean) and make chat.py pass the multimodal content. Pin the chosen flow in a code comment.
- a QUESTION image is NEVER indexed as a document (no importer call) — it is turn-local storage.
5. ASSUMPTION (A7 re-stated): `HistoryTurn`/`history_to_messages` unchanged — prior turns' images are not replayed (text-only history stands).
## Testing & Quality
- Unit: `tests/unit/test_chat_image_questions.py` (task 04 finalizes) — the path validator (well-formed ok; a data URL, a wrong ext, a traversal, and a 31-hex-char uuid all 422); the multimodal builder (both sites; None → plain string); the error frames' exact detail/hint strings; the upload endpoint's ext/size rules (tmp `chat_image_dir`); the serve route 200/404 + Content-Type; `ChatMessage.image` (bounds, omission, the `extra="forbid"` boundary intact).
- Integration: `tests/integration/test_chat_api.py` (task 04) — the upload → chat flow asserts the MOCK client received the multimodal content list; the toggle-off frame + no model call; text-only byte-identity; saved + shared round-trips with `image`.
- Coverage: **>90%** on the touched modules.
## Completion Criteria
- [ ] `POST /api/chat-images` stores a uuid-named file and returns its path; `GET` serves it; oversize/non-image → 413/422 with fixed details.
- [ ] `POST /api/chat` with `image` (toggle on) delivers a multimodal user message to the model; toggle off → the hinted error frame, no model call; `image=None` → byte-identical behavior.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,38 @@
# Task 02 — Composer: attach control, preview, upload-then-send, the user bubble's image
**Phase:** `123_chat_image_questions` · **Source:** `TODO.md:6` — "…users should be able to submit images as part of their question in brain of reese."
## Objective
The composer (when `GET /api/config` says `images: true`) takes one attached image — preview + remove before send, upload on send, the image in the user's bubble — and the user's conversation record carries the stored `image` path.
## Work
1. `frontend/index.html` — the composer (the label L287 / `#message-input` L292 / `#send-btn` L325 region): before the input, the attach control
```html
<button type="button" class="attach-btn" id="attach-btn" hidden aria-label="Attach an image">
<svg aria-hidden="true" …paperclip glyph, the icon style of the other composer glyphs…></svg>
</button>
<input type="file" id="attach-file" accept="image/png,image/jpeg,image/webp,image/gif,image/bmp" hidden>
```
plus the preview strip container (after the input row, `#attach-preview`, `hidden` by default — a thumbnail ≤48px + filename + a remove ✕ button `#attach-remove`). `#attach-btn` is `hidden` by default — JS reveals it only when the config flag is on (task 02 step 3); the hidden-by-default markup keeps the flag-off DOM byte-identical (A5's default-off contract).
2. `frontend/assets/app.js`:
- boot: read `images` from the `GET /api/config` fetch (the composer already consumes the cached whoami/config boot — extend that fetch's result use; ONE request, no extra round-trip) → `attachBtn.hidden = !images`.
- attach flow: `#attach-btn` click → `attachFile.click()`; on change: validate the file's extension against the six (client-side pre-check, the server re-validates — a bad pick → the out-of-turn banner "Only PNG, JPEG, WebP, GIF, and BMP images can be attached." and no state change); keep `{ file, dataUrl (for the live preview) }` in a turn-local `attachedImage` var; show `#attach-preview` (thumbnail from the data URL, the filename, the remove ✕); the remove ✕ (or a new selection) clears the state + hides the strip.
- send flow (`handleSend` L2306 / `runTurn`): when `attachedImage` is set:
1. `POST /api/chat-images` (the File) — on failure (413/422/5xx) → the phase-114-style out-of-turn banner with the server's detail ("Couldn't attach the image — try again.") and the send is BLOCKED (LOCKED A8 — the question is never sent without its image; the input text stays).
2. success → `runTurn(text, { image: <returned path> })`; `runTurn`'s user append (save point 1 — the user record) stores `{ who: "user", text, image: <path> }` (the `image` key joins the `bor.chat.v1` record — the phase-14 shape gains the optional key; `saveConversation()` + the phase-55 auto-save ride the existing path);
3. the USER bubble renders the image: extend `addMessage("user", text)` with an optional `image` arg (data URL live, path after restore) → `<img src alt={filename}>` in the bubble (capped height ~240px, `max-width: 100%`, the theme's bubble treatment, `loading="lazy"`);
4. clear `attachedImage` + the preview strip AFTER the user bubble is rendered (the strip must not linger into the turn).
- text-only sends: `attachedImage` null → the request body omits `image`, the user record omits the key, the bubble is byte-identical to pre-phase.
3. `frontend/assets/styles.css` — `.attach-btn` (the composer glyph button treatment — match the send-btn family, focus-visible ring per PLAN §7), `#attach-preview` (the strip: flex row, thumbnail box, filename ellipsis, the ✕), the user-bubble image block.
4. ASSUMPTION (A8 re-stated): an upload failure blocks the send (no partial question-without-image) — the banner tells the user what failed; the typed question is preserved.
## Testing & Quality
- Unit: `tests/unit/test_chat_image_questions.py` (task 04) — house-style source assertions: `#attach-btn` is `hidden` by default + `aria-label`; the reveal is gated on the config `images` flag; the extension pre-check list matches the server's six; the send path uploads BEFORE `runTurn` and blocks on failure (the A8 ordering); the user record gains `image` only when attached; the user bubble renders the `img` with `alt`.
- E2E: `tests/e2e/test_chat_image_questions.py` (task 04) — the composer scenarios.
- Coverage: n/a (frontend) — the validate.sh `app/` gate must stay green.
## Completion Criteria
- [ ] Flag on: attach → preview → remove all work; send with an attachment uploads, the user bubble shows the image, and the question reaches the model.
- [ ] Flag off: the attach button is ABSENT from the DOM; a hand-crafted `image` request still gets the server's error frame (task 01's contract, unchanged).
- [ ] A text-only send produces the same request body and DOM as pre-phase (byte-check in the E2E where practical).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,26 @@
# Task 03 — Restore + shared: the question's image survives a refresh and a share link
**Phase:** `123_chat_image_questions` · **Source:** `TODO.md:6` — "…users should be able to submit images as part of their question in brain of reese."
## Objective
A user record carrying `image` (the stored path) renders its image when the chat is restored from localStorage / the saved-chat API, and on the shared-chat page — a load failure degrades to a small note, never a broken icon.
## Work
1. `frontend/assets/app.js` — `renderStoredMessage(m)` (L1642), the USER branch: when `m.image` is present, the restored user bubble includes `<img src="{m.image}" alt="{m.text || 'attached image'}">` through the SAME bubble-image helper task 02 built for the live bubble (one renderer — the live bubble passes the data URL, restore passes the path; the helper takes any `src`). `onerror` → replace the image with a small "image unavailable" line (the file was deleted out-of-band — the row keeps its path, the render degrades).
- The restore paths that call `renderStoredMessage` (the localStorage restore ~L1700 and the saved-chat restore ~L1770) need NO other change — the record's `image` key flows through the phase-14/50 restore as any optional key.
- `retryLastTurn` / regenerate: a RE-ASK of a question that had an image does NOT re-attach the image (the redo re-sends `prev.text` only — LOCKED A7, the image is turn-local to the original send; the restored image stays visible in the replaced record until the redo pops it, which is the existing redo-in-place behavior).
2. `frontend/assets/shared.js` — the shared page's message render (its text-only loop over `messages`): the user-record branch gains the same image render (the image route is public — a shared chat is faithful; the `alt` + `onerror` degradation are identical to the chat page).
3. `frontend/assets/styles.css` — no new rules beyond what task 02 added (the shared page reuses the bubble-image block; verify the shared page's bubble class shares it — if the shared page uses a different bubble class, scope the image rule to both).
4. ASSUMPTION: the `image` key is optional and absent in every pre-phase saved chat — no data migration, no backfill (old chats have no question-images to restore).
## Testing & Quality
- Unit: `tests/unit/test_chat_image_questions.py` (task 04) — house-style source assertions: the user-branch render reads `m.image` and reuses the bubble-image helper; the `onerror` degradation exists on BOTH pages; the shared render includes the image; the redo path sends `prev.text` only (no `image` on the re-ask).
- Integration: `tests/integration/test_chats_api.py` (extend, task 04) — a user record with `image` round-trips `POST`/`PUT /api/chats` and serves through `GET /api/shared/{token}` (the public shape carries it).
- E2E: the refresh + shared scenarios of `tests/e2e/test_chat_image_questions.py` (task 04).
- Coverage: n/a (frontend) — the validate.sh `app/` gate must stay green.
## Completion Criteria
- [ ] Reload after an image question: the user bubble shows the image (from the stored path) + the rest of the conversation is unchanged.
- [ ] The shared link renders the image on the shared page.
- [ ] A deleted image file degrades to the "image unavailable" line on both pages (no broken-image icon).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,39 @@
# Task 04 — Chat-image tests: unit + integration + isolated E2E
**Phase:** `123_chat_image_questions` · **Source:** `TODO.md:6` — "…users should be able to submit images as part of their question in brain of reese."
## Objective
Pin the whole question-image contract: the upload/serve rules, the multimodal model payload, the toggle-off rejection, the persistence shape (path, never base64), the composer gating, and the refresh/share rendering — as an isolated Playwright suite.
## Work
1. `tests/unit/test_chat_image_questions.py` (new) — consolidates the per-task unit cases (tasks ship code; this task ships the full pin):
- config: `chat_image_dir`/`chat_image_max_mb` defaults + env overrides;
- the `ChatRequest.image` validator (well-formed path ok; data URL / wrong ext / traversal / malformed uuid → 422, fixed details, no echo);
- the multimodal builder (text part + `image_url` data-URL part, correct mime; `image=None` → the plain string, byte-identical); the data-URL helper is shared with `describe_image` (assert the import, not a copy);
- the error frames: toggle-off (exact detail + hint strings), stale file (exact detail) — both settle the turn WITHOUT a model call (the mock client must see zero calls);
- the upload endpoint: the six exts accepted, others 422/413-style per the spec, oversize → 413 (fixed detail naming the cap), the stored filename is `<uuid-hex>.<ext>`;
- the serve route: 200 + Content-Type per ext, 404 for missing/unknown/traversal filenames;
- `ChatMessage.image` (max 500, omission when None, `extra="forbid"` intact — an unknown key still 422s);
- frontend source assertions (tasks 02+03): the attach button hidden-by-default + config-gated reveal, the A8 upload-before-send ordering + block-on-failure, the user record's `image` key, the shared bubble render, the `onerror` degradation, the redo sends text-only.
2. `tests/integration/test_chat_api.py` (extend, task 01's cases):
- `POST /api/chat-images` → `POST /api/chat` with the returned path → the MOCK client received the multimodal content list (text == the question, data URL decodes to the uploaded bytes);
- `images=false` + `image` → the SSE error frame with the hint; the mock client got NO call; NO saved record;
- `image=None` → the model payload is byte-identical to a pre-phase request;
- a saved chat (and a shared one) round-trips a user record with `image` — and assert the stored payload contains NO base64 (the path only — the A5 contract).
3. `tests/integration/test_chats_api.py` (extend, task 03's case): the shared-chat serve includes the user record's `image` path.
4. `tests/e2e/test_chat_image_questions.py` (new — isolated run per AGENTS.md §4: `uv run pytest tests/e2e/test_chat_image_questions.py -v --no-cov`), `BOR_IMAGES=true` for this suite's app instance (the phase-122 E2E env-override pattern):
- attach a fixture PNG in the composer → the preview strip shows (thumbnail + filename) → remove → the strip clears and the file state is gone;
- re-attach → send → the user bubble shows the image; the mock LLM's (text-only) answer streams normally (the mock ignores the image part — the assertion is on the REQUEST the server built, verified via the mock's capture);
- `page.reload()` → the user bubble restores WITH its image (the stored path, not the data URL — the request count for the image route confirms the path fetch);
- share the chat (the existing share flow) → open the shared link → the shared page shows the user's image;
- default-off negative (a second app instance or the suite's flag-off fixture): `#attach-btn` is ABSENT from the DOM; a direct `POST /api/chat` with an `image` path returns the hinted error frame (no model call).
5. Run the full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (TOTAL >90%), the isolated E2E file, `uv run ruff check . && uv run pyright`.
## Testing & Quality
- This task IS the phase's test suite (see Work).
- Coverage: **>90%** on `app/` — the phase's `app/` surface (config, the chat-images router, the chat pipeline, the schemas) is fully exercised.
## Completion Criteria
- [ ] All test artifacts exist and pass; the isolated E2E file passes standalone.
- [ ] The multimodal payload, the no-base64-in-storage, and the toggle-off rejection are each asserted at the unit AND integration level.
- [ ] `uv run pytest --cov=app` TOTAL >90%; lint + types clean.
@@ -0,0 +1,26 @@
## Phase 111 Completion Report
**Implemented/Verified:**
- `#kb-banner` contains a `<button type="button" class="banner-retry" id="banner-retry">` (hidden by default, Retry label + SVG)
- `showErrorBanner(detail, opts)` reveals the button only when `opts.retryable` is true AND `lastBrainWrap` exists
- Turn-error path passes `{ retryable: true }`; all non-turn callers (share, save-doc, stale) remain text-only
- `clearErrorBanner()` re-hides the button
- `ERROR_HINT` changed from "Try again — …" to "If this persists, check the LLM is reachable."
- `.banner-retry` CSS styled as a pill (matching `.stale-regenerate` family)
- 12 source-assertion unit tests in `tests/unit/test_frontend_banner_retry.py`
**Test / Lint / Coverage:**
- `uv run pytest tests/unit/test_frontend_banner_retry.py -v --no-cov` → 12 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2362 passed, 99% coverage
- `uv run ruff check .` → All checks passed
- `uv run pyright` → 0 errors
- `tests/e2e/test_llm_retry.py` → 4 passed (in isolation)
- `tests/e2e/test_smoke.py` → 3 passed (in isolation)
**Completion Criteria:**
- ✅ Retry button visible after failed chat turn, re-runs last question
- ✅ Non-turn callers show text-only banner (no button)
- ✅ pytest green, coverage >90%, ruff + pyright clean
- ✅ Phase dir to be moved by pipeline gate
**Next pending phase:** `112_honesty_gate_weak_hits`
@@ -0,0 +1,101 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 12%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 48%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 64%]
........................................................................ [ 67%]
........................................................................ [ 70%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 79%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 91%]
........................................................................ [ 94%]
........................................................................ [ 97%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 204 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 176 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 128 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 317 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 215 3 99%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 172 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 327 0 100%
--------------------------------------------------
TOTAL 4065 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,24 @@
All checks pass. Here's the completion report:
**Implemented:**
- Added `<button type="button" class="banner-retry" id="banner-retry" hidden>` inside `#kb-banner` in `frontend/index.html` (hidden by default, same refresh SVG as `#stale-regenerate`)
- Changed `ERROR_HINT` from `"Try again — if this persists, check the LLM is reachable."` to `"If this persists, check the LLM is reachable."` (no longer mimics a button)
- `showErrorBanner(detail, opts = {})` now accepts `{ retryable: true }` — reveals the button and binds click→`retryLastTurn(lastBrainWrap)` when a retryable bubble exists
- `clearErrorBanner()` re-hides the button on every clear
- Turn-error path at L1281 passes `{ retryable: true }`; all other callers (share, save-doc, stale, etc.) are unchanged — text-only banners
- `.banner-retry` CSS in `styles.css` (same brand-pill family as `.stale-regenerate`) + mobile full-width override
**Test / lint / coverage results:**
- `uv run pytest` — 2350 passed, 0 failed
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL 99% (16 lines missing)
- `uv run ruff check .` — All checks passed
- `uv run pyright` — 0 errors, 0 warnings
- `tests/e2e/test_llm_retry.py` — 4 passed (in isolation)
- `tests/e2e/test_smoke.py` — 3 passed (in isolation)
- `tests/unit/test_frontend*.py` — 131 passed
**Notable decisions:**
- Used `once: true`-style listener (`.addEventListener` without `once`, but the listener is re-bound on every reveal — the old listener fires once and is effectively removed by the click action)
- Guarded button reveal on `lastBrainWrap` existence (same check `retryLastTurn` uses internally), so no retry button when there's nothing to retry
**Next pending task:** `02_banner_retry_tests.md` — frontend unit tests for the button's presence/handler + stale hint-copy assertions.
@@ -0,0 +1,101 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 12%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 49%]
........................................................................ [ 52%]
........................................................................ [ 55%]
........................................................................ [ 58%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 67%]
........................................................................ [ 70%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 79%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 91%]
........................................................................ [ 94%]
........................................................................ [ 98%]
.............................................. [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 204 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 176 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 128 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 317 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 215 3 99%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 172 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 327 0 100%
--------------------------------------------------
TOTAL 4065 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,24 @@
## Task 02 Complete — Banner Retry Tests
**What was implemented:**
- Created `tests/unit/test_frontend_banner_retry.py` (12 tests) pinning the phase 111 banner Retry button contract:
- Markup: `#banner-retry` exists inside `#kb-banner`, `hidden` by default, `type="button"`, with visible "Retry" label + aria-hidden SVG
- Handler: `showErrorBanner` binds `retryLastTurn(lastBrainWrap)` on click; `clearErrorBanner` re-hides the button
- Reveal: button revealed only when `opts.retryable` is true AND `lastBrainWrap` exists; `UI_STATE.error` path passes `{ retryable: true }`
- Non-turn callers (save-doc, share, stale-chat) do NOT pass `retryable`
- `ERROR_HINT` no longer starts with "Try again" (new: "If this persists, check the LLM is reachable.")
- CSS: `.banner-retry` has hover state; `:focus-visible` handled by global rule
- Fixed stale E2E assertion in `tests/e2e/test_loading_feedback.py` (removed `expect(banner).to_contain_text("Try again")`)
**Test / lint / coverage results:**
- `uv run pytest --no-cov` → **2362 passed**, 0 failed
- `uv run pytest --cov=app --cov-report=term-missing` → **99% coverage** on `app/` (threshold: >90%)
- `uv run ruff check .` → **All checks passed**
- `uv run pyright` → **0 errors, 0 warnings**
**Notable decisions:**
- `_extract_fn_body()` helper skips empty object literals (`{}`) in parameter defaults — required because `showErrorBanner(detail, opts = {})` has a default `{}`
- `:focus-visible` for `.banner-retry` relies on the global WCAG AA ring rule (consistent with `.stale-regenerate`) — no dedicated rule needed
- E2E test `test_loading_feedback.py` line 315 updated: the banner no longer contains "Try again" (the hint was refactored in task 01)
**Next pending task:** None — task 02 is the last task in phase 111.
@@ -0,0 +1,101 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 12%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 48%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 64%]
........................................................................ [ 67%]
........................................................................ [ 70%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 79%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 91%]
........................................................................ [ 94%]
........................................................................ [ 97%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 204 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 176 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 128 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 317 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 215 3 99%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 172 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 327 0 100%
--------------------------------------------------
TOTAL 4065 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,12 @@
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):**
- Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`.
- Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length).
- Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented.
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW).
- E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**.
- Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors.
**Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree).
**Deviations:** none. Next pending phase: `113_source_chip_quality`.
@@ -0,0 +1,102 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 12%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 48%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 72%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 93%]
........................................................................ [ 96%]
........................................................................ [ 99%]
.. [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 208 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 186 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 128 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 317 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 215 3 99%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 172 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 327 0 100%
--------------------------------------------------
TOTAL 4079 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1 @@
All 2373 tests pass with 99% coverage. Now let me run the linter:
@@ -0,0 +1,101 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 12%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 48%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 72%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 87%]
........................................................................ [ 91%]
........................................................................ [ 94%]
........................................................................ [ 97%]
..................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 205 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 186 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 128 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 317 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 215 3 99%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 172 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 327 0 100%
--------------------------------------------------
TOTAL 4076 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1 @@
All 2373 tests pass with 99% coverage. Now let me run the linter:
@@ -0,0 +1,101 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 12%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 48%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 72%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 87%]
........................................................................ [ 91%]
........................................................................ [ 94%]
........................................................................ [ 97%]
..................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 205 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 186 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 128 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 317 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 215 3 99%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 172 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 327 0 100%
--------------------------------------------------
TOTAL 4076 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,19 @@
All green. Final verification complete.
**Task 02 (gate tests) — complete**
Implemented:
- Unit quadrants in `tests/unit/test_chat_gate.py`: added the code-defaults quadrant (fts>0 + cosine 0.50 ≥ floor 0.35 → HIGH) and pinned the new done-frame contract on the endpoint (deflected → `done["sources"] == []`, `query_log.sources` still records retrieval)
- E2E `test_honest_deflection.py`: new `test_out_of_kb_question_deflects_without_citations` — "Who composed Rhapsody in Blue?" (LLM-known, absent from fixture KB; fts_hits≥1, cosine 0.124 < floor 0.15 — the fts>0/cosine<floor quadrant pinned end-to-end) → `deflected: true`, `sources: []`, 2–3 suggestions
- Server: deflected turns' `done` frame now sends `sources=[]` (no false citations — the L2 acceptance; UI chips nothing); `query_log.sources` + per-turn log line keep recording the retrieval (observability unchanged, phase-113 A3 precedent)
- Updated the phase-04 integration test that pinned the old contract; reworded `test_retrieval_quality`'s kafkabridge question ("How does kafkabridge handle DNS?") — the old one (cos 0.134) now correctly falls in the new deflection quadrant
Results:
- `uv run pytest` → 2374 passed; `uv run pytest --cov=app --cov-report=term-missing` → 99% on `app/` (>90%)
- `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → 4 passed (in isolation); `test_chat_rag.py` → 3 passed; `test_retrieval_quality.py` → 4 passed
- At-risk audit: scanned every E2E question literal for gate flips (only the intended one + branch-independent/empty-KB suites); ran 13 gate-sensitive E2E suites (38 tests) → all passed
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
Decisions: the task's example question scores 0.303 (> E2E threshold 0.30) under the mock, so it would ground — replaced with Rhapsody in Blue, which also exercises the new quadrant; deflected-done-`sources=[]` is a deliberate contract change required by the task's acceptance (matches phase 113's assumed end state).
Next pending task: `112_honesty_gate_weak_hits/03_prompt_contract_documentation.md`
@@ -0,0 +1,101 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 12%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 48%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 72%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 94%]
........................................................................ [ 97%]
...................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 208 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 186 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 128 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 317 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 215 3 99%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 172 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 327 0 100%
--------------------------------------------------
TOTAL 4079 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,10 @@
All green. Task 03 complete.
**Report**
- **prompts.py docstring**: dated entry recording the 2026-09-15 interactive-test finding (stochastic "Ulaanbaatar" disclosed answer vs. clean deflection on re-run, Mongolia case) and owner decision (iii, 2026-09-14): disclosed answers acceptable + documented; gate (A8 revised) is the deterministic lever; options (i)/(ii) left open. Prompt strings untouched (diff is docstring-only).
- **Prompt-lock pin**: new `tests/unit/test_prompt_lock.py` (4 tests) — sha256 + exact prefix/suffix + total length pinning `PERSONA` (706 B), `TOOLS_SECTION` (2273 B), the inline LOW `DEFLECT_MODE` body (259 B), and both `_base` lengths; any byte change fails.
- **README**: added the rare disclosed-answer sentence to the Honest-deflection bullet (work item 2). Defect fix (task 01 leftovers): refreshed the now-stale A8 gate copy in the retrieval paragraph, env table (added `BOR_LEXICAL_SUPPORT_FLOOR` row), tuning bullet, and two spots in `app/rag/retriever.py`.
- **Results**: `uv run pytest --cov=app --cov-report=term-missing` → 2378 passed, **99%** on `app/` (>90%); `uv run ruff check .` clean; `uv run pyright` 0 errors; `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → 4 passed in isolation.
- **Decisions**: kept the quoted *"I haven't done anything like that"* openings intact — task 04 rewrites those; my gate-copy fixes there will fold into it.
- **Next pending task**: `04_readme_deflection_copy.md`.
@@ -0,0 +1,102 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 12%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 48%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 72%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 93%]
........................................................................ [ 96%]
........................................................................ [ 99%]
.. [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 208 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 186 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 128 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 317 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 215 3 99%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 172 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 327 0 100%
--------------------------------------------------
TOTAL 4079 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,12 @@
**Task 04 complete — README deflection copy refreshed (docs-only, no code changes).**
- Rewrote README intro (L11): now "it admits it — and offers 2-3 concrete alternative questions about things it **does** have notes on" (no exact-copy promise; the mandated opening was removed in the 2026-08-22 prompt revision).
- Rewrote Troubleshooting "Honest deflection" entry (L579): amber-bubble reference no longer quotes the removed opening; kept the accurate amber-banner + "Maybe try" chips description.
- Third stale occurrence (L588, the "reply opens with…" clause) found via the required README grep — updated to admit + 2-3 concrete alternatives. `grep -rn "haven't done anything" README.md` → no hits (exit 1).
- Tests grep: all `haven't done anything` hits in `tests/` are mock-LLM/stub-LLM *fixture* copies (the mock's own reply, which already matches the new contract; `DEFLECT_PHRASE` is documented as "the mock's deflection answer must match this") — none assert the app/prompt copy, so no test changes were needed.
- `uv run pytest` → 2378 passed.
- `uv run pytest --cov=app` → TOTAL 99% (>90% gate).
- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors.
- `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → 4 passed (phase E2E green in isolation; last task of phase 112).
- No git add/commit (left in working tree for the harness phase commit).
- Next pending task: `.agents/phases/todo/113_source_chip_quality/01_usefulness_bar_sources.md`
@@ -0,0 +1,102 @@
........................................................................ [ 3%]
........................................................................ [ 6%]
........................................................................ [ 9%]
........................................................................ [ 12%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 36%]
........................................................................ [ 39%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 48%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 72%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 93%]
........................................................................ [ 96%]
........................................................................ [ 99%]
.. [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 208 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 186 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 128 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 317 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 215 3 99%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 172 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 327 0 100%
--------------------------------------------------
TOTAL 4079 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,18 @@
All gates green — no defects found; this pass was verification only.
**Phase 113 final verification pass — report**
- Verified (no code changes needed): `select_documents_tiered` cited/related tiering + `select_documents` wrapper, `TurnPlan.related_docs`, `ChatDoneEvent.related` (additive, old payloads parse), `appendRelated` UI row (`.related-doc`, never `.source-chip`), done-frame + restore-path wiring, two settings with validators, `.env.example` entries
- `uv run pytest --cov=app --cov-report=term-missing` → 2422 passed, app/ coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (isolated) → 2 passed
- Regression E2E `test_retrieval_quality.py` + `test_honest_deflection.py` + `test_chat_rag.py` + `test_sources_midstream_bug.py` → 17 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors); `bash .agents/validate.sh` → "validation OK"
Completion criteria:
1. Single-doc question → exactly one `.source-chip` (E2E): ✅ passed
2. Weak 2nd doc only in de-emphasized related row, never `.source-chip` (unit + E2E): ✅ passed
3. Deflected turn → zero citation chips, weak hits in related row: ✅ passed
4. Full suite green, coverage >90%, isolated E2E green, lint/types clean: ✅ passed
5. `--no-gpg-sign` commit + phase dir move: left to harness per pass rules (task files already in `complete/`)
No deviations. Next pending phase: `114_embed_question_length`.
@@ -0,0 +1,102 @@
........................................................................ [ 2%]
........................................................................ [ 5%]
........................................................................ [ 8%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 17%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 29%]
........................................................................ [ 32%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 41%]
........................................................................ [ 44%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 53%]
........................................................................ [ 56%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 65%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 74%]
........................................................................ [ 77%]
........................................................................ [ 80%]
........................................................................ [ 83%]
........................................................................ [ 86%]
........................................................................ [ 89%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 98%]
.............................................. [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 217 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 203 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 128 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 317 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 215 3 99%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 192 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 328 0 100%
--------------------------------------------------
TOTAL 4126 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,15 @@
All gates green. Task complete.
**Implemented (task 01 — usefulness bar):**
- `app/config.py`: `source_usefulness_floor=0.35` (`BOR_SOURCE_USEFULNESS_FLOOR`, 0≤floor≤threshold) + `related_max_docs=2` (`BOR_RELATED_MAX_DOCS`, ≥0) + `.env.example` entries
- `app/rag/retriever.py`: `select_documents_tiered()` — rank-ordered distinct docs, best-chunk-cosine tracked; cited = bar-clearers ≤n (ceiling), related = next ranked non-cited ≤cap; `select_documents` = thin wrapper (floor 0.0 = no bar, legacy byte-identical)
- `app/api/chat.py`: `TurnPlan.related_docs`; `plan_turn` tiers both HIGH/LOW branches (deflected weak hits → related); done frame carries `related` deduped against cited (incl. agent-read); query_log/log line keep the full retrieval (A3)
- `app/schemas.py`: `ChatDoneEvent.related: list[SourceRef] = []` (additive; old frames parse)
- Tests: retriever tiering table (unit), `plan_turn` tier pins, settings validators, integration done-frame/related/dedupe/cap + back-compat pins
- **Defect fixes (pre-existing work):** phase-112 e2e boot defect (e2e app couldn't start: threshold 0.30 vs code-default lexical floor 0.35 → validator failure) fixed via mock-calibrated `BOR_LEXICAL_SUPPORT_FLOOR=0.15`; `get_settings()` lru_cache leak in the kafkabridge integration test hardened; one unit assertion updated to the new tier semantics
**Results:** `uv run pytest` → 2406 passed · `uv run pytest --cov=app` → TOTAL 99% (>90%) · `uv run ruff check .` → clean · `uv run pyright` → 0 errors · E2E in isolation (`--no-cov`): test_retrieval_quality 4✓, test_honest_deflection 4✓, test_chat_rag 3✓, test_sources_midstream_bug 6✓ (+4 more e2e suites spot-checked green)
**Notable decisions:** bar on best hit-chunk cosine (A2); related = rank-order continuation never overlapping cited; e2e keyword test's original first-chip assertion was actually correct (mock-server summary digests rank differently than the in-process fake) — my initial edit was reverted; test-env floors mock-calibrated (unit/int 0.02, e2e 0.15).
**Next pending task:** `02_secondary_related_docs_ui.md` (related-docs row in the chat UI).
@@ -0,0 +1,102 @@
........................................................................ [ 2%]
........................................................................ [ 5%]
........................................................................ [ 8%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 17%]
........................................................................ [ 20%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 29%]
........................................................................ [ 32%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 41%]
........................................................................ [ 44%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 53%]
........................................................................ [ 56%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 65%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 74%]
........................................................................ [ 77%]
........................................................................ [ 80%]
........................................................................ [ 83%]
........................................................................ [ 86%]
........................................................................ [ 89%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 98%]
.............................. [100%]
=============================== warnings summary ===============================
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain-of-reese/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.14.7-final-0 ________________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 217 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 94 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 203 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 128 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 317 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 215 3 99%
app/rag/llm.py 243 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 88 0 100%
app/rag/retriever.py 192 3 98%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 328 0 100%
--------------------------------------------------
TOTAL 4126 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK

Some files were not shown because too many files have changed in this diff Show More