Compare commits

...
183 Commits
Author SHA1 Message Date
ducoterra bef24e05e2 phase: 123_chat_image_questions
Build and Push Containers / build-and-push-app (push) Successful in 1m54s
Build and Push Containers / build-and-push-db (push) Failing after 13s
All gates green. Verification complete.

**Phase 123 — final verification pass (all 4 tasks already in `complete/`)**

- Verified the full implementation is in the working tree: `app/api/chat_images.py` (upload/serve pair), `ChatRequest.image`/`ChatMessage.image` (path-validated, omitted-when-None), toggle-off + stale-file hinted error frames, `build_user_content` multimodal build at both sites (chat.py deflected branch + `run_agent`), config-gated composer attach/preview/upload-then-send, restore + shared rendering, CSP `img-src 'self' data:` carve-out, mock-LLM capture buffer.
- `uv run pytest` → **2796 passed**, exit 0 (unit + integration).
- `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (29/4615 missed; phase-123 modules 99–100%).
- `uv run pytest tests/e2e/test_chat_image_questions.py -v --no-cov` → **5 passed** in isolation.
- `uv run ruff check . && uv run pyright` → clean (0 errors).

**Completion criteria:** (1) attach→send→multimodal text+image to the model, bubble/reload/shared all render it, saved chat stores the PATH with `"base64" not in json.dumps(stored)` — **verified** (E2E tests 1–4 + integration round-trip); (2) `BOR_IMAGES=false` — control hidden, exact hinted error frame, zero model calls / no query_log row — **verified** (E2E test 5 + integration); (3) text-only byte-identical (`content` stays a plain `str`) — **verified** (unit + integration); (4) all gates green — **verified**; (5) commit + phase move — left to the harness per pipeline rules (no `git add`/`commit` run).

No defects found; no live-infrastructure changes (repo + local dev DB only). **Next pending phase: none** — 123 is the last phase in `todo/`.
2026-09-25 05:19:18 -04:00
ducoterra a19d78d284 phase: 122_image_documents
Build and Push Containers / build-and-push-app (push) Successful in 1m57s
Build and Push Containers / build-and-push-db (push) Failing after 13s
**Phase 122 (image documents) — final verification pass: all green. No code changes were needed; defects found: none.**

**Verified (implementation already complete in working tree, reviewed end-to-end):**
- Toggle (`BOR_IMAGES`/`BOR_IMAGE_EXTENSIONS`/`BOR_IMAGE_DIR`, off by default) + `GET /api/config` `images` flag
- Ingest: bytes digest, `image_dir` persistent copy, `content = summary = vision description` (chat-model call; only text embedded), fail-soft skip + `images_failed` counter
- Serve/display: `/api/documents/{id}/image` route (404 matrix), viewer `<img>` + description, Sources 48px lazy thumbnails, chat inline source figure (alt = summary), agent `read` marker
- Prune guard: images-off syncs never prune `is_image` docs

**Test / lint / coverage (exact commands & outcomes):**
- `uv run pytest` → exit 0 (green; note: pytest 9.1.1 `-q` omits the final count line in output — exit code authoritative)
- `uv run pytest --cov=app --cov-report=term-missing` → **2715 passed, exit 0, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors, 0 warnings, 0 informations"
- `uv run pytest tests/e2e/test_image_documents.py -v --no-cov` → **4 passed, exit 0** (isolation)

**Completion criteria:** (1) images=true → described/embedded/displayed docs: ✅ (E2E + integration) · (2) images=false byte-identical + image docs survive sync: ✅ (E2E negative app + unit/integration) · (3) viewer + chat rendering with alt text; failed description skips + logs, sync completes: ✅ · (4) test/lint/coverage gates: ✅ · (5) commit + phase move: deferred to harness per this pass's rules (working tree left uncommitted).

**Notable deviation (pre-existing, documented in code):** image route uses `require_user` (phase-79 posture, same gate as the document content endpoint) rather than the phase text's "public" parenthetical — matches the endpoint it mirrors.

**Next pending phase:** `123_chat_image_questions`.
2026-09-25 01:54:23 -04:00
ducoterra 0f77e9a876 phase: 121_git_source_tokens
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Build and Push Containers / build-and-push-db (push) Failing after 14s
**Phase 121 final verification pass — all green** (all 4 tasks already in `complete/`; verified, no defects found, no changes needed)

- Verified implementation vs phase design: migration `0021` (reversible, round-tripped via `alembic downgrade base` + `upgrade head` → head `0021`), `GitSource.token` column, `normalize_credential`/`clone_url_for`/`sanitize_url`, clone callers switched (`sync.py`, `import_docs.py`), masked token fields in add form + editor, `extra="forbid"` output shapes
- Tests: `uv run pytest` → 2662 passed, 0 failed (exit 0); `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (≥90% gate)
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov` → **4 passed**

Completion criteria:
1. Private repo (UI add or pasted embedded-token URL) clones with injected token; token absent from every API response, page text, title attr, and full HTML — **PASS** (integration raw-JSON assertions + E2E `_assert_token_nowhere`)
2. Legacy embedded-token rows still clone from stored URL; output sanitized — **PASS** (`test_sync_legacy_row_clones_with_original_stored_url`, `test_get_masks_legacy_embedded_token_row`, env-fallback masking)
3. Public/local sources byte-identical — **PASS** (verbatim-URL + no-userinfo-unchanged tests)
4. pytest / coverage / ruff / pyright — **PASS** (see above)
5. Commit + phase move — harness responsibility; task files already in `complete/`, changes left in working tree (no commit made, per protocol)

Notable: no deviations; DB left at head, functional. Next pending phase: **122_image_documents** (then 123_chat_image_questions).
2026-09-24 20:51:39 -04:00
ducoterra 3a0fc3db05 phase: 120_failed_turn_retry
Build and Push Containers / build-and-push-app (push) Successful in 3m5s
Build and Push Containers / build-and-push-db (push) Failing after 2m13s
All verification complete. Final report:

**Phase 120 — Failed-turn retry: verification pass (all 3 tasks were done; final verification + 1 regression fix)**

**Verified:** `ChatMessage.failed`/`error` (≤500, `extra="forbid"` intact); `finalizeFailedTurn` funnel on the 3 failure paths (catch-else, stream-drop guard, zero-frame fallback) with `failed: true` + capped detail + `markLastRetryable`; `appendFailedNote` restore branch (Save-as-doc/Tune excluded); `showErrorBanner`/`retryLastTurn` byte-pinned untouched; only the three paths persist `failed: true` (grep + unit pin); no test asserts the old broken behavior.

**Defect found & fixed (rule 7):** a real navigate-away mid-turn let the browser's teardown fetch rejection (TypeError, not AbortError) leak into the failed funnel, persisting a phantom failed brain record — `test_sources_midstream_bug.py::test_no_orphan_brain_message_when_navigated_before_first_token` failed (2 `.msg` after reload) and violated the phase-20 navigate-away convention. Fixed: turn-scoped `leftThePage` flag (set unconditionally on `pagehide`, reset in `runTurn`) skips the funnel in the catch-else branch; pinned by new unit test `test_navigate_away_is_not_a_failed_turn`. No phase-overview/PLAN/todo/complete files touched; no commits made.

**Gates (exact):**
- `uv run pytest` → 2577 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL 4271 stmts, 99% (>90%)
- `uv run pytest tests/e2e/test_failed_turn_retry.py -v --no-cov` → 4 passed (isolated)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- Regression E2E, isolated: `test_sources_midstream_bug.py` 6/6 (was 5/6); `test_llm_retry`/`test_tool_scaffolding_guardrails`/`test_stop_generation`/`test_navbar_refresh` 17/17

**Completion criteria:** (1) network error → banner + in-bubble Retry, re-ask without re-typing ✅ (E2E A); (2) refresh restores failed bubble + working Retry, no "new chat" ✅ (E2E C); (3) stopped/successful turns byte-identical ✅ (negative E2E, stop suite, byte-identity units); (4) pytest/coverage/lint/types ✅; (5) commit + phase move — left to the harness per pass rules.

**Notable:** deviation = the regression fix above (a navigation is not a failed turn; phase-20 partial-persist convention restored). Next pending phase: `121_git_source_tokens`.
2026-09-24 18:50:36 -04:00
ducoterra 0ff1f8c4d6 chore(agent): phase roadmap from TODO.md — 4 phases (120–123)
Protocol B append: failed-turn retry (L3–4), git source tokens (L5),
image documents (L6 ingest), chat image questions (L6 chat side).
TODO.md items now live in .agents/phases/todo/ and the file is cleared.

LLM-Generated: true
2026-09-24 16:27:44 -04:00
ducoterra a5b63f83ad phase: 119_name_signal_read_chips
Build and Push Containers / build-and-push-app (push) Successful in 2m1s
Build and Push Containers / build-and-push-db (push) Successful in 18s
All verification complete. Final report:

**Phase 119 final verification pass — all criteria verified, one stale pin fixed.**
- Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry.
- Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged.
- New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2.
- Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed).
- Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met).
- Next pending phase: **none** — `todo/` holds only phase 119.
2026-09-16 15:50:48 -04:00
ducoterra 795fb56425 add phase 119
Build and Push Containers / build-and-push-app (push) Successful in 15s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-16 10:05:52 -04:00
ducoterra 9820c361b0 phase: 118_summary_seed_context
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 14s
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.

- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
2026-09-16 06:57:49 -04:00
ducoterra 21aad84a6d chore(agent): phase roadmap from TODO.md — phase 118 summary seed context (top-5 summary seeding, all-doc summaries)
Build and Push Containers / build-and-push-app (push) Successful in 17s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-15 22:43:01 -04:00
ducoterra c851d1a1c5 phase: 117_tool_call_compact
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification complete — every gate green. Final report:

**Phase 117 final verification pass — all criteria verified** (tasks 01–04 were already in `complete/`; I re-verified the full gate myself, no code changes needed)
- **Verified:** `appendToolLine` wraps lines in `details.tool-calls-disclosure` (open live, `closeToolCalls` at delta/done/stop + restore-closed, count via textContent); `addToolLines` parity (closed); `.tool-call` deboxed / `code` dechipped; new disclosure/summary CSS; only the one pinned CSS assert updated
- `uv run pytest` → **2466 passed**
- `uv run pytest --cov=app --cov-report=term-missing` → **99%** on `app/` (>90% floor)
- E2E in isolation: `test_tool_call_compact.py` **4 passed** (fold/expand/deboxed-flow/restored-folded); `test_agent_document_tools.py` **4 passed**; `test_big_read_progress.py` **4 passed** (live disclosure open + `.tool-elapsed` visible); `test_thinking_display.py` **5 passed**; `test_smoke.py` **3 passed**
- `uv run ruff check . && uv run pyright` → clean (pyright 0 errors, 0 warnings)
- Completion criteria: pins 1–4 ✅ (E2E above); live "calling tool" state byte-identical ✅ (`test_frontend_tool_states.py` 10 passed incl. `test_calling_tool_label_strings`); scope ✅ — `git diff` limited to the 3 frontend assets + 2 unit tests + E2E story + phase files, **0 changes in `app/`**
- Notable: `TODO.md` carries a **pre-existing uncommitted owner edit** (new retrieval-context idea) already in the tree before this pass — left untouched, will ride along in the harness commit; pre-existing committed quirk: the brain-wave commit's styles.css comment mislabels itself "Phase 117" (cosmetic, out of this diff)
- No commit made (harness commits + moves the phase); **next pending phase: none** — `todo/` holds only phase 117
2026-09-15 18:26:38 -04:00
ducoterra 2ac3fc89c2 feat(ui): swap the in-turn loader for a brain-wave sweep left of the send button
Build and Push Containers / build-and-push-app (push) Successful in 2m11s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Phase 117 (owner request, live-mockup-confirmed): the phase-109 3-dot
cue becomes a compact ECG trace (49px, P/QRS/T) with a brand sweep
traveling the path (bwdraw, 42/140 dash segment, 0.9s loop), the
loader repositioned left of the button so its appearance never shifts
it. setUiState stays the sole owner of the loader's hidden attribute;
the reduced-motion variant stills the sweep. Unit + lifecycle-E2E pins
updated for the new contract.
2026-09-15 11:55:59 -04:00
ducoterra adf31a4a35 phase: 116_modal_scrollbar_theme
Build and Push Containers / build-and-push-app (push) Successful in 2m1s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification complete — every gate re-run independently and green. Final report:

**Phase 116 final verification pass — all green (work was already in the tree; verified, nothing to fix)**

- **Verified** `styles.css`: `--scrollbar-thumb`/`--scrollbar-track` tokens in `:root` (color-mix derivations — follow the saved theme, not in BUILTIN_COLORS; contrast 3.73:1 ≥ 3:1, recomputed independently) + scoped rules `.doc-modal .doc-raw` (overflow-x already owned by the element)
- **Verified** E2E test 8 `test_raw_code_block_themed_scrollbar`: overflow, scroll action, computed-style theme assertions, screenshot
- **Screenshots visually confirmed**: headed `doc_modal_scrollbar_themed_1280.png` shows muted rounded thumb on dark track, no native light-gray bar; headless suite shot records scrolled state (headless Chromium paints no bar UI — documented in test docstring)

**Test / lint / coverage (re-run by me):**
- `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` → **8 passed** (isolation)
- `uv run pytest --cov=app --cov-report=term` → **2457 passed, app/ 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_dark_tech_theme.py -v --no-cov` → **6 passed** (no token collision)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:**
1. Modal scrolls horizontally, scrollbar themed, screenshot shows no native bar — ✅
2. No global scrollbar change (all new selectors under `.doc-modal`; grep-confirmed), regressions green — ✅
3. pytest green, coverage >90%, e2e isolated green, ruff+pyright clean — ✅
4. `--no-gpg-sign` commit + phase move — harness job (left uncommitted in working tree, per rules)

**No defects found; no deviations. Next pending phase:** none — `todo/` contains only phase 116.
2026-09-15 06:11:38 -04:00
ducoterra 990c8adf13 phase: 115_doc_draft_discard
Build and Push Containers / build-and-push-app (push) Successful in 2m12s
Build and Push Containers / build-and-push-db (push) Successful in 14s
All green — this was the final verification pass; everything from the four completed tasks was already in the working tree and verified.

**Phase 115 — Doc drafts: Discard + DELETE route + title fix — verification report**

- Verified all 4 task deliverables present: DELETE route (`app/api/doc_drafts.py`), Discard UI (`doc-edit.html` + `doc-edit.js` + `.discard-draft` CSS), title fix (`defaultDocTitle(wrap)` pairing + `saveAsDoc` call site), and all test pins (integration, frontend unit, E2E). No code changes needed.
- **Completion criteria:**
  1. ✅ Orphaned draft discardable from edit screen; row gone — `test_delete_removes_row_and_invalidates_token` (204 → GET 404), unknown-token 404, admin-gate 403 on all routes, E2E `test_discard_draft_from_edit_screen` all pass.
  2. ✅ Title after retry redo = redone answer's own question — E2E `test_save_title_is_the_redo_question_after_retry` passes.
  3. ✅ Push flow byte-identical — `git diff` shows only the new DELETE route + module docstring; all 7 existing push tests green.
  4. ✅ `uv run pytest --cov=app` → **2457 passed**, app coverage **99%** (>90%); `uv run pytest tests/e2e/test_save_doc_session.py -v --no-cov` → **4 passed**; `uv run ruff check .` → clean; `uv run pyright` → 0 errors.
  5. ⏳ Commit + phase-dir move left to the harness (per executor rules, no `git` run; all changes left in the working tree).
- No defects found; no deviations.
- Next pending phase: none in `todo/` other than this one (`115_doc_draft_discard` is the last).
2026-09-15 05:35:05 -04:00
ducoterra 3846f26a58 phase: 114_embed_question_length
Build and Push Containers / build-and-push-app (push) Successful in 2m6s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification passes complete — the phase was already fully implemented in the working tree, and every gate is green. No defects found; no code changes were needed.

**Final verification pass — Phase 114 (embed question length):**
- Verified truncation: `chat.py:459` embeds `request.message[:settings.embed_question_max_chars]` (default 1200, `BOR_EMBED_QUESTION_MAX_CHARS`, `>0` validator); full question still reaches the LLM prompt/log.
- Verified error mapping: `EmbeddingInputTooLargeError(EmbeddingError)` (byte-identical message) caught **before** `EmbeddingError` → no retry, terminal frame `detail="Question too long — trim it and re-ask."` + reachability-fine hint; `ChatErrorEvent.hint` additive.
- Verified frontend chain: frame `hint` → `err.hint` → `setUiState(error, …, {hint})` → `showErrorBanner(…, {retryable: true})` — hint replaces default `ERROR_HINT`, phase-111 `#banner-retry` button revealed. 4,000-char clamp untouched (A2).
- `uv run pytest tests/unit/test_embed_question_length.py -v --no-cov` → 21 passed
- `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov` (isolation, DB up) → 1 passed (4,000-char question → done, no banner)
- Regression: `test_llm_retry.py` 4 passed · `test_oneshot_llm_retry.py` 2 passed · `test_chip_sizing_question_cap.py` 6 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2444 passed, TOTAL **99%** (>90% gate)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:** (1) 4,000-char question embeds prefix + full prompt ✅ · (2) too-large → accurate frame + hint + Retry button ✅ · (3) reachability failure byte-identical (retries + old copy) ✅ · (4) all gates green ✅ · (5) commit/phase-move → left to the harness per instructions (no `git add`/`commit` run).
**Deviations:** none. **Next pending phase:** `115_doc_draft_discard`.
2026-09-15 04:16:55 -04:00
ducoterra 97d663d16d phase: 113_source_chip_quality
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 15s
All gates green — no defects found; this pass was verification only.

**Phase 113 final verification pass — report**

- Verified (no code changes needed): `select_documents_tiered` cited/related tiering + `select_documents` wrapper, `TurnPlan.related_docs`, `ChatDoneEvent.related` (additive, old payloads parse), `appendRelated` UI row (`.related-doc`, never `.source-chip`), done-frame + restore-path wiring, two settings with validators, `.env.example` entries
- `uv run pytest --cov=app --cov-report=term-missing` → 2422 passed, app/ coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (isolated) → 2 passed
- Regression E2E `test_retrieval_quality.py` + `test_honest_deflection.py` + `test_chat_rag.py` + `test_sources_midstream_bug.py` → 17 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors); `bash .agents/validate.sh` → "validation OK"

Completion criteria:
1. Single-doc question → exactly one `.source-chip` (E2E): ✅ passed
2. Weak 2nd doc only in de-emphasized related row, never `.source-chip` (unit + E2E): ✅ passed
3. Deflected turn → zero citation chips, weak hits in related row: ✅ passed
4. Full suite green, coverage >90%, isolated E2E green, lint/types clean: ✅ passed
5. `--no-gpg-sign` commit + phase dir move: left to harness per pass rules (task files already in `complete/`)

No deviations. Next pending phase: `114_embed_question_length`.
2026-09-15 03:11:05 -04:00
ducoterra 1374faf136 phase: 112_honesty_gate_weak_hits
Build and Push Containers / build-and-push-app (push) Successful in 2m15s
Build and Push Containers / build-and-push-db (push) Successful in 14s
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):**

- Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`.
- Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length).
- Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented.
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW).
- E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**.
- Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors.

**Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree).

**Deviations:** none. Next pending phase: `113_source_chip_quality`.
2026-09-15 00:37:38 -04:00
ducoterra 2683128876 phase: 111_chat_banner_retry
Build and Push Containers / build-and-push-app (push) Successful in 2m24s
Build and Push Containers / build-and-push-db (push) Successful in 14s
## Phase 111 Completion Report

**Implemented/Verified:**
- `#kb-banner` contains a `<button type="button" class="banner-retry" id="banner-retry">` (hidden by default, Retry label + SVG)
- `showErrorBanner(detail, opts)` reveals the button only when `opts.retryable` is true AND `lastBrainWrap` exists
- Turn-error path passes `{ retryable: true }`; all non-turn callers (share, save-doc, stale) remain text-only
- `clearErrorBanner()` re-hides the button
- `ERROR_HINT` changed from "Try again — …" to "If this persists, check the LLM is reachable."
- `.banner-retry` CSS styled as a pill (matching `.stale-regenerate` family)
- 12 source-assertion unit tests in `tests/unit/test_frontend_banner_retry.py`

**Test / Lint / Coverage:**
- `uv run pytest tests/unit/test_frontend_banner_retry.py -v --no-cov` → 12 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2362 passed, 99% coverage
- `uv run ruff check .` → All checks passed
- `uv run pyright` → 0 errors
- `tests/e2e/test_llm_retry.py` → 4 passed (in isolation)
- `tests/e2e/test_smoke.py` → 3 passed (in isolation)

**Completion Criteria:**
- ✅ Retry button visible after failed chat turn, re-runs last question
- ✅ Non-turn callers show text-only banner (no button)
- ✅ pytest green, coverage >90%, ruff + pyright clean
- ✅ Phase dir to be moved by pipeline gate

**Next pending phase:** `112_honesty_gate_weak_hits`
2026-09-14 23:07:56 -04:00
ducoterra f37c517590 chore(agent): phase roadmap from TODO.md — 6 phases (111–116): banner retry, honesty gate, chip quality, embed length, draft discard, modal scrollbar
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 13s
2026-09-14 22:07:32 -04:00
ducoterra 2b75f3cc85 docs(todo): log live testing findings L1-L7 (brain.reeseapps.com session)
Interactive browser test session (15 tests) against the deployed
instance. Findings: L1 banner 'Try again' is plain text, L2 honesty
gate eagerness + stochastic deflection compliance, L3 unstyled modal
scrollbar, L5 recurring weak-hit 2nd source chip, L6 4000-char
question clamp exceeds the embed input cap (500 surfaced as
'unreachable'), L7 save-as-doc drafts have no discard path. Each
entry carries observed evidence, suggested fixes and acceptance.
2026-09-14 21:31:08 -04:00
ducoterra 9ce13b6290 add vulns.md
Build and Push Containers / build-and-push-app (push) Successful in 29s
Build and Push Containers / build-and-push-db (push) Successful in 13s
2026-09-14 16:01:58 -04:00
ducoterra 3a4035fc96 phase: 110_fix_sse_db_pool_exhaustion
Build and Push Containers / build-and-push-app (push) Successful in 2m14s
Build and Push Containers / build-and-push-db (push) Successful in 13s
---

**Phase 110 — Fix SSE DB Connection Pool Exhaustion (SEC-14-04): COMPLETE**

**What was implemented/verified:**
- All three tasks (pool config, short-lived sessions, concurrency cap) were already implemented in code
- Fixed `tests/integration/test_chat_db_sessions.py` — added FakeChatLLM mock, fixed LLM signature (`tools=` not `_tools=`), used `fastapi_app.dependency_overrides` instead of `client.app.dependency_overrides`
- Fixed `tests/e2e/test_chat_db_pool.py` — added FakeChatLLM mock, fixed admin password to match `tests/conftest.py`, removed unused imports
- Fixed lint errors (unused imports, import order) in both test files

**Test / lint / coverage results:**
- `uv run pytest` → 2350 passed, 1 warning, 56.4s
- `uv run pytest --cov=app --cov-report=term-missing` → 99% coverage (4065 lines, 16 uncovered)
- `uv run pytest tests/e2e/test_chat_db_pool.py -v --no-cov` → 3 passed
- `uv run pytest tests/integration/test_chat_db_sessions.py -v --no-cov` → 4 passed
- `uv run pytest tests/integration/test_chat_concurrency.py -v --no-cov` → 11 passed
- `uv run pytest tests/unit/test_db_pool_config.py -v --no-cov` → 14 passed
- `uv run pytest tests/unit/test_agent_short_lived_sessions.py -v --no-cov` → 7 passed
- `uv run ruff check .` → all checks passed
- `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:**
- [✓] `app/db.py::create_engine` receives explicit `pool_size=5`, `max_overflow=10`, `pool_recycle=3600` from settings
- [✓] `run_agent` accepts `db_factory: Callable[[], Session]` and creates short-lived sessions per tool call
- [✓] Each tool round uses a separate DB session closed after the tool result
- [✓] Concurrency cap (`BOR_CHAT_MAX_CONCURRENT`, default 10) limits concurrent turns; excess get 503
- [✓] All test gates green, coverage 99%, lint/types clean

**Notable decisions:** Tests needed LLM mocking (the original test files lacked `FakeChatLLM` mocks, causing hangs on real LLM calls).

**Next pending phase:** None — this is the last phase in `todo/`.
2026-09-14 15:55:13 -04:00
ducoterra 35d65d2f25 feat(rag): summarize single-document folders (MIN_DOCS_PER_FOLDER 2 → 1)
Build and Push Containers / build-and-push-app (push) Successful in 2m10s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Relax the phase-94 folder-summary scope rule from ≥ 2 documents to
≥ 1: a folder (or source root) is a candidate while ANY document
lives under it, so single-file folders and single-file source roots
get their own lite-written description. A row is now pruned only
when its folder loses its last document (vanishes from the
catalogue).

The constant is the single source of truth, so the flip propagates
to the generator's candidate set, the prune pass, the
missing_folder_summaries gap probe (the next sync self-heals the
new gaps), and the KB-tree summary_pending markers (1-doc folders /
sources now read "Summary pending" until their row lands).

Docstrings/comments across app/, scripts/import_docs.py, and the
E2E fixtures updated to the ≥ 1 wording. Unit + integration tests
updated to the new semantics (the pruned-below-minimum scenario is
now a folder losing its LAST doc; single-doc folders are pinned as
candidates/pending). Full suite: 2314 passed, app coverage 99%;
ruff + pyright clean; folder-summary E2E stories pass in isolation
(ls_tree_drilldown, sync_summary_visibility, kb_tree,
kb_tree_nav, document_dates, oneshot_llm_retry).
2026-09-14 08:57:57 -04:00
ducoterra 3a81793565 phase: 109_turn_progress_loader
Build and Push Containers / build-and-push-app (push) Successful in 4m39s
Build and Push Containers / build-and-push-db (push) Successful in 35s
All gates green. Final report:

**Phase 109 — final verification pass (all 3 tasks were already complete; no re-implementation needed)**

- Verified D15 (thinking-handler re-open `block.open = true` after `ensureThinkingBlock`; delta keeps closing; "never reopens" narrative removed; restore path still collapses) and D16 (single static `#turn-loader` in composer row, `aria-hidden`, sole `turnLoader.hidden = !inFlight` write in `setUiState`; CSS reuses `typing` keyframes + reduced-motion override after the main rule + provenance; global `[hidden]` guard intact; mock `TURN_PROGRESS_TRIGGER` repro marker; `app/` untouched — pure UI phase).
- Unit pins: `tests/unit/test_frontend_turn_loader.py` (9 tests) — green via full run.
- `uv run pytest tests/e2e/test_turn_progress_loader.py -v --no-cov` → **3 passed** (isolation, DB up): loader visible send→tool-gap→final-thinking, hidden after done; re-opened scratchpad carries `marker-thought-42`/`marker-final-thought-42`; `#send-status` carries state text, back to empty idle; reduced-motion context stills dots (computed `animation-name: none`) with loader still visible.
- Regressions in isolation: `test_thinking_display` 5 passed, `test_stop_generation` 3 passed, `test_big_read_progress` 4 passed, `test_loading_feedback` 5 passed.
- `uv run pytest` → **2314 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90%); `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings.
- All terminal paths (done→idle L2554, stream error→error L2543, timeout→error L2247) funnel through `setUiState` — never-stale by construction.
- No defects found; no code changes made in this pass. Commit + phase-dir move left to the harness per pipeline rules (working tree carries all changes).
- Next pending phase: `108_history_wire_check` (still in `todo/`; out of scope here).
2026-09-14 01:27:16 -04:00
ducoterra fbbd98d734 phase: 108_history_wire_check
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 12s
Final verification pass complete — everything independently re-verified green. Report:

**Phase 108 — History wire check (final verification pass; verdict: NO BUG, tests-only per D13)**

- Verified all 3 tasks complete in `.agents/phases/complete/108_history_wire_check/`; re-ran every pin and gate myself
- Layer 1 (unit): 2 owner-shape pins in `tests/unit/test_history.py` (2-turn keep-whole + A4 thinking mapping) — pass
- Layer 2 (integration): `test_endpoint_two_turn_history_reaches_the_llm` — captured LLM request = `[system, user Q1, assistant R1, user Q2]`, `history_msgs=2` — pass
- Layer 3 (E2E): `tests/e2e/test_history_wire_check.py` — owner's verbatim 3-message scenario, echo byte-exact (T2: `2 prior messages`+R1 tail; T3: `4 prior messages`+R2 tail) — pass
- `VERDICT.md` present in phase dir: layer outcomes + "no bug — model behavior/user error" + pin names

**Gates (exact results)**
- `uv run pytest --cov=app --cov-report=term-missing` → 2305 passed, TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` → 2 passed (isolation, DB up)
- `uv run pytest tests/e2e/test_llm_history.py -v --no-cov` (phase-74 regression) → 3 passed
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors

**Completion criteria:** all six met — no defects found; diff scoped to `tests/**`, `VERDICT.md`, `.agents/phases/**`, `.agents/reports/**` (no `app/`/`frontend/` changes, consistent with no-bug verdict). Per executor rules, no git commit made — left for the harness.

**Next pending phase:** `109_turn_progress_loader`
2026-09-13 23:57:54 -04:00
ducoterra 6bcee831ec phase: 107_git_full_history_dates
Build and Push Containers / build-and-push-app (push) Successful in 1m49s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All green — verification complete. Final report:

**Phase 107 — final verification pass (all tasks already in `complete/`)**

- **Verified:** `clone_or_pull` full-history fix (no `--depth` clone; `--is-shallow-repository` probe → one-time `git fetch --unshallow` self-heal → `git pull --ff-only`; D12 fail-loud) in `scripts/git_sync.py`; E2E suite `tests/e2e/test_git_source_dates.py` present; all five stale shallow/tip-date narratives corrected (no residual claims repo-wide; `docs_push.py` `--depth` fetches untouched as designed).
- **Unit+integration:** `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py tests/integration/test_import_docs_git.py tests/integration/test_sync_api.py -v --no-cov` → **66 passed** (D10 tip-date test replaced by `test_url_clone_yields_true_per_file_dates` + `test_existing_shallow_checkout_self_heals`; unit pins: clone argv w/o `--depth`, probe→unshallow→pull order, unshallow-failure → `GitSyncError`).
- **New E2E in isolation:** `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` → **4 passed** (old file 2020-06-15 ≠ tip file 2024-06-15 in `GET /api/docs`, Sources `Created`/`Updated` columns, viewer `Created` badge ISO title; folder `updated_at` subtree maxes).
- **Phase-106 regressions in isolation:** `test_document_dates.py` → 6 passed; `test_sync_button.py` → 3 passed; `test_git_sources_admin.py` → 6 passed.
- **Full gates:** `uv run pytest` → **2302 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90%); `uv run ruff check . && uv run pyright` → **clean, 0 errors**.
- **Completion criteria:** 1 ✅ non-shallow URL clone + true per-file dates · 2 ✅ self-heal / plain-pull / fail-loud · 3 ✅ both suites green, bug-pin replaced · 4 ✅ E2E green in isolation · 5 ✅ all regressions + full suite + coverage + lint · 6 ✅ narratives corrected · 7 — no commit made (harness override: changes left in working tree; task files already in `complete/`).
- **No defects found; no deviations.** Next pending phase: **108_history_wire_check**.
2026-09-13 23:21:39 -04:00
ducoterra addbd4ca08 add phase 107
Build and Push Containers / build-and-push-app (push) Successful in 34s
Build and Push Containers / build-and-push-db (push) Successful in 26s
2026-09-13 22:28:04 -04:00
ducoterra e2d08a95a9 chore(agent): phase roadmap from TODO.md, 2 phases (108 history-wire check, 109 turn progress loader) 2026-09-13 22:27:05 -04:00
ducoterra ee3efb28c9 phase: 106_document_dates
Build and Push Containers / build-and-push-app (push) Successful in 4m35s
Build and Push Containers / build-and-push-db (push) Successful in 14s
Everything is verified green. Final report:

**Phase 106 — Document dates (final verification pass; all 10 tasks already complete)**

- Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed.
- `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up)
- 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation**
- `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings**

**Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions.

- **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring).
- **Next pending phase:** none — `todo/` holds only this phase.
2026-09-13 19:28:05 -04:00
ducoterra cec819743d add phase 106 - document dates
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 14s
2026-09-13 12:18:01 -04:00
ducoterra 8476dc1e07 phase: 99_kb_tree_table_and_back_nav
Build and Push Containers / build-and-push-app (push) Successful in 1m55s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification is complete and green. Final report:

**Phase 99 — final verification pass (all 3 tasks already in `complete/`)**

- Verified the one-line Description clamp: `.kb-desc-cell` flex wrapper + `.kb-desc-text` ellipsis triad + `nowrap` column + in-cell Edit button + hover `title` + untouched level block (source diffs match D1; unit pins present)
- Verified the back-button breadcrumb nav: `applyTarget(target, push)` state-only `pushState`, no-duplicate gate, popstate adopt/reset, `bor:view-refresh` alignment before `loadTree()`, anonymous gate listener-free, router.js untouched (D2)
- `uv run pytest --cov=app`: **2180 passed**, coverage **99%** (>90% gate)
- `uv run ruff check .`: clean; `uv run pyright`: **0 errors, 0 warnings**
- `uv run pytest tests/e2e/test_kb_tree_nav.py -v --no-cov` (isolated, DB up): **4/4 passed** (back-walk, jump-then-back, fresh-nav/active-reclick, measured clamp incl. row-height parity ±4px, computed styles, full text in DOM/title/level block)
- Regression suites isolated: `test_kb_tree.py` 8✓, `test_ls_tree_drilldown.py` 3✓, `test_navbar_refresh.py` 7✓, plus `test_sync_summary_visibility.py` 3✓ (marker pin updated to `kb-desc-text kb-summary-pending` — the sanctioned class-pair change) and `test_edit_summaries.py` 4✓
- Criteria 1–4: **met** (each as above). Criterion 5 (atomic commit): left to the harness per executor rules — no `git add`/`commit` run; all changes left in the working tree
- No defects found in prior phases; no deviations. Stray `__pycache__/test_zz_smoke_tree_ui.*.pyc` is a leftover (no `.py` on disk) — inert, not touched
- Next pending phase: **none** — `todo/` will be empty once the harness moves this phase
2026-09-13 01:57:40 -04:00
ducoterra f665a83b1a phase: 98_sync_summary_visibility
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All verification complete. Final report:

**Phase 98 — Sync summary visibility: final verification pass** (all 5 tasks already complete; implementation verified against the design, no defects found, no code changes needed)

- **Implementation checked:** `SyncStatus` phase machine (4 new keys, terminal-keep counts), `on_progress` hook in `generate_folder_summaries`, `summary_pending` on `KbTreeSource`/`KbTreeFolder` + D3 rule in `build_kb_tree`, phase-aware sync labels + pending UI in `sources.js`, `.kb-summary-pending` CSS — all match decisions D1–D5.
- **Unit + integration:** `uv run pytest` → 2184 tests, 0 failed/errors (exit 0)
- **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **99%** on `app/` (criterion >90% ✓; `app/api/sync.py` and `app/rag/folder_summaries.py` at 100%)
- **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- **Phase E2E (isolation):** `uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov` → **3 passed** (phase machine, live label, pending markers + gap-fill self-heal)
- **Regression suites (each isolated, `--no-cov`):** test_kb_tree ✓, test_ls_tree_drilldown 3 ✓, test_sync_button 3 ✓, test_sync_upload_progress 4 ✓, test_oneshot_llm_retry 2 ✓, test_local_directory_sources 3 ✓
- **Completion criteria:** all 7 verified green — status phase fields + terminal semantics; `Writing KB overview…`/`Summarizing folders… (n/m)` labels (title + aria-live); pending set == `missing_folder_summaries` (integration cross-check pinned at `test_docs_api.py:428`); CLI/`ls` byte-identity (no changes to those paths, pins green); suite/coverage/lint gates; dedicated + regression E2E. Commit left to the harness per protocol (no `git add`/`commit` run).
- **Decisions/deviations:** none — no fixes were required this pass.
- **Next pending phase:** `99_kb_tree_table_and_back_nav`.
2026-09-13 00:23:05 -04:00
ducoterra 909c96c7bc minor fix to prevent null bytes from breaking sync
Build and Push Containers / build-and-push-app (push) Successful in 1m54s
Build and Push Containers / build-and-push-db (push) Successful in 19s
2026-09-12 22:12:09 -04:00
ducoterra d731169b8b phase: 105_hidden_folders_toggle
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All completion criteria verified. Everything is green.

**Phase 105 final verification pass — all criteria verified**

- Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page
- Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified)
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓
- Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation)
- Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed
- Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings)

**Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched).

**Deviations:** none — no defects found; no code changes were needed on this pass.

**Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`.
2026-09-12 21:26:30 -04:00
ducoterra ecc921098a phase: 104_chip_sizing_question_cap
Build and Push Containers / build-and-push-app (push) Successful in 1m48s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All completion criteria verified. Final report:

**Phase 104 — final verification pass: all green**
- Verified (no code changes needed): single-line ellipsized `.suggestion-chip` CSS + deleted `.maybe-try` override, `renderChips` full-text `title` + `aria-label`-when-clipped, `maxlength="4000"` + `#char-count` counter + `handleSend` over-cap guard, unit pins file, schemas boundary pins, dedicated E2E suite.
- E2E (isolation): `uv run pytest tests/e2e/test_chip_sizing_question_cap.py -v --no-cov` → **6 passed**; regressions: `test_suggestion_chips.py` 8 passed, `test_pinned_composer.py` 4 passed, `test_responsive_polish.py` 7 passed, `test_chat_history.py` 5 passed.
- `uv run pytest` → **2102 passed**; `--cov=app` → **99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- Criteria: chip E2E (single-line, clipped, title+aria-label full text) ✅; paste caps at exactly 4,000, send streams, counter hides ✅; programmatic 5,000-char fill → banner, no turn, text kept ✅; 4,000/4,001 boundary pinned + HTML maxlength == JS constant cross-file pin ✅.
- Diff scope: `frontend/`, new unit file, `tests/unit/test_schemas.py`, new E2E file, phase files — **no `app/` diff, no migration, no `shared.js` diff**.
- Deviations: 4 regression test files touched — 2 genuine DOM-pin conflicts from the new `#char-count` child (explicitly anticipated by the overview) + 3 documented **pre-existing E2E flake fixes** (smooth-scroll race, tab-walk heuristic, 10 ms timeout), each verified pre-existing on the pre-phase-104 tree.
- No commit made (harness commits per the execution protocol override).
- Next pending phase: `98_sync_summary_visibility`.
2026-09-12 19:45:00 -04:00
ducoterra 1f1c01c9f7 phase: 103_suggestions_session_openers
Build and Push Containers / build-and-push-app (push) Successful in 2m32s
Build and Push Containers / build-and-push-db (push) Successful in 12s
Phase 103 final verification pass — all green.

**Verified (all 3 tasks already in `complete/`; no code changes needed):**
- `opening_questions` in `app/api/suggestions.py` — forward walk, one opener per chat (first non-blank user msg, A3), reads raw `messages` not `title` (A4), phase-80 order/dedup/cap/seed contracts; `last_questions` name gone from `app/`+`tests/`
- Docs updated: `app/config.py` seed docstring, `.env.example` `BOR_SUGGESTIONS`, `README.md` — "session openers" wording
- Diff scope correct: only the 6 expected files + phase-file moves; `app/rag/suggestions.py` and `frontend/` untouched

**Test / lint / coverage results:**
- `uv run pytest tests/integration/test_suggestions_api.py -v` → 12 passed
- `uv run pytest tests/e2e/test_suggestion_chips.py -v --no-cov` → 8 passed in isolation (opener-only core pin included)
- `test_responsive_polish.py` → 7 passed; `test_chat_persistence.py` → 4 passed (both isolated, no edits)
- `uv run pytest --cov=app --cov-report=term-missing` → 2086 passed, TOTAL 99% (>90%); `app/api/suggestions.py` 100%
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:** all 7 ✅ (follow-ups-never-surface pin; cap-across-chats pin; seed/dedup/case/partial/A3/401 pins; E2E suites isolated; deflection chips unchanged; full suite + lint; commit + dir move left to harness per executor rules).

**Deviations:** none — no defects found; nothing changed in this pass.
**Next pending phase:** `98_sync_summary_visibility` (numeric order in `todo/`).
2026-09-12 16:37:50 -04:00
ducoterra 3b2dea5685 phase: 102_extensionless_filenames
Build and Push Containers / build-and-push-app (push) Successful in 1m38s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification complete — every gate green, no defects found in previously completed work.

**Phase 102 final verification pass — report**

Verified (all three task files present in `complete/`; working-tree implementation matches D1–D5 design):
- `match_extension` choke point in `app/rag/importer.py` (walk + `formats` counter), `doc_format` name-token badge in `app/api/docs.py`, config/`.env.example` docs, fixture `tests/fixtures/extensionless_kb/`, integration + E2E suites — all present and correct
- Completion criteria: end-to-end sync (✓ integration + E2E), case matrix incl. `mydockerfile`/`Dockerfile.dev`/`.dockerfile` exclusions (✓ unit), `formats=dockerfile:1` not `unknown` (✓ log-line assertion), badge `dockerfile`/`containerfile` + `text` fallback + suffixed unchanged (✓ unit/integration/E2E), prune-on-token-removal (✓ `pruned==2`), suffixed-path rule byte-identical (✓ single-line swap, existing cases untouched)

Test / lint results (exact commands):
- `uv run pytest --cov=app --cov-report=term-missing` → 2084 passed, **99%** coverage (>90% gate)
- `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` → 2 passed, isolated, DB up
- Regressions isolated: `test_import_documents` 3✓, `test_import_extensions_env` 2✓, `test_quadlet_jinja_import` 4✓, `test_document_viewer` 7✓, `test_kb_tree` 8✓
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings

Notable: commit intentionally not made (harness commits the phase); 102's task files already sit in `complete/`, overview stays in `todo/` for the harness.
Next pending phases: 98, 99, 103, 104, 105 (numeric next after 102: `103_suggestions_session_openers`).
2026-09-12 15:56:43 -04:00
ducoterra 4dbac1660a phase: 101_tokens_page_overhaul
Build and Push Containers / build-and-push-app (push) Successful in 1m49s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All gates verified green — no defects found, nothing to fix. Final report:

## Phase 101 — final verification pass (all tasks already complete; verified, not re-implemented)

**Verified against the working tree** (task files 01–04 present in `complete/`):
- Active/revoked table split (Status column gone, section hidden while 0 revoked, per-table search with query state surviving re-renders) in `frontend/index.html` + `tokens.js`
- `generateToken()` refuses blank/whitespace names (`Give the token a name first.`, no request); `|| "token"` fallback deleted
- Atomic rotation: `regenerate_token` service + `POST /api/tokens/{id}/regenerate` (201 new-plaintext-once / 404 / 409 / router-wide 403) + `TokenListItem.revoked_at` (D5)
- Regenerate two-step confirm UI + CSS (`.token-regenerate`, neutral hover, no new hue); A4 pins intact

**Test / lint / coverage results:**
- `uv run pytest` → **2065 passed**
- `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90% ✓)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- `uv run pytest tests/e2e/test_tokens_page.py -v --no-cov` → **4 passed** (isolation, DB up)
- Regression, each in isolation: `test_api_tokens.py` **9 passed**, `test_admin_auth.py` **6 passed**, `test_shared_header.py` **6 passed**, `test_theme_semantic_completion.py` **8 passed** (its revoked-pill pin was correctly re-scoped to the revoked table in this phase)

**Completion criteria:** 1 ✓ split+search (E2E 1–2) · 2 ✓ required name (E2E 3 + source pin) · 3 ✓ rotation end-to-end, old token refused at gate (E2E 4 + API 404/409 pinned) · 4 ✓ A4 holds (list carries no plaintext/hashes) · 5 ✓ suite/coverage/lint green · 6 ✓ E2E + regressions green in isolation · 7 commit left to the harness per executor rules (all changes uncommitted in the working tree)

**Deviations:** none. Next pending phase: `98_sync_summary_visibility`.
2026-09-12 15:16:02 -04:00
ducoterra 58e9d94cff phase: 100_page_width_consistency
Build and Push Containers / build-and-push-app (push) Successful in 4m3s
Build and Push Containers / build-and-push-db (push) Successful in 18s
**Phase 100 — final verification pass: all green.**

- Verified the shipped CSS contract directly: `--chat-column: 72rem` in `:root`; 0 literal `max-width: 46rem`; no `@media (min-width: 1500px)` block; exactly 4 token-capped reading columns; tuning/theme/doc-edit shells cap-free, structurally `.sources-shell`; `mock_llm.py` diff is comment-only.
- Defect found & fixed (phase-93 suite): `test_theme_semantic_completion.py::test_reset_removes_tag_byte_identical` raced theme.js's post-PUT refetch — it asserted the `#bor-theme` tag was gone right after the result line, but tag removal lands in the reconcile after the re-fetch (failed ~1 in 5 runs after `test_chat_rag`, reproduced). Fixed with an auto-waiting `expect(...).to_have_count(0)` (settled-state wait, the file's existing house pattern). 6/6 clean on the repro loop after.
- Tests/lint/coverage: `uv run pytest --cov=app --cov-report=term-missing` → 2052 passed, **99%** on `app/`; `uv run ruff check .` + `uv run pyright` → 0 errors.
- E2E in isolation (all passed): `test_wide_desktop_column.py` 3 (the phase suite — chat==tuning==theme==RAG ±4px at 1280 & 1920, ≈1152px; shared ≈1152px; standalone doc ≈1112px; modal unchanged ≈1100px; 360px overflow-free), `test_ui_customization` 4, `test_admin_theme_tab` 5, `test_document_viewer` 7, `test_save_share_ux` 5, `test_sticky_navbar` 3, `test_markdown_tables` 6, `test_responsive_polish` 7, `test_chat_rag` 3, `test_theme_semantic_completion` 8.
- Completion criteria: ① measured 72rem everywhere (≥~1200px, ±4px) + full-width below + 360px clean — **PASS** (E2E); ② zero 46rem rules / no 1500px block / four token selectors — **PASS** (grep + unit pins); ③ B4 byte-identical no-op + mobile squeeze — **PASS** (theme + responsive suites green); ④ full suite / coverage / lint — **PASS**; ⑤ atomic commit — left to the harness per executor protocol (all changes in the working tree, uncommitted).
- Deviations: none from phase decisions; only change in this pass is the race fix above (test-only, behavior unchanged).
- Next pending phase: `98_sync_summary_visibility` (numeric order in `todo/`; also pending: 99, 101–105).
2026-09-12 13:37:19 -04:00
ducoterra a2ca2f905f chore(agent): phase 105 (hidden-folders toggle) from TODO.md, clear the list 2026-09-12 11:55:03 -04:00
ducoterra ffa6bda3e5 add phases for fixing suggestion chips
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-12 10:52:51 -04:00
ducoterra 17dd3bfac1 docs(plan): remove hardcoded phase/migration references to prevent staleness
Build and Push Containers / build-and-push-app (push) Successful in 14s
Build and Push Containers / build-and-push-db (push) Successful in 16s
Refactor §12 roadmap to point to ls commands and 00_phase.md files
instead of listing specific phase numbers and descriptions.

Remove ~20 phase-number references from anchors, revision notes,
tool surface, feedback, and sync sections — keep only the one
stable reference (phase 03 convention).

Replace migration ranges with 'ls alembic/versions/'.
Replace hardcoded retry values with 'configurable (defaults: ...)'.
Status header now points to §12 instead of listing counts.
2026-09-12 10:22:45 -04:00
ducoterra 800bea769a docs(phases): author todo queue phases 98-102
98 sync summary visibility (status phases + pending markers), 99 catalog
one-line clamp + breadcrumb back-nav, 100 72rem page-width consistency,
101 tokens page overhaul, 102 extensionless filename import.
2026-09-12 10:15:22 -04:00
ducoterra ad7585d474 phase: 97_kb_tree_catalog
Build and Push Containers / build-and-push-app (push) Successful in 2m11s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All completion criteria verified — everything is green, no defects found. Final report:

## Phase 97 final verification pass — ALL GREEN

**Verified (no code changes needed):**
- `GET /api/docs/tree` (admin), `build_kb_tree` pure builder, `PATCH /api/folders/summary`, migration 0018 (`manually_edited`, head confirmed), generator skip/keep + `kept_manual` stat, RAG tree UI + edit affordance in `sources.js`/`index.html`/`styles.css`
- `tests/e2e/test_kb_tree.py`: 8 passed — top level, drill source/folder, edit round-trip, clear, manual-desc-survives-sync, reload fallback, anonymous gate
- Integration: tree shape/order/403/empty/indexed-only + PATCH update/create/root/clear/404/403/no-LLM + stat-walk equivalence (in `test_docs_api.py`); 3-field `folder_summaries=` import token preserved

**Gates (exact commands):**
- `uv run pytest --cov=app --cov-report=term-missing` → **2053 passed**, TOTAL coverage **99%** (>90% ✓)
- `uv run ruff check . && uv run pyright` → **All checks passed / 0 errors**
- `uv run pytest tests/e2e/test_kb_tree.py -v --no-cov` → **8 passed** in isolation
- 30 story/RAG-view E2E suites run **one per process**: all passed, incl. `test_ls_tree_drilldown` (agent `ls` byte-identical ✓), `test_import_documents`, `test_edit_summaries`, `test_admin_auth`, `test_kb_overview`

**Completion criteria:** tree view ✓ · edit round-trip + clear ✓ · manual persists/clear resets ✓ · `ls` unchanged ✓ · pytest/coverage/lint ✓ · E2E isolation ✓ · commit — left to harness per protocol (working tree untouched, `git add/commit` not run)

**Deviations:** none. **Next pending phase:** none — `todo/` contains only 97 (96 already committed).
2026-09-11 22:48:02 -04:00
ducoterra a49be80b8e phase: 96_oneshot_resilience
Build and Push Containers / build-and-push-app (push) Successful in 1m34s
Build and Push Containers / build-and-push-db (push) Successful in 10s
All checks complete. Final report:

**Phase 96 (one-shot resilience) — final verification pass, all green** (all 4 task files already in `complete/`; verified the working-tree implementation against the design)

- `LLMClient.chat()` empty-content retry (D1–D3) via `_chat_once` + `_EmptyContentError` (carries `finish_reason`), under `BOR_LLM_RETRIES`/`BOR_LLM_RETRY_DELAY` — verified in diff
- `missing_folder_summaries()` + `generate_folder_summaries(only_missing=…)` — verified; `folder_summary_table_empty` deleted, both sync gates switched to the gap probe
- `.env.example` comments updated (chat-turn stream + one-shot summary calls)

**Test / lint / coverage results**
- `uv run pytest --cov=app --cov-report=term-missing` → **1988 passed**, coverage **99%** (gate >90%)
- `uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov` → **2 passed** (isolated)
- Regressions, each isolated: `test_ls_tree_drilldown` 3 passed, `test_sync_button` 3 passed, `test_local_directory_sources` 3 passed, `test_llm_retry` 4 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:** retry-then-recover unit-pinned ✓ · exhaustion + `BOR_LLM_RETRIES=0` byte-identical ✓ · streaming path untouched ✓ · gap-fill both sync paths, other rows byte-identical incl. `updated_at` ✓ · no-gap zero-burn ✓ · phase E2E green ✓ · regression E2Es green ✓ · full suite + >90% + lint/types ✓ · no completed-phase behavior change (full suite green) ✓. Commit left to the harness per executor rules (working tree, 16 files).

**Deviations:** none. **Next pending phase:** `97_kb_tree_catalog`.
2026-09-11 13:16:20 -04:00
ducoterra bcaef800c5 phase: 95_read_truncation_cap
Build and Push Containers / build-and-push-app (push) Successful in 1m38s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All gates are green. Final verification is complete — no fixes were needed; the phase is fully implemented and passing.

**Phase 95 final verification report**

- Verified all 3 tasks already implemented (task files in `complete/`): read cap + notice + `ToolResultPiece` (task 01), `tool_result` SSE + live/saved/shared marker (task 02), E2E suite (task 03).
- Code checks passed: `BOR_READ_MAX_CHARS` (default 128 000, validator), pinned `READ_TRUNCATION_NOTICE` + `TRUNCATION_MARKER`, byte-identical at/under-cap read, holder `read_truncations` (no `tool_calls` impact), piece order (after `tool`, before next round), `ChatToolResultEvent`, `ToolCall` back-compat fields, `app.js` live/restore handlers, `shared.js`, `.truncated-note` CSS, `.env.example` entry, mock-LLM echo script.
- **Tests:** `uv run pytest` → 1966 passed; `uv run pytest --cov=app --cov-report=term-missing` → all green, TOTAL **99%** (>90% gate).
- **E2E:** `uv run pytest tests/e2e/test_read_truncation_cap.py -v --no-cov` → **3 passed** (frame order + live marker + LLM notice via echo; save→shared fidelity; short-read control).
- **Regression (isolated):** `test_agent_document_tools` 4 ✓, `test_chat_history` 5 ✓, `test_share_chat` 4 ✓, `test_big_read_progress` 4 ✓, `test_stop_generation` 3 ✓.
- **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings.

**Completion criteria:** ① over-cap read → first-cap-chars + marker + pinned notice — ✓ (unit-pinned: at-cap/cap+1/notice tests); ② user marker live/saved/shared — ✓ (E2E + frontend tests); ③ at/under cap byte-identical, no frame — ✓ (unit + control E2E); ④ top-2 `<documents>` retrieval untouched — ✓ (`app/rag/retriever.py` unmodified vs HEAD); ⑤ suite green, >90% coverage, ruff+pyright clean — ✓; ⑥ no completed-phase behavior change — ✓ (all gates green; commit left to harness per pass rules).

- No defects found; no changes made this pass. Next pending phase: none in `todo/` (96 is the next free number).
2026-09-11 03:42:51 -04:00
ducoterra d4943b4822 phase: 94_ls_tree_drilldown
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 25s
All green. Verification complete.

**Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)**

- Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal
- Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths
- Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met
- `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched)
- Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed
- Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol)

**Next pending phase:** `95_read_truncation_cap`
2026-09-11 00:59:35 -04:00
ducoterra 9188be259b phase: 93_theme_semantic_completion
Build and Push Containers / build-and-push-app (push) Successful in 1m56s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All verification is complete. Final report:

**Phase 93 — Theme semantic completion: FINAL VERIFICATION PASS — ALL GREEN**

- Verified full implementation in tree: migration `0016` (8 nullable semantic columns, applied at head), 17-var `BUILTIN_COLORS`/`COLOR_FIELDS`/`effective_settings`, API validation, `#view-theme` State-colors fieldset (17 pickers), `theme.js` FIELDS/PAIRS (5→8), `.page-head` surface panel (6 shell views + doc-edit + shared.html; login card / document sticky header audited as already-surfaced), mock_llm `content: None` fix
- Fixed 2 pre-existing defects (both fail identically on baseline `d4f38ad`, proven via worktree A/B): `test_nav_rename_sources` — expected nav tail missing the phase-91 "Theme" link; `test_stale_ui_copy` — now truncates `saved_chats` before/after (house `test_suggestion_chips` pattern) so the seed-chip contract is deterministic on the shared dev DB (owner's 22 saved chats triggered phase-80 last-3-questions)
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → **1868 passed, app/ 99%** (>90% ✓); `uv run ruff check .` → clean; `uv run pyright` → **0 errors**
- E2E: dedicated `uv run pytest tests/e2e/test_theme_semantic_completion.py -v --no-cov` → **8/8 in isolation** (all-gray 17-color theme: zero residual color on saved-result/Stale/Revoked/Local/tool-call elements, text labels intact, gray heads non-transparent, pre-paint tag, Reset → byte-identical no-tag); 15 theme/header/nav/responsive suites green in isolation; full 85-file combined run: only the 2 fixed pre-existing failures + 1 combined-run artifact (`test_sync_upload_progress`, green in isolation)
- Completion criteria: (1) monochrome E2E ✓ (2) default byte-identical, no `#bor-theme` tag ✓ (3) all page heads on solid surface ✓ (4) suite/coverage/lint/E2E green ✓ (5) phases 01–92 no behavior change ✓ (6) commit left to harness per protocol
- Notable: cleaned stray uvicorn leftovers from prior implementation pass (owner's `--reload` dev server untouched); no deviations from the phase design
- Next pending phase: `94_ls_tree_drilldown`
2026-09-10 16:43:08 -04:00
ducoterra d4f38ad3ce add PLAN.md
Build and Push Containers / build-and-push-app (push) Successful in 39s
Build and Push Containers / build-and-push-db (push) Successful in 25s
2026-09-10 13:08:55 -04:00
ducoterra bf308eb795 chore(agent): phase 93-95 roadmap from TODO.md — theme completion, ls tree drill-down, read truncation cap
Convert the three unchecked TODO.md items into an executable phase
roadmap (Protocol B, appended after phase 92):

- 93_theme_semantic_completion (TODO L3): the ok/err/accent state
  families become Theme-tab-controlled (B3 revised, owner permission
  2026-09-10) + surface panels behind every page head
- 94_ls_tree_drilldown (TODO L4): ls becomes a source -> folder ->
  file tree with sync-time lite-model folder summaries; controlled
  tool-calling battery as the accuracy/performance gate
- 95_read_truncation_cap (TODO L5): read capped at BOR_READ_MAX_CHARS
  (128k chars ~= 32k tokens, spec'd on the 128k-token minimum context),
  LLM-visible truncation notice pointing at grep, new tool_result SSE
  event (A15 extension) + the visible UI marker

Owner decisions (B3 / A7 scope / A15) are recorded in the phase files;
.agents/PLAN.md is being redone separately per the owner.
2026-09-10 11:37:25 -04:00
ducoterra dac4a3eec0 docs(agent): restore master plan at .agents/PLAN.md with locked decisions through phase 92
Re-lands the master plan (removed from VCS in 281f355 as .agent/PLAN.md)
at .agents/PLAN.md — the path AGENTS.md and the codebase's PLAN §…
references use. Rebuilt from the original text plus every
owner-permitted revision recorded in the completed phases: A1–A17
anchors with the full revision log (auth, SSE thinking/tool/retry
events, agent tools, history, SPA shell), new A18–A20 (docs push,
deploy caching, security headers), current data model (11 tables),
locked persona + <tools> copy, current UI/UX contracts (46rem/92rem
column, theme tab, never-stale feedback, no autoscroll), the full
per-turn log line, and the phase 01–92 roadmap with the open TODO.md
items as next up.
2026-09-10 09:28:58 -04:00
ducoterra df91c6316c phase: 92_theme_save_and_coverage
Build and Push Containers / build-and-push-app (push) Successful in 1m47s
Build and Push Containers / build-and-push-db (push) Successful in 11s
**Phase 92 final verification pass — all green.** This pass re-verified the completed tasks (all 5 task files already in `complete/`) against every completion criterion; no defects found, nothing to fix.

- Verified: 9th identity var `grid_line` end-to-end (migration `0015` at head, model/`theming.py`/schemas/API, 422 + built-in→NULL tests present); `styles.css` zero hardcoded literals outside `:root` + derived `--brand-*` vars; 9th picker in theme form; wordmark themed; `theme.js` save/reset/re-show/mount live-sync; dedicated E2E suite + phase-91 suite updated.
- `uv run pytest --cov=app --cov-report=term-missing` → **1845 passed, exit 0, TOTAL 99%** (>90%)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- `uv run pytest tests/e2e/test_theme_save_and_coverage.py -v --no-cov` → **3 passed** (save-live, reset-live, whole-site)
- `uv run pytest tests/e2e/test_admin_theme_tab.py -v --no-cov` → **5 passed**
- Criteria: (1) Save/Reset repaint open page, no nav, SPA-nav survives, pre-paint intact ✅; (2) both `rg` gates green (only `:root` + documented `#fff` Stop label; zero SVG hex attrs), grid/selection/hovers/wash/wordmark E2E-proven ✅; (3) no-op contract live-checked: row-less `/` = no tag + exact A1 CSP, grid-only row = 9-var tag in `COLOR_FIELDS` order + sha256 CSP, with-row ≡ row-less bytes ✅; (4) full suite/coverage/lint/both E2E ✅; (5) commit left to the harness per instructions.
- Deviations (previously made, probe-verified, kept): live repaint uses CSSOM `<html>` overrides because Chromium blocks `<style>` textContent mutations under the locked sha256-only CSP (tag text still mirrors the next load; `<html>` style exact-saved after Save, empty after Reset); wordmark themed via 3 `.brand-mark` CSS rules instead of inline styles (task 03's inline attrs were CSP-blocked — fixed during task 04).
- Next pending phase: none — `todo/` contains only `92_theme_save_and_coverage`.
2026-09-10 00:23:08 -04:00
ducoterra d22d260b8b phase: 91_admin_theme_tab
Build and Push Containers / build-and-push-app (push) Successful in 5m43s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification is complete — this pass needed no code changes. Final report:

**Phase 91 — Admin Theme tab: final verification pass (all 6 tasks already in `complete/`)**

- Verified pre-paint theming end-to-end: `ui_settings` store + resolver, admin `GET/PUT /api/ui-settings`, `CachingMiddleware` inline-`<style id="bor-theme">` injection before `</head>` (incl. `/shared/<token>` prefix branch, unit-pinned), CSP sha256 exemption for the inline tag, Theme tab shell + `theme.js` editor, CSS-file theming fully retired.
- No defects found; zero changes made — working tree left exactly as the task executors left it.
- Tests: `uv run pytest --cov=app` → 1841 passed, 0 failed (TOTAL coverage **99%**; theming/ui_settings/caching all 100%); `uv run pytest tests/e2e/test_admin_theme_tab.py -v --no-cov` → **5 passed** in isolation.
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings.
- Criteria: (1) unset deployment byte-identical, no `#bor-theme` anywhere — ✓ (unit no-op test + E2E reset byte-compare); `rg "BOR_THEME|themes/"` → single hit is the permitted doc-history comment in `frontend/index.html`. (2) admin-only gate + 403s for anonymous and token users — ✓ (E2E test 3). (3) saved theme inline before `</head>` on every page incl. `/shared/<token>`, computed `--brand` on first paint for admin + anonymous — ✓ (E2E test 2 + unit). (4) reset → byte-identical; 5 contrast pairs warn <4.5:1, non-blocking — ✓ (E2E tests 4–5). (5) suite green, >90% coverage, lint clean — ✓. (6) commit deferred to harness per rules.
- Notable: `.agents/PLAN.md` is absent from the repo — the phase overview's Design section was used as the binding spec; no deviation resulted.
- Next pending phase: **none** — 91 is the last phase in `todo/`.
2026-09-09 17:22:24 -04:00
ducoterra 3095c4c577 phase: 90_upload_no_scan
All verification is complete and green. Final report:

**Phase 90 (upload_no_scan) — final verification pass: all criteria verified, no fixes needed**

- Verified (not re-implemented): all 3 tasks already complete; working tree carries the phase's uncommitted changes
- Upload pipeline: `_run_upload` stops after row upsert; success = `{"message": "uploaded"}`, null/0/0 progress; `UploadOut` removed from `app/schemas.py`; gates/unpack/swap/failed states intact
- Copy: button "Upload", bare "Processing…", result line "Uploaded \<name\> — press Sync sources to import it."; hint + caption re-pointed at Sync; no "Upload &" remnants in `frontend/`
- Tests: `pytest tests/unit/test_git_sources.py tests/integration/test_git_sources_upload.py tests/unit/test_frontend_sync_upload.py` → 56 passed; E2E phase suite `tests/e2e/test_upload_no_scan.py` → 3 passed (zero docs after upload; ignore edit honored by Sync — 2 added, `notes/skipme.md` excluded; re-upload in-place, still zero docs); affected suites `test_archive_upload_sources.py` + `test_sync_upload_progress.py` + `test_source_removal_cleanup.py` → 15 passed
- Full suite: `uv run pytest --cov=app --cov-report=term-missing` → **1808 passed, 99% coverage** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors

**Completion criteria:**
1. Upload settles "ready for sync", zero docs, row + ignore editor — ✅ (E2E test 1)
2. Button "Upload", no "Upload &" copy — ✅ (`rg` empty in frontend/, app/, tests/)
3. RAG-page "Sync sources" imports upload honoring edited ignores — ✅ (isolated E2E test 2)
4. Suite green, >90% coverage, lint clean — ✅ (99%, ruff+pyright clean)
5. Atomic commit — ✅ left to harness per executor rules (no git add/commit run; tree staged-ready)

**Deviations:** `regenerate_overview` still appears once in `git_sources.py` — inside `delete_git_source` (phase-69 locked total-removal behavior, not the upload leg; upload leg is scan-free). `.agents/PLAN.md` does not exist in this repo; locked decisions A1–A4 were enforced from the phase files.

**Next pending phase:** `91_admin_theme_tab` (in `todo/`).
2026-09-09 14:08:18 -04:00
ducoterra 0c8a7b9974 chore(agent): phase 90-91 roadmap from TODO.md — upload without scan + admin theme tab 2026-09-09 08:50:14 -04:00
ducoterra 8c706259e9 phase: 89_source_ignore_paths
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed.

## Phase 89 — final verification pass: ALL GREEN

**Verified (all 6 task files present in `complete/`):**
- `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`)
- Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources`
- API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403)
- Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py`
- Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box

**Test/lint results:**
- `uv run pytest` → 1808 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up)
- Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules.

**Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green).

**Next pending phase:** none — `todo/` holds only this phase.
2026-09-09 01:45:42 -04:00
ducoterra 0495e4e7e4 chore(agent): phase 89 roadmap from TODO.md — per-source ignore paths for imports 2026-09-08 22:55:58 -04:00
ducoterra 1f0e4c6bb9 fix(ui): bind the shared header controls on explicit init, not at module import
Build and Push Containers / build-and-push-app (push) Successful in 1m53s
Build and Push Containers / build-and-push-db (push) Successful in 11s
header.js's control bindings (sign-out, the mobile hamburger, the
SINGLE New chat button) ran at module import. The Containerfile stage-1
build inlines header.js into every bundle that imports it (the shell's
app.js, token-gate.js and the router's lazy views), so the shell page
registered the #nav-toggle click handler twice, and two toggle handlers
cancel each other — one tap = open + close = the mobile menu dead in
the deployed image only. The dev tree's single ESM instance (and every
test that runs against it) never showed it; a lazy view load adding a
THIRD copy made the menu work again, which is why the failure looked
state-dependent (chat cold boot dead, /sources.html alive).

- header.js: the three bindings move into an exported
  bindSharedHeaderControls(), guarded by a marker on <body> (NOT module
  state — every bundle copy has its own function instance), so later
  bundle copies and repeated inits (the token gate's mid-page header
  re-boot) are no-ops; header.js is now side-effect-free at top level,
  which also lets esbuild tree-shake the dead copies out of the bundles
  that do not need them (the token-gate bundle no longer carries the
  binding code at all)
- app.js / login.js / shared.js / document.js: call
  bindSharedHeaderControls() once at module top — import-time parity,
  unconditional (no async boot path to miss); doc-edit.js ships no
  header controls and calls nothing
- unit: tests/unit/test_header_bindings_once_per_document.py pins the
  contract — the init export, the document-level idempotency marker,
  all three bindings inside the init, NO top-level addEventListener
  remaining, and exactly one module-top call in each header-carrying
  page script; stale import-time docstrings in the legacy header pins
  updated to the new contract

Verified: full unit + integration suite (1746 passed), the hamburger /
pinned-composer / smoke E2E stories green in isolation, ruff + pyright
clean. Containerfile-equivalent esbuild 0.25.5 rebuild probed in
Chromium: exactly ONE #nav-toggle click listener on chat cold boot,
/sources.html and login.html, and a touch tap opens the menu in all
three states (pre-fix production: two listeners on cold boot = dead,
three on sources = alive).
2026-09-08 22:31:45 -04:00
ducoterra 4d287155c0 phase: 88_mobile_chat_hamburger_boot
Build and Push Containers / build-and-push-app (push) Successful in 2m16s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All completion criteria verified green. Final state confirmed: phase stays in `todo/` per A4 (owner device gate), task files in `complete/`, no `app/` changes, no commits made (harness commits).

## Phase 88 — final verification pass report

**Verified (all 6 tasks already complete):** `touch-action: manipulation` on mobile `#nav-toggle`; `#view-chat.chat-booted` double-rAF sticky handover (CSS gate + app.js boot IIFE); `body.nav-menu-open` marker in `setNavMenu` + ≤640px `visibility: hidden` rule; router boot contract (`wasMounted && !opts.boot` + `boot: true`); new unit module (10 pins) + 4-test real-touch E2E story.

**Test / lint / coverage:**
- `uv run pytest --cov=app --cov-report=term-missing` → 1741 passed, **99%** on `app/` (>90% ✓)
- `tests/e2e/test_mobile_chat_hamburger_boot.py` 4/4, `test_mobile_hamburger_nav.py` 8/8, `test_pinned_composer.py` 4/4, `test_smoke.py` 3/3 — each in isolation, `--no-cov` ✓
- `uv run ruff check .` clean; `uv run pyright` 0 errors ✓

**Completion criteria:** (1) 360px cold-boot touch-action + sticky handover + tap-opens-menu-with-cluster-hidden (E2E 1–2) ✓; (2) boot fires no `bor:view-refresh`, re-show fires exactly one, `/sources.html` regression (E2E 3–4) ✓; (3) full gate green ✓; (4) diff limited to 4 assets + 2 new tests + phase files, 0 changes in `app/` ✓; (5) commit deferred to harness per executor rules ✓; (6) owner device re-verification **pending** (A4 — gates the archive; no Owner report recorded yet).

**Deviations (both documented in-tree):** unit pins updated in `test_frontend_router.py`/`test_hamburger_nav.py` (their exact-text pins collided with the mandated new guard/marker text — without them the suite goes red); `boot: true` count pinned at 1 not 2 (codebase has one boot call site, no `history.state` branch — verified against git HEAD).

**Next pending phase:** none in `todo/` — pipeline awaits the owner's on-device report (archive, or `?dbg=nav` instrumentation follow-up if the menu is still dead).
2026-09-08 16:02:45 -04:00
ducoterra 10fd367962 fix(chat): render typing-indicator elapsed hint as horizontal text
Build and Push Containers / build-and-push-app (push) Successful in 1m43s
Build and Push Containers / build-and-push-db (push) Successful in 15s
The phase-87 ".typing-elapsed" dot-geometry reset (specificity 0,1,0)
lost every shared declaration to the ".typing span" dot rule
(0,1,1): the hint rendered as an 8x8px bouncing dot and the "Ns" text
wrapped one character per line below the bubble (overflow-wrap:
anywhere on .bubble). Phase 87's e2e checked text values only, so the
squish shipped unseen.

- retarget the reset at ".typing span.typing-elapsed" (0,2,1) so it
  actually wins; center the dots while the hint line is taller
- unit: pin the reset's selector context (specificity regression guard)
- e2e: layout pin on the live hint — no dot animation, not an 8px box,
  horizontal single-line bounding box
- before/after verification screenshots in
  .agents/reports/87_big_read_progress/

Verified: unit 41 passed, phase-87 e2e 4 passed (isolated), ruff +
pyright clean.
2026-09-08 10:03:10 -04:00
ducoterra 5abe8871e3 fix(chat): raise pre-token guard from 120 s to 300 s
Build and Push Containers / build-and-push-app (push) Successful in 1m48s
Build and Push Containers / build-and-push-db (push) Successful in 15s
The client-side TURN_TIMEOUT_MS was the binding constraint: turns with
slow prompt processing (no first SSE frame within 120 s of visible time)
errored with the 'stuck' copy even though nginx (300 s) and
BOR_LLM_TIMEOUT (300 s) would have let them run. Raise the guard to
300 s so the upstream timeouts are reachable, and re-pin the tests:
the unit constant pins and the fake-clock E2E timeline (295 s hidden
+ 290 s after the re-arm = 585 s: past the original 300 s deadline,
short of the re-armed 595 s deadline).

Verified: tests/unit (full, 100% pass), tests/e2e/test_hidden_tab_stream.py
and tests/e2e/test_loading_feedback.py in isolation.
2026-09-08 09:37:33 -04:00
ducoterra 7cfe58fb21 phase: 87_big_read_progress
Build and Push Containers / build-and-push-app (push) Successful in 2m48s
Build and Push Containers / build-and-push-db (push) Successful in 19s
All criteria verified — no defects found, nothing to fix. Final report:

**Phase 87 — big read progress: final verification pass (all tasks already complete in `complete/`)**
- Verified implementation vs. overview: `TOOL_LINE_ELAPSED_AFTER_MS = 5_000`, single live `armToolLineClock` site, settle on thinking/retry/delta, `stopToolLineClock` in `setUiState`, visible `.typing-elapsed` (aria kept byte-identical), CSS AA pairing — all match design; `app/` byte-identical (0 changes)
- **Tests/lint (exact outcomes):**
  - `uv run pytest --cov=app --cov-report=term` → 1732 passed, coverage **99%** (>90% ✓)
  - `uv run pytest tests/e2e/test_big_read_progress.py -v --no-cov` → **4 passed** (ticking suffix, visible hint, settle, no-timer restore)
  - `test_thinking_display.py` → 5 passed · `test_agent_document_tools.py` → 4 passed · `test_smoke.py` → 3 passed (all isolated)
  - 3 pinned frontend suites + new unit pins → 62 passed · `uv run ruff check . && uv run pyright` → clean, 0 errors
- **Completion criteria:** E2E pins 1–4 ✓ · guard/state-machine byte-identical ✓ (diff is additive only) · diff scope limited to `app.js`, `styles.css`, 2 new test files, phase files; nothing in `app/` ✓
- **Notable:** no deviations; commit + `00_phase.md` move left to the harness per executor rules (task files already in `complete/`)
- **Next pending phase:** none — `todo/` contains only this phase (87 is the last)
2026-09-08 05:51:23 -04:00
ducoterra 0f6b9ff7e6 phase: 86_history_page_width
All criteria verified. Final report:

**Phase 86 final verification pass — all green, no defects found.**

- **Verified (previously implemented):** `position: relative` on shared `.table-wrap` (the one-rule A3 fix, +12 lines in `frontend/assets/styles.css`); unit pin `tests/unit/test_table_wrap_stacking.py` (3 tests); E2E story `tests/e2e/test_history_page_width.py` (5 tests: History 626→≤375, Tokens 618→≤375, SPA switch with window-sentinel, RAG regression, desktop 1280×800)
- **Tests/lint (this pass):** `uv run pytest` → 1720 passed · `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% floor) · E2E in isolation: `test_history_page_width.py` 5 passed, `test_chat_history.py` 5 passed, `test_api_tokens.py` 9 passed, `test_smoke.py` 3 passed · `uv run ruff check . && uv run pyright` → clean, 0 errors
- **Completion criteria:** ① 375px direct+SPA `scrollWidth ≤ innerWidth` on History/Tokens — PASS (E2E pins) ② in-card scroll preserved + full-width + a11y spans in DOM — PASS ③ Sources + desktop regression — PASS ④ full suite/coverage/E2E/lint gates — PASS ⑤ `git diff --stat` limited to `styles.css` (+12) + new test files + phase files, no `app/`/markup/JS — PASS ⑥ commit/move — left to harness per executor rules (working tree intact, `todo/` dir removal already reflected)
- **Notable:** red→green (CSS reverted → 626px failure) and manual live check already recorded in `.agents/reports/86_history_page_width/`; pre-existing untracked `.agents/remediation_plan.md` (Sep 7 security audit) untouched
- **Next pending phase:** `87_big_read_progress`
2026-09-08 02:38:45 -04:00
ducoterra 412a560348 phase: 85_mobile_menu_gate_overlap
All verification complete. Final report:

**Phase 85 final verification pass — all green** (tasks 01–03 already complete; independently re-verified from scratch)
- Verified fix in tree: `styles.css` `.auth-gate` z-index 500→15 + rewritten stacking comment (no `z-index: 500` left); `index.html`/`document.html` comment-only; `token-gate.js` docstring-only (logic byte-identical); `test_mobile_hamburger_nav.py` real-click conversion + new `test_anonymous_toggle_tappable_with_gate_up`; new `tests/unit/test_gate_header_stacking.py` (3 pins); `test_api_tokens.py` untouched
- `uv run pytest` → 1717 passed, 1 warning (exit 0)
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
- E2E in isolation: `test_mobile_hamburger_nav.py` **8 passed**; `test_api_tokens.py` **9 passed** (unchanged); `test_smoke.py` **3 passed**; `test_document_viewer.py` **7 passed**
- `uv run ruff check . && uv run pyright` → clean / 0 errors
- Live probe (375×812, anonymous, fresh server): on `/` and `/history.html` `elementFromPoint` at toggle → SVG `path`, never `#auth-gate`; real click opens menu (`aria-expanded=true`); exactly one visible nav link (Chat) + sign-in copy; Chat link topmost over gate; `#main` inert — criteria 1, 2, 3 confirmed directly
- Criteria: (1) real-click menu on / + /history ✅ (2) dropdown above gate ✅ (3) anonymous contents + #main inert ✅ (4) admin byte-identical (phase-46 tests green) ✅ (5) doc-viewer gate under bar (CSS pin + doc-viewer suite) ✅ (6) full gate green ✅ (7) diff scoped to the 5 code files + new unit test + phase files, nothing in `app/` ✅ (8) commit + todo→complete move left to harness per executor rules (task files already in `complete/`)
- Note: `.agents/remediation_plan.md` (untracked) is a pre-existing artifact of the earlier security audit — left untouched; a missing "N passed" line seen mid-pass was my own `-q`+addopts `-q` = `-qq` quirk, not a defect
- Next pending phase: `86_history_page_width`
2026-09-08 01:56:37 -04:00
ducoterra f4150421bb phase: 84_docs_push_error_sanitization
**Phase 84 — final verification pass: all green, no defects found**

- Verified implementation: `app/core/errors.py` (verbatim lift of sync masker), `app/api/sync.py` alias import, docs-push 502 `detail=sanitize_error(str(exc))`, all five `llm.py` error sites sanitized; new/extended test pins in place
- Tests: `uv run pytest` → **1714 passed, 0 failed**; targeted pins (new unit ×2 + integration ×1, existing 502 pin) → 13 passed; sync/git-sources regression → 67 passed
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **99%** (`app/core/errors.py` 100%, `app/rag/llm.py` 100%) — >90% met
- E2E isolation: `uv run pytest tests/e2e/test_smoke.py -v --no-cov` → **3 passed**
- Lint/types: `uv run ruff check .` → clean; `uv run pyright` → **0 errors**
- Criteria: 502 masks `*****@`/never token + row untouched ✅; LLM base-URL masked, credential-free strings byte-identical ✅; `_CREDS_RE` only in `app/core/errors.py` (working-tree grep) ✅; full gate green ✅; `git diff --stat` limited to the 4 app files + 2 modified test files + 3 phase task files (untracked: new module, new unit test, complete/ dir, reports, audit plan) ✅
- Commit/phase move left to the harness per instructions (task files already in `complete/`)
- No deviations; nothing to fix
- Next pending phase: **85_mobile_menu_gate_overlap**
2026-09-08 01:11:45 -04:00
ducoterra fa189dede7 phase: 83_chat_save_payload_limits
All verification criteria are confirmed green. Final report:

**Phase 83 — final verification pass (all tasks already complete; verified, nothing to fix)**

- Verified `app/schemas.py` carries every A1 cap (text/thinking 32 000, sources 20, suggestions 50×200, tools 50, SourceRef 120/1000/500, ToolCall 100/2000, messages ≤200) — boundary-only, `app/api/chats.py` untouched
- Verified all test pins exist and pass: 32 unit boundary tests (both edges of every cap + round-trip), 4 oversized-422 integration pins (text/201-msgs/21-sources/PUT, each with nothing-stored/row-unchanged assertions), A3 SSE done-event pin at column maxima, dedicated E2E (anon 40 000-char POST → 422 + small save → 201)
- `uv run pytest tests/unit/test_schemas.py -v` — 32 passed
- `uv run pytest tests/integration/test_chats_api.py` — exit 0; A3 pin — passed
- `uv run pytest tests/e2e/test_chat_save_payload_limits.py -v --no-cov` — 2 passed (isolation); `uv run pytest tests/e2e/test_chat_history.py -v --no-cov` — 5 passed
- `uv run pytest` — exit 0 (~1 704 tests, 0 fail/skip); `uv run pytest --cov=app` — TOTAL 99%, `app/schemas.py` 100% (>90% ✓)
- `uv run ruff check . && uv run pyright` — clean (0 errors)
- `git diff --stat` — only `app/schemas.py`, 3 test files (+`test_chat_api.py` A3 pin, sanctioned by task 02), phase files; no `app/api/chats.py`/`alembic`/`frontend`/`pyproject`/`uv.lock` diff ✓
- All completion criteria met; commit + phase move left to the harness per pipeline rules (changes stay in working tree)
- Deviation note: list caps use `max_length` instead of `max_items` — identical in pydantic 2.13 (`max_items` is a deprecated alias); both boundaries behaviorally pinned
- Next pending phase: `84_docs_push_error_sanitization`
2026-09-08 00:33:21 -04:00
ducoterra e29d68d9f0 phase: 82_security_headers
All completion criteria verified green — no defects found, nothing to fix. Final report:

**Phase 82 (security headers) — final verification pass: all green**

- Verified prior-run implementation: `app/core/security_headers.py` (pure-ASGI, header-only, exact A1 CSP), registration in `app/main.py` after `configure_caching` (outermost), unit/integration/E2E suites.
- Deviation confirmed sound: `data:`-URI favicon (blocked by locked CSP) → static `frontend/assets/favicon.svg` in 5 templates + Containerfile `cp`; SVG element byte-identical to the old data-URI (verified programmatically); serves 200 with all three headers.
- Curl check (server booted like e2e conftest, log: `/tmp/curl_security_headers_final.log`): `/`, `/api/health`, `/assets/styles.css`, `/nope` (404) → all three headers, CSP exactly `default-src 'self'; base-uri 'none'; frame-ancestors 'none'`.
- `uv run pytest tests/unit/test_security_headers.py tests/integration/test_security_headers.py -v --no-cov` → 13 passed (incl. SSE byte-identity pin).
- `uv run pytest tests/e2e/test_security_headers.py -v --no-cov` (isolated) → 2 passed (headers + zero CSP violations + painted page).
- SSE tripwire `uv run pytest tests/e2e/test_chat_rag.py -v --no-cov` → 3 passed.
- `uv run pytest --cov=app --cov-report=term-missing` → 1665 passed, app/ 99% (>90%); `uv run ruff check . && uv run pyright` → clean (0 errors).
- `git diff --stat` limited to phase-82 files + the two documented deviations (favicon set, `tests/unit/__init__.py`); no `pyproject.toml`/`uv.lock`/JS diffs.
- Commit + phase-dir move left to the harness per pipeline rules (not executed by me).

Next pending phase: `83_chat_save_payload_limits`.
2026-09-07 23:54:41 -04:00
ducoterra 42a4222949 phase: 81_login_rate_limit
All gates verified. The phase is complete — everything was already implemented by prior task runs; this pass verified each completion criterion end-to-end and found no defects.

**Phase 81 final verification report** (tasks 01–03 all present in `complete/`; no code changes needed this pass)

- Verified: `app/core/rate_limit.py` (stdlib sliding window, fail-open) + both login routes' 429 pre-check/record/reset wiring + all unit/integration pins present (11th-429, blocked-success-still-429, clean-counter reset, shared counter, autouse `clean_rate_limit` fixture documented).
- Live check (task 03): dev server + 11 rapid wrong logins → `401 ×10, 429`; 429 body carries `retry-after: 900` + generic detail; server restarted (per-process counter cleared by design).
- `uv run pytest tests/unit/test_rate_limit.py -v --no-cov` → 10 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 1652 passed, **TOTAL 99%** (>90%; rate_limit.py 100%, auth.py 100%)
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov` (isolation) → 3 passed
- `uv run ruff check . && uv run pyright` → All checks passed / 0 errors, 0 warnings
- Completion criteria: all met, except commit + phase-dir move — per harness rules I left all changes uncommitted in the working tree (harness commits atomically and moves the phase).
- Diff scope: exactly `app/core/rate_limit.py`, `app/api/auth.py`, `tests/unit/test_rate_limit.py`, `tests/integration/test_auth_api.py` + phase files; `pyproject.toml` / `uv.lock` / `frontend/` untouched.
- Deviations: none in code; commit/move deferred to harness as instructed.
- Next pending phase: `82_security_headers`.
2026-09-07 23:08:46 -04:00
ducoterra 894637108c add configurable llm timeout
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-07 22:01:53 -04:00
ducoterra e2bed52751 chore(agent): phase roadmap from TODO.md, 3 phases (85-87)
TODO.md L3-L5 converted to executable phases (protocol B - append):
- 85_mobile_menu_gate_overlap (L3): the phase-79 token gate (z 500, fixed
  full-viewport) sits above the sticky header (z 20), so an
  unauthenticated visitor's tap on the mobile hamburger hits the gate
  overlay and the menu is unreachable until login. Reproduced:
  elementFromPoint at the toggle resolves to #auth-gate on every shell
  view; real clicks are intercepted. Fix: gate at z 15 (below the
  header + its mobile dropdown), #main stays inert-locked; the
  phase-46 E2E's programmatic-click workaround becomes a real click +
  a new TODO-regression pin.
- 86_history_page_width (L4): at 375px the History page panned ~250px
  into a blank region (document scrollWidth 626) although the table
  scrolled fine inside its card. Root cause: the .visually-hidden
  Actions header span is position:absolute with no positioned ancestor,
  so its 1px box (at the 640px table's right edge) leaks into the
  document's scrollable overflow. Fix: position:relative on the shared
  .table-wrap card (the identical Tokens-view defect, measured 618, is
  fixed by the same rule; RAG is clean and pinned). New E2E story.
- 87_big_read_progress (L5): after a tool read the UI sat on a static
  'Reading <path>' line while the model prefilled the big context -
  the turn looked frozen. Frontend-only: a ticking '(Ns)' suffix on
  the latest tool line after 5s of frame silence (settle on the next
  frame, live-only - restored lines stay timer-free) + the existing
  10s aria-only typing clock promoted to a visible 'Ns' hint. New
  source-level unit pins + slow-proxy E2E story (deterministic >=6s
  gaps via the mock tool flow).

TODO.md cleared (items now live in .agents/phases/todo/). The pre-existing
uncommitted 81-84 phases + remediation_plan.md are a separate workstream
and are NOT part of this commit.
2026-09-07 21:12:03 -04:00
ducoterra 7baca3d289 update README
Build and Push Containers / build-and-push-app (push) Successful in 17s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-07 14:26:26 -04:00
ducoterra 2174caff33 fix button widths on mobile
Build and Push Containers / build-and-push-app (push) Successful in 1m39s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-07 14:18:45 -04:00
ducoterra f664aafeed fix(ui): give the mobile refresh controls labels and full width
At <=640px the RAG "Sync sources" pill and the History "Refresh"
pill squeezed down to tiny icon-only buttons — hard to discover and
tap on a phone. They are now full-width labeled pills:

- the RAG page-head row wraps so the Sync pill drops below the
  "Knowledge base" title at full width; the History page-head
  already wrapped the pill below its title block
- the Sync label's min(16rem, 40vw) cap lifts on mobile (min-width: 0
  engages the ellipsis) so the live-file text truncates against the
  full width instead of the 40vw cap
- the Refresh glyph joins its visible label (it stays hidden on
  desktop, where the label carries the pill)

This matches the established mobile full-width pill language (New
chat / Share / stale-ban Regenerate). The three unit tests that
pinned the old icon-only CSS are updated to pin the new behavior.
2026-09-07 14:16:41 -04:00
ducoterra 7fce6572d0 feat: phases 77–80 — navbar view refresh, static background, API tokens, history suggestion chips
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Single consolidated commit for four completed, validated phases (77, 78,
79, 80). The pipeline run left all work uncommitted because the harness
commits only with PHASE_COMMIT=1 while child executors are forbidden from
committing; the phases themselves all passed validation and moved to
.agents/phases/complete/.

Phase 77 — navbar view refresh
- router.js dispatches bor:view-refresh on re-show / active re-click /
  popstate (gated on wasMounted; first show and boot exempt)
- History / RAG / Sources / Tuning re-fetch on refresh (admin branch);
  Chat deliberately excluded (stream survival)
- History "Refresh" button (admin-only, in-flight disable + status line)
- New story suite tests/e2e/test_navbar_refresh.py (7 tests)

Phase 78 — static background
- Removed the animated glow layers; static 44px grid over the flat --bg
  canvas; default and reduced-motion renders byte-identical
- Updated background/theme E2E suites; removed bg-glow test pins

Phase 79 — API tokens
- api_tokens model + migration 0012; hash-only token service
- Admin tokens API + Tokens admin view; POST /api/token-auth;
  live-revoking require_user on chat / suggestions / document content
- Frontend token gate with localStorage cache; anonymous E2E suites
  migrated to token login
- New story suite tests/e2e/test_api_tokens.py (9 tests)

Phase 80 — history suggestion chips
- last_questions() endpoint with SEED fallback; startNewChat() refetch
- Seed-semantics docs (config.py, .env.example, README)
- Integration state matrix + E2E suite rewritten to the 4 chip states

Also included: phase-76 report artifacts and the repo restore-test-db
skill (previously untracked), scripts/* ruff fixes from phase 77.

Final gate state (phase 80 final pass, covers everything above):
- uv run pytest --cov=app → 1637 passed, 0 failed, app/ coverage 99%
- uv run ruff check . && uv run pyright → clean, 0 errors
- Per-phase story E2E suites green in isolation
2026-09-07 12:39:01 -04:00
ducoterra 495d042a98 chore(agent): phase roadmap from TODO.md — 4 phases (77–80)
Build and Push Containers / build-and-push-app (push) Successful in 1m54s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Protocol B append: navbar refresh + History refresh button (77, TODO L3),
static background — glow layers removed (78, TODO L4), admin-issued API
tokens with the in-app gate + browser caching, only shared chats stay
anonymous (79, TODO L5), onboarding chips as the last 3 questions asked
with the env seed only before the first (80, TODO L6).

TODO.md cleared — its items now live in .agents/phases/todo/.
Owner-confirmed assumptions recorded in each phase overview
(A1–A7, chat 2026-09-06).
2026-09-06 23:54:11 -04:00
ducoterra b78afc08f2 docs(bench): add chat model results to CSV benchmark
Added 6 rows for chat model results (lite + turbo, fixture + derived) to benchmarks/model_benchmarks.csv.
2026-09-06 21:59:06 -04:00
ducoterra f221b40fce feat(agent): add CSV benchmark recorder + summary/embedding test scripts and skills
New files:
- scripts/model_benchmark.py — shared CSV recorder for all model tests
- scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination)
- scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed)
- .agents/skills/test-summary-model/SKILL.md — skill for testing summary models
- .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models
- benchmarks/README.md — schema documentation

Updated:
- .agents/skills/test-chat-model/SKILL.md — now also records to CSV

All three scripts write to benchmarks/model_benchmarks.csv with one row
per run per check. The CSV accumulates results across runs for comparison.
2026-09-06 21:58:35 -04:00
ducoterra 70ba8710f3 docs(agent): record the turbo sanity check on the controlled fixture battery
2026-09-06 fixture runs: contract 100 %, executed 100 %, wall ~113 s (2 runs). Derived battery: FAIL only on usage floor (5/10 tool-turns) — answers seeded questions from context, which is ideal grounded behavior. Wall time ~2.8× lite (113 s vs 40 s). Model is clean.
2026-09-06 21:49:57 -04:00
ducoterra bf64c0d7e4 docs(agent): record the lite comparison on the controlled fixture battery
2026-09-06 fixture runs: contract 92–93 %, executed 64–75 %, wall ~40.5 s (2 runs). Derived battery: FAIL, 36 % executed (38.3 s). Same pattern — copy-invariant re-read habit blocks the ≥90 % executed bar under current ALREADY_IN_CONTEXT refusal semantics. Model is working correctly; the bottleneck is the app's dedupe refusal, not the model.
2026-09-06 21:42:14 -04:00
ducoterra ffa919b8bf fix(chat): keep in-flight answers alive across in-app view switches
Root cause (owner repro, verified in a real browser 2026-09-06): the
five navbar views (Chat, RAG, Sources, Tuning, History) were separate
HTML documents, so a navbar click was a REAL cross-document navigation
— the chat page unloaded, the in-flight SSE fetch was aborted, and the
phase-48 teardown (app/api/chat.py `finally`, "chat: turn cancelled")
stopped the model. Observed: send question -> click RAG mid-stream ->
click Chat -> the answer never finished: no `query_log` row, and on
return a dangling question with no brain record (the pre-token pagehide
partial persist skips because `acc` is empty).

Phase-48 LOCKED-DECISION REFINEMENT (owner-confirmed 2026-09-06,
flagged per AGENTS.md rule 3, not silently deviated): "real navigation
cancels the fetch" now means LEAVING THE APP — tab close,
external/other-document navigation, the Stop button. In-app navbar
switches are client-side view switches and no longer cancel.

Fix — Option A (SPA shell), chosen over B (Service Worker owns the
stream) and C (server-side turn registry + resume):
- frontend/index.html is the shell: ONE `<main id="main">` holds the
  five `<section class="view">` blocks; hidden views carry BOTH
  `hidden` and `inert` (WCAG — no focus/keyboard traversal). The
  shared header, the single `doc-modal-*` skeleton, and the
  `#app-version` footer each exist exactly once; the per-view copies
  from the four folded pages are dropped.
- New frontend/assets/router.js (vanilla module — no framework, no
  bundler, No-CDN rule intact): lazy-imports a view module on FIRST
  show only (mount-once, hide-forever — the chat view's in-flight SSE
  reader persists across switches; that persistence IS the fix);
  intercepts same-shell navbar links with preventDefault +
  history.pushState (never a document load); handles popstate; single
  writer of `.nav-link` active state (is-active + aria-current),
  document.title, and the per-view meta description (values carried
  over from the old pages' heads, brand-resolved at write time).
- Each folded page's JS becomes `export async function mount(root)` —
  root-scoped queries; `initSharedHeader()` dropped (the header boots
  once in the shell via the chat module; the admin flag comes from the
  same cached `fetchIsAdmin()` promise — zero extra requests).
- app/main.py: a small list-driven route factory serves the shell for
  /tuning.html, /sources.html, /git-sources.html, /history.html —
  registered AFTER the API routers and BEFORE the static catch-all
  (routes-first). The phase-33 caching middleware applies no-cache +
  `?v=` rewriting unchanged; app/core/caching.py needed NO change
  (the view paths did not change — pinned by the integration tests).
- The four old view .html files are DELETED (one source of truth);
  deep links to the old URLs keep working (the router picks the view
  from the pathname); `/?chat=<id>` is unaffected; the Containerfile
  bundles router.js (inlining the lazy view modules) and drops the
  folded page files.
- app/schemas.py: HistoryTurn.text cap 4000 -> 32000 — the shell
  keeps long saved answers in the chat, and the old cap (stricter than
  the 24_000-char total history budget) 422-rejected any second turn
  in such a chat (found by the phase-42 E2E suite on the shell).

Boundaries: login.html, shared.html, doc-edit.html, document.html
REMAIN separate documents (flow pages, not navbar tabs); a mid-stream
navigation to doc-edit/document.html still cancels per phase 48
(follow-up candidate, out of scope). The SSE API is unchanged. Real
departures still cancel the turn — phase 48 intact (pinned by
tests/e2e/test_stop_generation.py, unchanged, and by the new suite's
real-departure control).

Tests:
- Phase-20 suite REWRITTEN to the new semantics
  (tests/e2e/test_sources_midstream_bug.py): a navbar switch no longer
  cancels — the stream survives the switch and the FULL answer
  settles; the pagehide partial persist REMAINS for real departures
  (the partial's exact shape — first streamed chunk prefix, no done
  metadata — is still pinned there).
- NEW story suite tests/e2e/test_nav_switch_keeps_stream.py (mock
  LLM): the owner repro (send -> RAG mid-stream -> Chat: window
  sentinel survives = same document, FULL answer, exactly one brain
  turn in bor.chat.v1, exactly one settled query_log row, auto-saved
  row matches) + the same mid-stream switch against the other three
  views + the real-departure-still-cancels control + the no-switch
  baseline.
- tests/unit/test_frontend_router.py: source-level pins of the router
  invariants (click interceptor targets ONLY same-shell view paths,
  pushState-only switches, mount-once guard, hidden+inert pair,
  single-writer active state/title); shell-route integration tests
  (each folded path serves the shell with no-cache + `?v=` body; a
  non-view path still 404s); the file-reading unit pins re-pointed at
  the shell (the four view files are gone — the shell is the source
  of truth).

Verification (this commit): full suite green — 1565 unit+integration
tests, app/ coverage 99% (>90% floor); ruff + pyright clean; the
phase's E2E suites green in isolation (house protocol, AGENTS.md rule
9). Owner repro verified in a real browser against the real LLM
(dev server :8010, headful Chromium): "tell me about everquest" ->
RAG mid-stream -> Chat — the answer completed with one brain bubble
and no error banner, `query_log` gained exactly one settled row
(deflected=True: the dev KB holds no EverQuest docs — the settle, not
the topic, is the proof), zero "chat: turn cancelled" lines for that
turn; the control (real navigation to /shared.html mid-stream) still
cancelled (no settled row, the cancel line logged, the partial
persisted on return). Screenshots: .agents/screenshots/76_manual_*.

Phase 76 (76_spa_nav_shell) complete — moved to
.agents/phases/complete/.
2026-09-06 06:31:31 -04:00
ducoterra 7e567bddf3 prepping for SPA conversion
Build and Push Containers / build-and-push-app (push) Successful in 14s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-05 21:50:56 -04:00
ducoterra 0e4651c779 feat(docs): save the whole chat session as a doc
Build and Push Containers / build-and-push-app (push) Successful in 1m46s
Build and Push Containers / build-and-push-db (push) Successful in 12s
Phase 75 (TODO.md L4): "Save as doc" now drafts a document from the
ENTIRE chat session — every question and answer up to the click, in
order — instead of only the clicked bubble's answer; the existing
doc-edit screen's free-form body editing is how the user edits out
anything they don't want to keep from previous replies (no new UI
surface).

Task 01 (frontend):
- app.js buildSessionTranscript(): walks the bor.chat.v1 conversation
  record in order — a numbered section per user turn ("## N.
  <question, raw>" + blank line + the raw answer text; more answers
  join under the same heading), sections blank-line separated, all
  trailing whitespace collapsed to one final newline. Only the raw
  persisted text travels (m.who + m.text — no thinking blocks, no
  source chips, no tune metadata); a brain record before the first
  user record is skipped; a heading-only section marks a user turn
  whose answer never landed (A6, owner-confirmed 2026-09-08).
- saveAsDoc(btn): the draft body is buildSessionTranscript(); the
  dead single-bubble markdown parameter is dropped (the button's
  appendSaveAsDocButton signature is unchanged — one button per
  bubble). Title/path/double-click guard/hand-off are unchanged
  (defaultDocTitle: the last question, whitespace-collapsed,
  <=120 chars; docs/<slug>.md).
- Unit: the app.js source pins move to the transcript shape (whole
  session, no thinking, no dead parameter).

Task 02 (E2E):
- tests/e2e/test_save_doc_session.py (bare-repo fixture, the
  phase-59 convention — git as source of truth): three DISTINCT
  on-topic turns in one session (turn 1 carries the phase-17
  "think out loud" trigger so its record has a thinking block the
  transcript must exclude) -> save on the LAST bubble -> the
  prefilled body is ## 1./## 2./## 3. in order, byte-exact against
  the deterministic mock, thinking-free -> edit the whole
  section-2 block out of the body -> push -> git show
  bor-docs:<path> equals the EDITED body byte-for-byte (section 2's
  question and answer provably absent; sections 1 and 3 byte-exact;
  the UI's sha prefix is git rev-parse bor-docs). Second test:
  the button on the FIRST bubble still drafts the whole session
  (A6 — the transcript is the session at click time, title stays
  the last question); canceling leaves the branch tip untouched.
- tests/e2e/test_response_to_docs.py: the phase-59 single-turn body
  expectation moves to the transcript shape ("## 1. <question>" +
  the answer's markdown) — the rest of the suite unchanged.

Also lands the phase-74 file moves (00_phase.md /
03_mock_marker_e2e.md -> complete/) and the phase reports — the
house convention of committing .agents/ with the phase.
2026-09-05 16:59:31 -04:00
ducoterra 055c0b5d85 feat(rag): pass chat history with prior thinking to the LLM
Build and Push Containers / build-and-push-app (push) Successful in 1m39s
Build and Push Containers / build-and-push-db (push) Successful in 11s
Phase 74 (TODO.md L4): a follow-up question now reaches the model WITH
the conversation so far — every prior user/brain turn and the prior
thinking blocks on brain turns (preserve-thinking) — while
POST /api/chat stays stateless (A10): the client provides the history
in the request body and the server stores nothing new.

Server (task 01):
- ChatRequest.history: optional list[HistoryTurn] (who: user|brain,
  text, optional thinking) — absent/empty keeps the request
  byte-identical to pre-phase-74 (the two-message [system, user]
  request; the kill-switch semantics are pinned in the integration
  suite).
- app.rag.prompts.history_to_messages: pure mapper — walks the turns
  newest-first against the settings budgets (history_max_turns=40 /
  history_max_chars=24000, BOR_HISTORY_MAX_TURNS /
  BOR_HISTORY_MAX_CHARS); a capped turn is dropped WHOLE (never cut
  mid-answer); the kept window is returned oldest-first; brain turns
  carry their thinking as reasoning_content (A4) only when
  non-empty.
- Both branches feed it: the deflected path splices it between the
  system prompt and the current user message (the phase-71 recovery
  still rebuilds from messages[1:]), the grounded agent receives
  run_agent(..., history=hist); llm.py's message params widen to
  list[dict[str, Any]] (string-only messages stay byte-identical on
  the wire — the SDK passes message dicts through verbatim).
- The per-turn log line (PLAN §9) gains history_msgs=N after
  kb_chars=N.
- Pins: tests/unit/test_history.py (mapper: mapping, reasoning
  gating, both budgets, drop-whole, ordering, empty default),
  tests/unit/test_config.py (the two settings + env overrides),
  tests/unit/test_agent.py (the history splice + the default),
  tests/integration/test_chat_api.py (deflected AND grounded forward
  the history incl. reasoning_content, no-history byte-identity, 422
  pins, the log field).

Client (task 02):
- runTurn — the single funnel for fresh send / phase-49 retry /
  phase-53 stale-regen — sends history = the conversation record
  minus the current question, with thinking only on brain records
  that streamed one (undefined drops the key from the JSON, the
  record's convention); the question is never duplicated into the
  history.

Wire proof (task 03):
- The mock's echo my history marker (HISTORY_TRIGGER) answers with
  the deterministic history echo — history: N prior messages; last
  answer tail: <last 24 chars>; thinking: yes|no — checked BEFORE
  the DEFLECT_MODE branch (like TABLE_TRIGGER), so it fires on both
  turn branches whatever the gate says; the module docstring records
  the user/assistant-only history invariant that keeps every
  existing (tool-result-classified) marker flow unaffected.
- tests/e2e/test_llm_history.py (isolated): a grounded follow-up and
  a deflected follow-up both receive history: 2 prior messages +
  thinking: yes + the byte-exact tail of turn 1's answer (derived
  from the persisted bor.chat.v1 record — the same array the client
  maps into the body); a cold start receives history: 0 prior
  messages / last answer tail: none / thinking: no.
- Regressions green in isolation: chat_rag, chat_history (phase 50),
  agent_document_tools, harness_aligned_tools, stop_generation,
  retry_answer, response_to_docs.
2026-09-05 16:04:40 -04:00
ducoterra a16130c71d fix(chat): keep generating while the tab is hidden
Root cause (task 01): none of C1-C3 - in Chromium 151 (real mode) a
merely-hidden tab neither stops the stream (frames arrive at full rate;
turn completes) nor fires pagehide on tab switch; C1's double-record
path was proven latent via a synthetic pagehide (trigger is
browser-dependent, e.g. Safari) and C2 (the 120s pre-token guard) was
confirmed to fire while hidden.

- C1: the pagehide partial-persist is correlated with the turn's settle
  (leavePartialIndex) - the done/stop settle REPLACES it in place
  (identity-guarded rememberBrainTurn in-place mode), so bor.chat.v1
  and the auto-saved saved_chats row keep exactly ONE brain turn per
  question; a real navigation never runs a settle, so the leave-save
  is unchanged.
- C2: the visibility re-arm gives the still-armed pre-token guard a
  fresh TURN_TIMEOUT_MS when the tab returns to visible - hidden time
  no longer counts toward the 120s guard.
- Phase-48 teardown contract untouched: Stop / tab close / real
  navigation still cancel the fetch and stop the model.
- Unit pins: tests/unit/test_frontend_hidden_tab.py (the app.js
  mechanisms without a browser).
- E2E pins: tests/e2e/test_hidden_tab_stream.py - synthetic pagehide
  mid-stream completes exactly once with one brain turn (localStorage
  + auto-saved row), reload restores one bubble, no-event baseline,
  and the fake-clock pre-token guard re-arm (discriminating: fails
  with the re-arm disabled).
2026-09-05 14:34:33 -04:00
ducoterra 45c3fa2863 chore(agent): phase roadmap from TODO.md (phases 73-75), clear the file
Convert the two unchecked TODO items into executable phases (Protocol B,
appended after the 72 completed phases):

- 73_hidden_tab_stream (TODO L3): a merely-hidden browser tab must never
  stop a generating answer; repro/root-cause decision tree + the pagehide
  partial-correlation fix + the hidden-tab E2E pin.
- 74_llm_chat_history (TODO L4, history): client-provided history in
  POST /api/chat (stateless, A10) mapped through both the deflected and
  grounded agent paths, prior thinking blocks preserved via
  reasoning_content, capped oldest-first; mock echo marker + E2E.
- 75_save_doc_full_session (TODO L4, save-as-doc): the Save-as-doc draft
  body becomes the full session transcript; edit-out happens in the
  existing doc-edit body; multi-turn git-verified E2E.

Owner-confirmed assumptions A1-A7 are recorded as ASSUMPTION lines in the
task files. TODO.md is cleared (items now live in .agents/phases/todo/).
2026-09-05 12:14:21 -04:00
ducoterra dbf2af26c6 refactor(agents): migrate .agent/ planning tree to .agents/
Standardize on the .agents/ directory (shared with project skills):
phases/, user_stories/, reports/, screenshots/, validate.sh, and
phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves
history; runtime artifacts move alongside).

Updates every reference in AGENTS.md, README.md, .gitignore, app
docstrings, and test story headers. Historical KB content in data/
and the runtime pipeline.log transcript are left untouched.
2026-09-05 10:57:07 -04:00
ducoterra 766702c750 finally getting accurate answers
Build and Push Containers / build-and-push-app (push) Successful in 1m46s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-05 10:26:39 -04:00
ducoterra bb2803bebd feat(skills): add test-chat-model skill — add a chat model and run the controlled tool-calling battery
Codifies the 2026-09-05 turbo comparison workflow as a project skill under
.agents/skills/: switch BOR_LLM_CHAT_MODEL in .env, run the fixture gate
(twice, for variance) + the locked derived gate with per-turn wall timing,
interpret the two metrics against the reference model rates (re-read habit:
lite ~100%, turbo ~12%; usage-floor MISS as test artifact; caps as real
regression), record the verdicts byte-exact in TOOL_CALLING_TESTING.md, and
commit the doc. Rules baked in: never touch the battery/thresholds/fixtures,
never edit app code, never commit .env.
2026-09-05 00:13:21 -04:00
ducoterra ce896ab8ab docs(agent): record the turbo comparison on the controlled fixture battery
turbo (2026-09-05, same fixture KB): fixture gate PASS 100%/100% on both
metrics, two runs (wall 105-135s vs lite 43-55s); the redundant re-read
of seeded documents that capped lite's executed ratio at 58-73% is
model-specific (turbo re-read rate ~12% vs ~100% in-sample), corroborating
section 7's framing. Locked derived battery: turbo fails only the >=6/10
tool-turn usage floor (it answers seeded read-target questions from
context instead of making the refusable read call) - accuracy on all
emitted calls still 100%/100%.
2026-09-05 00:09:55 -04:00
ducoterra 988ff78526 fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Build and Push Containers / build-and-push-db (push) Successful in 14s
Phase 72 (72_teaching_refusals) — completed under the 2026-09-04 controlled
methodology (owner directive: stop clearing/re-importing the homelab KB per
iteration; measure tool-calling accuracy on a controlled fixture KB, target
>90%).

Real-model gate verdicts (live, configured chat model 'lite', fixture KB):
- Controlled fixture battery (the new methodology's pass condition —
  contract accuracy >= 90%): PASS, 4 consecutive runs:
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) 2026-09-04 (wall 50.6s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) 2026-09-04 (wall 46.8s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/15 executed (60%) contract 14/15 (93%) 2026-09-04 (wall 54.8s)
- Locked derived battery (phase-72 task 05, executed >= 90% bar, run
  unchanged on the same fixture KB):
  gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04 (wall 47.7s)
  The teaching works — every bare-path trap self-corrects in exactly one
  round, zero cap hits, zero repeat loops, 10/10 answered. The locked
  executed bar is blocked by ALREADY_IN_CONTEXT dedupe refusals on the
  corrected re-reads (the trap question seeds its target, so the correct
  combined-form read is refused for redundancy) — a copy-invariant model
  behavior (five copy variants, 0/15 re-reads flipped, 2026-09-03 -> 04)
  and an app-semantics decision for the owner (TOOL_CALLING_TESTING.md
  sections 5 and 7), not a copy lever.

Copy changes this phase owns (unit pins updated to follow):
- app/rag/agent.py: ls teaching refusals (path-like scope -> document-path
  line; unknown source -> no-source line with the source-name
  parenthetical), read/grep 'did you mean source/path?' teaching
  (find_path_candidates: exact or suffix path match, catalog order, cap 3),
  ALREADY_IN_CONTEXT naming the correct action (answer from the text
  already in the prompt), read tool description front-loaded with the
  do-not-read rule (the 2026-09-04 controlled telemetry: the re-read is
  the only remaining refusal class; contract accuracy 92-100% across runs)
- app/rag/prompts.py: TOOLS_SECTION states the document-identity contract
  up front (ls path = source name; read/grep = combined source/path
  including the source name; do-not-read for <documents> documents placed
  next to the read teaching; one-call-per-reply and never-repeat rules)
- tests: refusal pins (unit + integration), new dedicated E2E suite
  tests/e2e/test_tool_path_teaching.py (mock misuse flow, green in
  isolation), regression suites green in isolation (harness_aligned_tools,
  agent_document_tools, agent_unlimited_tools, search_tool, chat_rag).

Gates: uv run pytest green (1501); coverage TOTAL 99% (>90%); ruff +
pyright clean. Carries the still-uncommitted phase-71 todo/ -> complete/
move and both phases' .agent/reports/ (AGENTS.md 8).
2026-09-04 13:11:07 -04:00
ducoterra 7909bdb8da test(agent): controlled fixture KB + one-command fast loop for tool-calling iterations
The phase-72 iteration loop cleared the database, git-cloned the homelab repo, re-imported 38-51 documents and re-embedded per run — many minutes per iteration against a different KB every time (owner directive 2026-09-04: stop importing the homelab repo on every test run). Replace it with:

- tests/fixtures/agent_kb/: 8 hand-written markdown docs (sources 'deployments'/'homelab') whose specifics (rack7, 10.77.42.0/24, VLAN 130, rbm-8842, 17 2 * * *, obsidian-bor:2026.7.14, 18765, 18443, ...) no model can guess; read targets carry non-topical filenames so their questions do not lexically seed them (the read must actually happen)
- tests/fixtures/test_kb.dump.sql: data-only snapshot (TRUNCATE + INSERTs incl. embeddings, self-contained git_sources rows, static KB overview) — verified by round-trip checksum at build time
- scripts/load_test_kb.py: one-off rebuild (real pipeline + embeddings, ~2s) that also prints the per-question retrieval report (all 10 battery questions must be grounded)
- scripts/restore_test_kb.py: sub-second one-transaction restore (no git clone, no re-embedding)
- scripts/agent_realmodel_check.py: the gate gains --restore / --mode fixture (curated 10-question battery with one unambiguously correct tool behavior per question) / --turns N (12s micro-loop) / --concurrency / per-turn + total wall timing, and a second accuracy metric (contract accuracy: well-formed calls targeting resolvable entities) alongside the phase-72 locked executed ratio — the re-read of a seeded doc is a copy-invariant model behavior (5 variants, 0/15 flipped) that the dedupe refusal counts as a failure
- TOOL_CALLING_TESTING.md: the human-readable methodology (fast loop, design rules, metrics, copy levers + tried-and-reverted table, current standing, open design question)

Measured: restore 0.03s; micro-loop ~12s; full loop ~43-55s; concurrency 2/3 gives no gain (endpoint serializes).
2026-09-04 13:10:15 -04:00
ducoterra 575d6c88d0 feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery 2026-09-03 13:39:15 -04:00
ducoterra 801639efcc feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?) 2026-09-03 11:17:47 -04:00
ducoterra 16f1cfbcaf add additional phases and launch.json
Build and Push Containers / build-and-push-app (push) Successful in 2m6s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-02 18:59:09 -04:00
ducoterra 0bf96f22e1 fix(agent): make read_document robust to combined source/path arguments
The model treated the combined 'source/path' string (as printed in
search result lines, read-result headers and refusals) as the
document's identity and passed it as 'source' — e.g.
source='homelab/active/container_caddy/caddy.md' instead of
source='homelab', path='active/container_caddy/caddy.md'.

- Rewrite the read_document description with the split rule (source =
  before the FIRST '/', path = after it) and a worked example; share
  the source/path parameter descriptions between read_document and
  search_documents; map search result lines back onto the split.
- New _resolve_document: on a lookup miss with a '/' in source, retry
  at the first slash (source names are directory basenames and can
  never contain '/'), plus a continuation candidate for a split at a
  later slash; a self-corrected combined form for an already-in-context
  document is still rejected as ALREADY_IN_CONTEXT.
- A slash-carrying source that matches nothing gets an educational
  refusal naming the corrected arguments instead of the generic line
  that repeated the combined form.

Verified live against aipi (lite) + the imported homelab KB: A/B on
the exact failure scenario (5 runs each, right after a
combined-source search result) — old descriptions 5/5 combined, new
descriptions 5/5 clean; two live UI turns (Playwright) produced only
clean split arguments, including a multi-hop read of
install_caddy_deskwork.yaml that landed in done.sources. Full suite:
1376 passed, app coverage 99% (agent.py 100%), ruff + pyright clean,
agent/search E2E green in isolation.
2026-09-02 17:42:57 -04:00
ducoterra 137d5fa1a5 feat(sources): removing a source deletes its files and index entries behind a confirmation modal
Build and Push Containers / build-and-push-app (push) Successful in 1m29s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-02 15:55:33 -04:00
ducoterra 265e736b3d add phase 68 complete 2026-09-02 13:33:45 -04:00
ducoterra 189aa92182 fix tool descriptions to prevent incorrect tool calls
Build and Push Containers / build-and-push-app (push) Successful in 1m32s
Build and Push Containers / build-and-push-db (push) Successful in 10s
2026-09-02 13:32:58 -04:00
ducoterra 8cf3a827ee feat(agent): search_documents tool — the model can grep the indexed documents for an exact string
Build and Push Containers / build-and-push-db (push) Canceled after 0s
Build and Push Containers / build-and-push-app (push) Canceled after 1m11s
2026-09-02 12:04:34 -04:00
ducoterra 88293ed02f feat(rag): retry a failed LLM request before the first token lands — BOR_LLM_RETRIES/BOR_LLM_RETRY_DELAY with a live 'retrying' status 2026-09-02 10:52:38 -04:00
ducoterra f04ddbe1f8 fix(web): history tab copy — every chat saves automatically, there is no Save button
Build and Push Containers / build-and-push-app (push) Successful in 1m35s
Build and Push Containers / build-and-push-db (push) Successful in 10s
meta description -> locked (A3) auto-save string (history.html L6).
page-sub -> locked (A3) string, the <strong>Save</strong> emphasis retired with the button (L106-109).
empty row -> locked (A3) string; colspan=6, hidden, and row id untouched (L163).
h1, the anonymous gate section, and history.js are byte-identical — state language verified accurate.
2026-09-02 01:29:42 -04:00
ducoterra 8a1f99cb38 feat(web): move the chat action cluster to the pinned bottom and align the button sets
- task 01: relocate the .chat-actions row (New chat + Share, comments byte-identical with a Phase 65 note) from the top of the column to the bottom of .chat-shell, directly above the composer
- task 02 (owner-locked A1): wrap the row + #composer in ONE sticky .chat-bottom unit (position: sticky; bottom: env(safe-area-inset-bottom, 0), no z-index) — the pills stay at the bottom of the screen at every scroll position and settle into flow above the footer
- task 03 (owner-locked A2): right-align the bottom row to the column's right edge (justify-content: flex-end), mirroring the right-aligned Save-as-doc corner; the five action pills share one 44px / 999px-pill geometry
- task 04: dedicated Playwright suite tests/e2e/test_bottom_chat_actions.py (resting geometry, the A1 pin across the sticky range, A2 alignment + DOM order + mobile stack + 360px overflow bound + 44px touch targets, New chat / Share click-through) — green in isolation
- task 05: regression matrix green in isolation (pinned_composer 4, save_share_ux 5, chat_persistence 4, share_chat 4, chat_history 5, smoke 3); full gate green — unit + integration pass, app/ coverage 99% (>90%), ruff + pyright clean
2026-09-02 01:08:18 -04:00
ducoterra 4677d86f49 feat(sources): real-time file progress for sync and upload — background upload with success toast 2026-09-01 23:51:43 -04:00
ducoterra cddc84c7db chore(agent): phase roadmap from TODO.md — 67_llm_retry + 68_search_tool 2026-09-01 20:15:08 -04:00
ducoterra 66419bf652 chore(agent): phase roadmap from TODO.md, 2 phases
Phase 65 (TODO.md L3): move the New chat + Share cluster to the pinned
bottom of the chat column (sticky .chat-bottom unit with the composer)
and right-align the row to the Save-as-doc action corner.
Phase 66 (TODO.md L4): History tab copy — every chat saves
automatically; retire the Save-button references.
2026-09-01 19:53:56 -04:00
ducoterra a6a1bf7143 chore(agent): phase roadmap from TODO.md, 1 phase (64_sync_upload_progress) 2026-09-01 19:27:31 -04:00
ducoterra 15a16a8fe0 fix(agent): unambiguous document listing format for LLM parsing
Build and Push Containers / build-and-push-app (push) Successful in 1m34s
Build and Push Containers / build-and-push-db (push) Successful in 10s
2026-09-01 12:44:53 -04:00
ducoterra c738105932 feat(web): customizable placeholder, footer text, and color theme via BOR_* env vars
BOR_INPUT_PLACEHOLDER / BOR_FOOTER_TEXT / BOR_THEME (+ the indigo.css example theme); authoring guide: frontend/assets/themes/README.md, docs: README 'Customizing the look'.
2026-09-01 12:04:06 -04:00
ducoterra baefcde668 fix(web): retire the stale homelab-era copy — neutral, accurate defaults on every page
Fixed: index.html meta description, empty-state sub and composer
placeholder (A1); app/config.py default suggestion chips → the four
neutral A2 defaults (BOR_SUGGESTIONS override unchanged); sources.html
KB page-sub → the current source model (git repos + local dirs +
uploaded archives, Sync pulls/imports); git-sources.html example URL
→ your-repo.git (A3); all 9 footers → neutral default in
span.footer-text (the phase-62 hook); E2E/unit conftests force the
code defaults so a local .env cannot leak corpus copy into tests;
new unit text pins + dedicated E2E suite.

Task 02 verification read-through — no change needed:
- sources.html sync result/error copy (matches the real sync behavior)
- tuning.html page-sub (accurate as written)
- history.html page-sub (accurate as written)
- doc-edit.html page-sub (accurate as written)
- git-sources.html page-sub (accurate as written)
- #sources-gate anonymous copy (accurate as written)
2026-09-01 10:54:50 -04:00
ducoterra 4971e2859d chore(agent): track .agent/ planning tree in git
Build and Push Containers / build-and-push-app (push) Successful in 12s
Build and Push Containers / build-and-push-db (push) Successful in 10s
Remove the blanket .agent/ gitignore so the phase roadmap, user
stories, reports, and PLAN.md are versioned with the code. Only
runtime artifacts (.agent/phase-sessions/, .agent/pipeline.log)
remain ignored. Update AGENTS.md git protocol rule to match.
2026-09-01 10:18:22 -04:00
ducoterra 5fa620fde5 fix(web): keep the navbar stuck to the top — drop the body height cap on the sticky range
Build and Push Containers / build-and-push-app (push) Successful in 2m28s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-01 04:20:26 -04:00
ducoterra 725af9fac1 feat(docs): save chat answers as docs — edit screen, commit + push to the .env docs branch 2026-09-01 03:52:03 -04:00
ducoterra 7b7a834a1a feat(web): 2x reading column on wide desktops — 92rem at >=1500px (chat, shared, document view) 2026-09-01 00:39:19 -04:00
ducoterra 140b97ebf3 feat(kb): edit + re-embed document summaries from the viewer (admin) 2026-08-31 23:52:22 -04:00
ducoterra d94f3d5a52 feat(import): user-extensible BOR_IMPORT_EXTENSIONS — any well-formed extension, A9 family stays the default 2026-08-31 22:42:41 -04:00
ducoterra 281f3555c3 remove .agent, it shouldn't be committed. Do not commit it again
Build and Push Containers / build-and-push-app (push) Successful in 1m47s
Build and Push Containers / build-and-push-db (push) Successful in 14s
2026-08-31 08:08:16 -04:00
ducoterra 914097abcf feat(chat): save by default + share anonymously — auto-saved chats, guest-facing Share, success toast, action row 2026-08-31 05:20:25 -04:00
ducoterra c564e317ed fix(web): never 304 a rewritten page — pages drop conditional validators, assets keep them 2026-08-31 01:35:28 -04:00
ducoterra 9518d9d5d1 chore(agent): phase 53 — record task 06 completion move to complete/ 2026-08-31 00:04:17 -04:00
ducoterra 32b7bfd4b3 feat(chat): invalidate saved chats on sources sync — versioned stamps, stale marker, Regenerate against the new index 2026-08-30 23:39:15 -04:00
ducoterra ea8e041189 chore(agent): phase roadmap from TODO.md — 1 phase (55 save/share UX) 2026-08-30 21:13:23 -04:00
ducoterra aba8615177 fix(chat): rest the composer at the viewport bottom — sticky alone left it mid-screen
Phase 52's first pass shipped `position: sticky; bottom` on `.composer` and
called the phase done, but the owner's requirement — "the chat message-input
textarea should be at the bottom of the screen" — still failed in the browser:
on an empty/short chat the input rested just under the empty state (~57% of
the viewport) with a dead band down to the footer.

`position: sticky` can only pull a box UP toward the scrollport's bottom edge;
it can never push a box DOWN to meet it, so on a page that does not overflow
it is a no-op. The old story suite only exercised an overflowing conversation
(one test even asserted the buggy resting position as expected), which is why
the half-fix passed.

- `.messages { flex: 1 1 auto }` — absorbs a short page's free space so the
  composer's resting in-flow position is the bottom of the full-height column
  (body min-height:100dvh -> .app-main flex:1 -> .chat-shell flex:1); basis
  stays `auto`, no height cap, no overflow — the document stays the scroller
- `.composer { bottom: env(safe-area-inset-bottom, 0) }` — the explicit 0
  fallback replaces the env()-only offset, which degraded to `auto` (no pin)
  wherever env() is unsupported
- E2E: `test_empty_chat_composer_sits_in_normal_flow` ->
  `..._at_the_screen_bottom` (chrome-only band below the resting composer);
  the phone suite now checks the resting position as well as the pinned one
- Unit pins: the flex-grow half and the full-height column are pinned, so the
  fix cannot silently regress to sticky-only

Still CSS-only — no DOM change, no JS, no new scroll call site (phase 42
never-auto-scroll contract intact), no z-index.

Verified: 1019 unit/integration tests pass (app/ coverage 99%), ruff and
pyright clean; tests/e2e/test_pinned_composer.py green in isolation (4), plus
the stop/autoscroll/persistence/mobile-nav suites and 14 layout/scroll
neighbours green in isolation.
2026-08-30 16:14:18 -04:00
ducoterra 820753948e feat(chat): pin the composer to the viewport bottom — Stop is always reachable while reading 2026-08-30 14:59:40 -04:00
ducoterra 619bf2187a chore(agent): phase roadmap from TODO.md — 2 phases (52 pinned composer, 53 stale saved chats) 2026-08-30 09:33:50 -04:00
ducoterra 114b115034 feat(chat): share a chat by link — anonymous read-only /shared/<token> page, share/unshare 2026-08-30 01:34:44 -04:00
ducoterra ece93a7c8f feat(chat): save and view chat history — admin-only saved_chats, History page, open-a-chat return 2026-08-29 21:22:25 -04:00
ducoterra 6832957ab0 feat(chat): retry the last answer — redo-in-place Retry button on the latest brain bubble 2026-08-29 18:32:15 -04:00
ducoterra 1a60ecbd8b feat(chat): stop an in-flight answer — Send becomes Stop, the partial is kept and persisted, the model stream is torn down 2026-08-29 17:27:04 -04:00
ducoterra 6bf7f456d4 show sync button for admin
Build and Push Containers / build-and-push-app (push) Successful in 1m38s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-08-28 23:24:23 -04:00
ducoterra 3a404eb161 fix(chat): stop the submit up-hop and keep the thinking pin alive across paragraph breaks
Build and Push Containers / build-and-push-app (push) Successful in 3m55s
Build and Push Containers / build-and-push-db (push) Successful in 13s
- scrollReveal lands at the document bottom (window.scrollTo) instead of
  scrollIntoView({ block: 'end' }): the old alignment sat above the
  in-flow composer, so every Enter hopped the page up by the
  composer+footer height and pushed the composer below the fold.
- The thinking window's pin state is now captured BEFORE the re-render
  (const pinned = block.open && isThinkingNearBottom(textEl)): the
  post-render distance read the new chunk's rendered height, not the
  user's position, so any chunk taller than the 32px band (real-model
  deltas, '\n\n' paragraph breaks) killed the follow at the first
  2-newline gap.
- Mock LLM: new 'think in paragraphs' trigger (scratchpad with real
  blank-line breaks, 60-char frames) — the 12-char mock frames never
  rendered past the band, which is why the bug survived the E2E gates.
- E2E (both verified red against the old code):
  test_submit_does_not_hop_up, test_thinking_window_follows_across_paragraph_breaks.
- Unit source-marker tests updated to the new contracts.
2026-08-28 17:10:02 -04:00
ducoterra 03d26255c6 feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place
Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):

* POST /api/git-sources/upload (admin-only, require_admin): accepts
  .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
  (bounds BOTH the compressed upload and the total extracted bytes —
  zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
  escape and device/FIFO members rejected), and atomically swaps the
  content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
  archive suffix — no missing window, a failed upload never touches the
  existing folder/row/KB). The git_sources row is upserted by path
  (kind='local', no duplicates, added_at preserved), the models are
  checked fail-fast (503 sanitized when down — the folder/row stay
  committed and the next sync/re-upload retries idempotently), and the
  source is scanned synchronously in the request (single-source
  import_sources prune=True + change-gated KB overview), answering 200
  with the sync-style counts. One upload at a time (409); the request
  session is released before the scan so a concurrent TRUNCATE cannot
  deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
  archive_source_name (safe-name derivation), unpack_archive (guarded
  zip/tar extraction with the extracted-byte cap, no partial state),
  swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
  ~/bor-sources/uploads, deliberately separate from the git checkouts)
  and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
  startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
  multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
  removed; #archive-upload-form takes its place (labeled file input,
  "Upload & scan" button, the §7.4 never-stale lifecycle, inline
  role=alert error, role=status count line); hint + table caption
  updated. The POST /api/git-sources kind=local API contract is
  UNCHANGED — a plain directory is still registrable via the API, and
  existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
  API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
  upload→scan→list (the deterministic "Uploading…" in-flight state, the
  Local row, /api/docs + the RAG catalog), same-filename re-upload
  (in-place replace, prune, no duplicate row, v2-only folder), the
  422 inline error + recovery (the form is not wedged), and the
  anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
  replace, both new settings), the local-directory form removal noted,
  config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.

Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.

Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.
2026-08-28 15:57:59 -04:00
ducoterra 872a07cee7 feat(ui): rename nav items — "Sources" becomes "RAG", "Git sources" becomes "Sources"
Owner request (2026-08-28): the two admin-only nav items read like the
same thing, so they are relabeled — the document-catalog link
(#nav-sources, /sources.html) becomes "RAG" and the source-manager link
(#nav-git-sources, /git-sources.html) becomes "Sources".

Phase 48 (48_nav_rename_sources), label-only per the locked decision:
- all six pages (index, sources, git-sources, tuning, document, login):
  the two <a> texts swap; ids, hrefs, hidden defaults, is-active /
  aria-current placement, and nav order (Chat, RAG, Sources, Tuning)
  are byte-unchanged otherwise.
- header.js: comment/docstring label mentions only — the reveal-by-id
  logic is untouched (ship-hidden/reveal contract intact).
- test_git_sources_admin.py / test_mobile_hamburger_nav.py: the two
  suites that asserted the old label text are updated; comment-only
  label fixes in test_shared_header.py / test_nav_consistency.py.
- tests/e2e/test_nav_rename_sources.py: the story E2E (green in
  isolation) — renamed labels + unchanged hrefs/order/markers on all
  six pages, click navigation with the active marker, the anonymous
  ship-hidden contract, and regression guards for the untouched
  controls (#sync-label "Sync sources", viewer #doc-back "Sources").
- All eight surrounding header/nav suites stay green in isolation;
  unit+integration green, app/ coverage 99% (frontend-only change),
  ruff + pyright clean.

Note: per this phase file-level staging, the six page files and
header.js also carry the same-day in-flight owner rework that was
already in the working tree when phase 48 ran (mobile sign-in dropdown
copy, sync button ship-hidden on the Sources page); the label rename
itself is the two-text swap on each page.
2026-08-28 12:33:18 -04:00
ducoterra 03bead092c various fixes 2026-08-28 09:42:19 -04:00
ducoterra 5d679f5184 feat(import): index quadlet unit files and jinja templates (A9 revision)
Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.

- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
  import_extensions CSV gain the ten names (the original seven first);
  the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
  still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
  (HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
  docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
  quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
  RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
  quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
  constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
  fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
  default CSV + dotted import_extension_set include the ten, the
  validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
  new suffix (parametrized), the .container fixture chunks >=2 under
  the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
  verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
  tree indexes exactly the ten new files (unknown/hidden/excluded
  filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
  over a temp tree with .container/.volume/.j2 -> documents + chunks
  rows with stem titles; delta re-import updates only the changed .j2
  doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
  isolation): GET /api/docs (admin session) lists the four new-format
  docs with non-zero chunk counts and stem titles; the Sources table
  renders a row + .doc-link per file; the phase-26 modal shows the
  .container TOML ([Container] section + sentinel) with stem title and
  the container format badge; a RESE-JINJA-SENTINEL-33dd question
  FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
  FTS hits) — the bubble is not .is-deflected and a source chip names
  templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
  revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
  the revision note under the anchors table, and the §5 chunking-policy
  + §11 workflow lines. The only PLAN edit this phase.

Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.

Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
2026-08-28 07:02:24 -04:00
ducoterra 6be692d999 feat(header): hamburger dropdown nav on mobile (owner permission)
TODO.md L9 (owner permission 2026-08-27, roadmap A5): "The navbar on
mobile is way too squished. Make it a hamburger dropdown menu with a
nice animation." At <=640px the nav links leave the bar — a 44px
#nav-toggle opens #app-nav as an animated (180ms slide+fade)
edge-to-edge dropdown with comfortable rows and the auth visibility
contract intact inside the menu; at >640px the bar is byte-identical
to pre-phase-46 (hamburger absent, inline pills as before).

- frontend/*.html (all six pages): the shared bar gains the
  #nav-toggle button (type=button, aria-expanded=false,
  aria-controls="app-nav", aria-label="Menu", aria-hidden 3-line
  SVG icon) immediately before the nav, and the nav gains
  id="app-nav" — one <nav>, no duplicated links, so the whoami reveal
  works inside the menu unchanged (phase-34 same-bar contract intact).
- frontend/assets/styles.css: .nav-toggle is display:none outside media
  queries (desktop untouched); the <=640px block adds the 44px toggle
  (+hover in the .steering-toggle:hover family, sized 20px icon), turns
  .app-nav into the dropdown (absolute top:100% edge-to-edge under the
  sticky header, surface + hairline + --shadow-lg, z-index 21 =
  header+1, closed state invisible + non-interactive with the 180ms
  opacity/transform/visibility-delayed pair, .is-open the only
  opener), and comfortable 1rem/0.75rem menu rows — superseding the
  phase-34/35 pill-squeeze rules for .nav-link/.app-nav (the 900px
  tablet block, action pills, and 58px bar height untouched). The
  reduced-motion block stills BOTH the closed and .is-open states: the
  .is-open rule (0,2,0) out-specifies a bare .app-nav (0,1,0), so the
  override must name both — verified live in Chromium (task 03).
- frontend/assets/header.js: ONE module-owned binding (import-time,
  null-safe like the sign-out binding): click toggles .is-open +
  aria-expanded in sync, a delegated nav-link click closes, Esc closes
  and refocuses the toggle, and matchMedia("(max-width: 640px)")
  change drops the state on resize back to desktop. The binding
  touches only the container — ship-hidden whoami links stay hidden.
- tests/unit/test_hamburger_nav.py (new): the markup/CSS/JS contract
  pins (six identical toggles in the shared row, desktop byte-
  identical, dropdown + .is-open + 180ms + reduced-motion rules, the
  superseded squeeze rules gone, the one-binding behavior).
- tests/e2e/test_shared_header.py: assert_shared_bar gains mobile=True
  (at <=640px the bar shows the hamburger + the closed nav; the
  per-role menu contents are pinned by the story suite).
- tests/e2e/test_mobile_hamburger_nav.py (new, story suite, 375x812):
  toggle is a visible >=44px target, menu closed (opacity 0 /
  visibility hidden), no horizontal overflow; anonymous menu shows
  exactly "Chat" (admin-only links stay hidden inside); admin menu
  shows all four links (whoami reveal inside the menu); a link click
  navigates + the arrival page ships closed; Esc closes and refocuses
  the toggle (outside click does NOT close — accepted: the locked
  close set is Esc + link + resize, no backdrop); the 180ms
  opacity/transform pair is live and reducedMotion:reduce stills both
  states with open/close still working; 1280x800 regression — toggle
  display:none, all four inline links inside the header band.

Gates: unit+integration 773 passed; app/ coverage TOTAL 99%
(unchanged — frontend-only phase); story E2E 7 passed in isolation
(mock LLM, DB up); regression suites test_nav_consistency (6) /
test_header_consistency (3) / test_shared_header (6) /
test_responsive_polish (7) / test_tuning_nav_link (4) all pass in
isolation; ruff check + pyright clean. A11 honored: no CDN, no new
assets.

Also records the 46_mobile_hamburger_nav todo/ -> complete/ move.
2026-08-28 06:07:02 -04:00
ducoterra b855d0aef9 feat(rag): unbounded agent tool calls behind a round cap (owner revision)
Phase 45 (owner permission 2026-08-27, TODO.md L8: "allow the LLM
to make as many tool calls as it wants"): the phase-37 per-turn tool
budgets (BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each)
and their exhaustion refusals are removed — a grounded turn now offers
list_documents / read_document for the whole turn (re-lists included),
bounded only by the round cap:

- app/config.py: agent_max_rounds (BOR_AGENT_MAX_ROUNDS, default 10,
  negative rejected) replaces agent_list_calls / agent_read_calls;
  .env.example + README document the single knob; app/rag/prompts.py
  docstrings follow.
- app/rag/agent.py: the loop runs tools until the model answers or
  rounds >= max_rounds, at which point it forces one final no-tools
  answer (the cap is the only forced exit); 0 = no tools — exactly one
  tools=None request, byte-identical to the pre-phase-37 path (the
  kill switch). Rejected calls (unknown tool / missing args /
  already-in-context / unknown path) still consume a round, so
  pathological rejected-call streams are bounded by the cap. The
  per-call log line is now tool/args/round=N/M; the per-turn
  tool_calls=N field and the tool SSE event are unchanged.
- tests/e2e/mock_llm.py: MULTI_READ_TRIGGER ("read two documents") —
  the deterministic list -> read #1 -> read #2 -> forced-answer flow
  (byte-stable "I read <sp1> and <sp2>." line), classified by the
  count of tool-role read results; the phase-37 single-read flow stays
  byte-identical (unit-pinned in tests/unit/test_mock_tool_flow.py).
- tests/e2e/test_agent_unlimited_tools.py (new, story suite,
  mock-only): three tool frames/lines in order (one list, two reads —
  the second read is what the old read budget refused) + the
  both-named non-deflected answer; done.sources + chips = retrieval
  doc + both reads, deduped; no budget refusal rendered; the
  single-read marker flow regression (exactly one read, single tool
  pair).
- .agent/PLAN.md: the phase-45 SSE revision note (owner-locked, R2) —
  the only PLAN edit this phase; the phase-37 note's budget clause is
  marked removed.

Unit/integration rewrites (test_agent.py round-cap matrix incl. the
kill switch and rejected-call spam, test_config.py, test_chat_api.py
agent_max_rounds=0 fixtures) landed with the server core so every gate
stays green.

uv run pytest: 756 passed, app/ coverage 99%; ruff + pyright clean;
story E2E 4/4 in isolation (ran twice); regression E2E suites
(agent_document_tools unmodified, chat_rag, smoke) green in isolation.

Also records the 45_agent_unlimited_tools todo/ -> complete/ task-file
moves (00/01/02 pending in the working tree, task 03 moves on success).
2026-08-28 04:50:56 -04:00
ducoterra bc70ce36e0 feat(chat): render markdown tables in answers, viewer, and thinking
GFM pipe tables in the shared renderer (TODO.md L6): a table-protection
pass in frontend/assets/markdown.js (fences -> tables -> escape order)
pulls each header+separator+body block out as a placeholder, renders
cells escape-first with the same inline transforms, and reinserts a
semantic <table class="md-table"> inside a horizontal-overflow
.md-table-wrap — so a pipe table in a chat answer, the document
viewer/modal, and the thinking block all render the same semantic
table. Fences win over tables; lone pipes stay text.

- styles.css: .md-table palette rules (PLAN §7.2 tokens, no motion);
  min-width: max-content so a WIDE table keeps its natural width and
  the wrapper is the real scroller (width:100% alone wrapped the wide
  table's cells — proven by the new E2E).
- mock_llm.py: TABLE_TRIGGER ("show me a table") -> byte-stable
  TABLE_ANSWER (3-column table, <img onerror> XSS probe line, wide
  5-column table), checked before DEFLECT_MODE like SUMMARY_MODE.
- tests/fixtures/docs/homelab/tables.md: 3x3 pipe table + pipe-heavy
  fenced block (viewer/fence subject); the shared fixture set grows
  8 -> 9 docs, so every suite pinning the count (added/formats/
  stat-docs/EXPECTED_ROWS) is updated accordingly.
- tests/e2e/test_markdown_tables.py (new, story suite): chat table
  shape + non-deflection, wide-table wrapper scroll (no page
  overflow), XSS probe inert, viewer modal table, fence-not-a-table,
  lone pipe stays text.
- tests/e2e/test_agent_document_tools.py: fix a pre-existing flake —
  the "Calling tool…" label window is ~0.4 s at the mock's 0.1 s
  tool-frame pacing, and a polling expect could stride over it
  (failed 3 of 5 runs on the committed baseline). The pre-submit
  MutationObserver record is the deterministic source of truth; the
  racy to_have_text gate is gone.

uv run pytest: 738 passed, app/ coverage 99% (TOTAL unchanged);
ruff + pyright clean; story E2E 6/6 in isolation; regression E2E
suites (chat_rag, document_viewer, document_summaries, smoke) green.
2026-08-28 03:35:50 -04:00
ducoterra 27b7cb96d5 feat(chat): thinking window scrolls again, follows the tail only while pinned
TODO.md L7: "Add scrolling back to the thinking block, but have it
autoscroll while thinking content is generating." Owner direction
2026-08-27 (roadmap A2) reverses the phase-21 no-scroll choice
(2026-08-24): details.thinking .thinking-text is user-scrollable again
(overflow-y: auto — the 320px clip stays, owner-confirmed), and the
phase-17 per-chunk bottom-pin is GATED: the window follows the live
tail only while the user is pinned near its bottom (THINKING_NEAR_
BOTTOM_PX = 32); scrolling up pauses the follow, returning to the
bottom re-arms it on the next chunk (the gate re-runs on every frame).

- frontend/assets/styles.css: .thinking-text overflow-y: hidden ->
  auto; the phase-21 owner-choice comment is replaced with the
  2026-08-27 direction; max-height: 320px and every other declaration
  in the rule byte-identical.
- frontend/assets/app.js: export const THINKING_NEAR_BOTTOM_PX = 32 +
  isThinkingNearBottom(textEl) (scrollHeight - scrollTop -
  clientHeight <= band); the thinking-handler pin becomes
  `if (block.open && isThinkingNearBottom(textEl))` — a scrolled-up
  reader is never re-pinned and a closed (restored) block is never
  pinned; everything else in the handler (and phase 42's no page
  scroll) untouched.
- tests/unit/test_thinking_scroll.py (new, replaces the deleted
  tests/unit/test_thinking_no_scroll.py): pins the CSS contract (auto
  + 320px + owner-direction comment, no hidden/scroll left), the
  exported 32px band, the gate math, the gated pin (no unconditional
  `if (block.open)` remains), and the surviving collapsed-restore pin.
- tests/e2e/test_thinking_scroll.py (new, mock-only, replaces the
  deleted tests/e2e/test_thinking_no_scroll.py — its pins asserted the
  reversed phase-21 behavior, so both phase-21 files are deleted in
  this commit): user scroll restored on the frozen 4s-hesitation tail
  (wheel is 1:1; click+Home keyboard — the plain div is not
  keyboard-focusable by design, tabindex is test scaffolding; the
  literal drag holds the user's position — headless Chromium's
  overlay scrollbars are not grabbable by synthetic mouse events,
  documented in the suite), follow-while-pinned at the 2nd-to-last
  and last chunk (±1px) with the last chunk's text inside the visible
  rectangle, no re-pin over ≥5 mid-stream chunks after a
  half-window scroll-up, re-pin on the next chunk after returning to
  the bottom, the CSS contract, plus the phase-11 (long answer: page
  scrolls, bubble overflow untouched) and phase-17 (restored
  collapsed block with full text) regressions.
- tests/unit/test_chat_persistence.py: the CSS pin flips with the
  contract (auto in, hidden out — owner direction 2026-08-27).
- tests/unit/test_frontend_scroll.py: the "page-level band constant is
  gone" pin now excludes the phase-43 window-level
  THINKING_NEAR_BOTTOM_PX (a different band — the window's, not the
  page's).

E2E (isolation): test_thinking_scroll 7/7 (twice); regressions
test_thinking_display 5/5, test_chat_persistence 4/4,
test_no_reply_autoscroll 5/5, test_smoke 3/3; unit+integration 725
passed, app/ coverage 99% (unchanged — frontend-only phase);
ruff + pyright clean.
2026-08-28 01:36:32 -04:00
ducoterra 7c6763319b fix(chat): stop autoscrolling while a reply streams (owner direction)
TODO.md L5: "Get rid of the chat reply autoscroll, it's breaking things
like making it impossible for the user to scroll while a reply
generates." Owner direction 2026-08-27 (roadmap A1) revises the
phase-18 follow-the-bottom choice: the page NEVER auto-scrolls while a
turn streams. Kept (owner decision): the submit reveal (the user's own
message) and the one-shot phase-14 restore landing.

- frontend/assets/app.js: delete NEAR_BOTTOM_PX + isNearBottom;
  scrollReveal becomes the one unconditional scrollIntoView (still
  smooth, still "auto" under prefers-reduced-motion via SCROLL);
  addMessage(who, html, scroll = false) carries an explicit scroll
  intent — only the submit (", true") and the two restore landings
  scroll. The thinking/tool/delta handlers and the typing indicator
  drop their page-scroll calls; the thinking block's INTERNAL
  bottom-pin (textEl.scrollTop, phase 17 — reworked separately in
  phase 43) and the turn-end focus({ preventScroll: true }) survive.
- tests/unit/test_frontend_scroll.py: rewritten pin for the new
  contract — phase-18 gate absent, helper unconditional, explicit
  intent at submit/restore, no page-scroll call in the streaming
  handlers, typing bubble scroll-free, SCROLL reduced-motion intact.
- tests/unit/test_chat_persistence.py: restore-landing pin updated to
  the new signature (the old forced "auto" is gone; the landing
  rides the default SCROLL — noted at the call site).
- tests/e2e/test_no_reply_autoscroll.py (new, replaces the deleted
  test_follow_bottom_scroll.py): no autoscroll across >=10 samples
  (1px tolerance) during a long answer and during the thinking stream;
  submit-from-the-top still reveals the user message; the restore
  landing lands one-shot on the latest message and stays; long answer
  + sources and the collapsed thinking block persist and restore.

E2E (isolation): test_no_reply_autoscroll.py 5/5; regressions
test_chat_rag 3/3, test_thinking_display 5/5,
test_chat_persistence 4/4, test_long_answers 2/2, test_smoke 3/3;
unit+integration 723 passed, app/ coverage 99%; ruff + pyright clean.
2026-08-28 00:29:25 -04:00
ducoterra 6cf1df9bf2 feat(sync): fail fast with a modal when a model is unavailable
TODO.md L4: with a dead model endpoint the sync discovered it only
mid-import, after slow clones — and a tooltip on the button is not a
readable error.

- app/rag/llm.py: ModelUnavailableError + check_models(llm) — a tiny
  pre-sync probe (one short embedding + one 1-token-scale completion)
  that fails naming the unavailable model (embed first, then the
  summary model); the sync sanitizer still masks credentials.
- app/api/sync.py: the probe is step 1 of _run_sync — before source
  resolution and before any clone_or_pull; a model failure is just
  another 'failed' state (no new endpoint, A10/A12 untouched).
- frontend/assets/header.js: applySyncFailure now also opens the
  module-owned error modal (every page carrying #sync-btn, zero
  page-markup changes): lazily built backdrop + role=alertdialog
  panel, error text via textContent, close via button / Esc /
  backdrop, focus in-and-out to #sync-btn (with a body→#sync-btn
  fallback — the run's disabled button drops focus to <body>).
- frontend/assets/styles.css: the modal on the phase-08 error palette
  (z-index above the header, .is-open open/close, reduced-motion
  stilling, 44px close target).
- Tests: probe unit tests (both up / embed down / summary down /
  custom model names), sync integration (fail-fast before any clone,
  probe-before-effective_sources ordering, credential masking,
  healthy regression), the phase-41 source pins, and the story E2E
  (two module apps on distinct ports — dead endpoint on a closed
  loopback port vs session mock: ≤10 s fail-fast + modal contract,
  all three dismissal paths with focus out to #sync-btn, button
  title/.is-error + Sources banner untouched, healthy phase-32
  lifecycle regression to 'Synced HH:MM').

E2E (isolation): test_sync_model_down.py 4/4, test_sync_button.py
3/3, test_git_sources_admin.py 6/6, test_local_directory_sources.py
3/3; unit+integration 721 passed, app/ coverage 99%; ruff + pyright
clean.
2026-08-27 23:44:35 -04:00
ducoterra 6f9e033117 fix(header): ship the tuning toggle hidden — no anonymous flash
#steering-toggle (the header 'Tuning' button) shipped visible in all
six pages and was only removed after /api/whoami resolved, so
anonymous visitors saw it flash for the whole round-trip (TODO.md L3).
It now ships hidden on every page and initSharedHeader unhides it only
for admin — the same ship-hidden / reveal-for-admin contract as the
admin-only nav links; the anonymous end-state (removed from the DOM,
phase-16 'absent, not hidden') is unchanged.

Adds the story E2E suite (MutationObserver proves zero visible frames
for anonymous on every page, admin reveal + panel + count badge,
nav-contract regression) and the source-level unit pins. Also fixes
test_steering.py's BASE_SCRIPT_COUNT (2 → 3: brand.js + markdown.js +
app.js, since phase 39).
2026-08-27 22:38:58 -04:00
ducoterra 02c76ad328 chore(agent): phase roadmap from TODO.md — 8 phases (40–47), 24 tasks
Converts the 9 TODO items into an executable phase roadmap (Protocol B,
appended after phase 39):

- 40 tuning toggle anonymous flash (TODO L3)
- 41 sync fail-fast + modal when a model is down (TODO L4)
- 42 no reply autoscroll (TODO L5)
- 43 thinking scroll back — user scroll + gated autoscroll (TODO L7)
- 44 markdown tables (TODO L6)
- 45 agent unlimited tool calls behind BOR_AGENT_MAX_ROUNDS (TODO L8)
- 46 mobile hamburger nav (TODO L9)
- 47 quadlet + jinja import formats, A9 revision (TODO L10–L11)

Each phase carries a user story, a dedicated Playwright E2E suite plan,
and owner-locked decisions (R1 A9 format extension, R2 phase-37 budget
revision, A1–A5 scope decisions) confirmed 2026-08-27.

Also records the completed phases 30–39 todo/ -> complete/ moves that
were pending in the working tree. TODO.md is cleared (items now live in
.agent/phases/todo/).
2026-08-27 18:25:53 -04:00
ducoterra 492d8275e7 add git to containerfile
Build and Push Containers / build-and-push-app (push) Successful in 2m17s
Build and Push Containers / build-and-push-db (push) Successful in 15s
2026-08-27 10:32:12 -04:00
ducoterra beaca4fe97 add db build and push
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 10s
2026-08-27 10:00:21 -04:00
ducoterra fe55be0c35 feat(brand): configurable app name — BOR_APP_NAME drives /api/config + the frontend brand layer
Build and Push Containers / build-and-push (push) Successful in 1m50s
One env var (BOR_APP_NAME, default "Brain of Reese") now drives the app's
display name everywhere (TODO.md L12 — owner ask: "a way to customize the
name for 'Brain of'. Should be an env var."). The existing app_name setting
is the source of truth (phase locked decision — no new variable, no rename);
with the variable unset the app is byte-identical to before.

Endpoint (A10 public/stateless, no secrets):
  GET /api/config → exactly {app_name, version} (app/api/config.py, the
  health.py pattern; registered before the static mount). Integration tests:
  anonymous 200, default values, a Settings override follows, key set is
  exactly two keys — no other setting may leak in later.

Frontend brand layer (A11 — runtime fetch, static templates stay static):
  assets/brand.js — a CLASSIC script, first on all six pages, so its top
  level runs at parse time: window.BOR_BRAND = "Brain of Reese"
  synchronously (the default renders immediately, no blank flash), then a
  no-store fetch of /api/config applies the name — document.title (global
  replace), every .brand-text (a name starting "Brain of " keeps the bold
  split Brain of <strong>rest</strong>, any other name renders plain; the
  operator-controlled name is HTML-escaped before innerHTML), a TreeWalker
  over text nodes (script/style rejected — page source never rewritten),
  and the aria-label/placeholder/meta-content attributes. Fetch failure
  keeps the default + console.warn (the loadHealth house style).
  app.js (status labels, typing label, elapsed-hint aria, tool labels) and
  document.js (viewer titles) read window.BOR_BRAND at CALL time via
  brand() — a label set after the fetch lands carries the configured name.
  Containerfile: esbuild minify line for brand.js (classic, like markdown.js);
  the phase-33 ?v= cache-busting picks the new asset ref up automatically.

E2E (A16 — one story, one file, isolated): test_configurable_brand.py boots
a SECOND app instance (same DB/mock-LLM/admin-auth env block, port APP_PORT+1,
BOR_APP_NAME="Brain of Testy") — the shared conftest server keeps the
default name so every other suite's title/label assertions stay untouched —
and asserts /api/config on both instances, the index title/brand/greeting/
#messages aria-label, the sources + login page titles, and one pre-token
chat turn (think out loud marker) whose #send-status reads "Brain of Testy
is thinking"; the no-op regression pins the shared server's default bytes.

Docs: .env.example App section + README configuration reference — what it
affects (titles, header brand, status labels, aria text), the default, the
bold-split rendering rule.

Gates: 695 unit+integration passed, app/ coverage 99% (>90%), story E2E
green in isolation (two consecutive runs), brand-string suites (smoke,
shared header, header consistency, chat persistence) green, ruff + pyright
clean.
2026-08-27 02:24:16 -04:00
ducoterra 94d7228510 feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.

Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.

API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).

Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.

Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.

Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.

Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
2026-08-27 01:04:16 -04:00
ducoterra 15c1272828 feat(rag): agent document tools — list/read tools with env-tuned budgets, SSE tool events + "calling tool" UI
Grounded chat turns now run the agent loop (app/rag/agent.py) instead
of a bare chat_stream: while the per-turn budgets last
(BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each) the model
gets list_documents (the indexed catalog, /api/docs order) and
read_document (full text, never truncated — A7-revised contract); once
both budgets are spent the tools key is dropped from the request and
the model must answer. Rejected calls (unknown tool, unknown/missing
path, document already in context, spent budget) consume no budget.
Budgets 0/0 make exactly one tools=None request — byte-identical to
the pre-phase path (budgets-as-kill-switch). Deflected turns keep the
direct chat_stream (A8 unchanged; the LOW prompt never carries the
<tools> section).

SSE contract gains {"type":"tool","name":...,"argument":
"source/path"|null} frames ahead of the answer deltas (PLAN §4
extension, owner permission 2026-08-26); done.sources, query_log.sources
and the per-turn log line (gains tool_calls=N) report the retrieval
docs + read docs, deduped. The UI shows a "calling tool"
button/label state and one visible .tool-call line per call above the
answer; the lines persist with the chat record and re-render on
reload. chat_stream passes tools through and accumulates streaming
tool_calls deltas into ToolCallPiece (tools=None stays byte-identical).

E2E: deterministic mock tool flow ("use your tools" + <tools> marker:
list -> read first catalog line -> quoted answer) plus the story suite
(marker flow, reload re-render, plain/deflected no-tool regressions).
Docs: .env.example + README (the two tools, the budgets, the SSE tool
frame, the "calling tool" UI state).

probe: turbo tool_calls=supported 2026-08-26 (uv run python -m
scripts.llm_probe --tools — non-streaming + streaming
finish_reason=tool_calls, indexed delta.tool_calls partials)
2026-08-26 22:39:14 -04:00
ducoterra 9efffcb428 feat(viewer): show document summary together with the original (TODO.md L5) 2026-08-26 19:11:15 -04:00
ducoterra 1925bb66a8 feat(sources): admin page to add and remove git sources (TODO.md L4) 2026-08-26 18:42:28 -04:00
ducoterra b2d8696741 feat(ui): one consistent navbar on every page (TODO.md L3) 2026-08-26 15:39:42 -04:00
ducoterra 0a46f07fa8 chore(agent): phase roadmap from TODO.md, 3 phases (34-36) 2026-08-26 09:41:32 -04:00
ducoterra 8fabb7efda perf(ui): cache busting — HTML no-cache + versioned asset URLs (?v=) with immutable 1y asset caching
Phase 33 (story: .agent/user_stories/cache-busting.md).

- app/core/caching.py: asset_version() — git short SHA (a commit is a
  deploy), stable content-hash fallback for non-git checkouts, "dev"
  for a missing static dir; computed once per process. CachingMiddleware
  — the five HTML pages revalidate (no-cache) with ?v=<token> asset refs
  rewritten in flight; /assets/* is public, max-age=31536000, immutable;
  everything else (all /api/*, the SSE chat stream in particular) passes
  through byte-identical.
- tests/e2e/test_cache_busting.py: fresh-Chromium wire assertions —
  document no-cache, versioned CSS/JS request URLs sharing one token,
  immutable asset headers, /api/health baseline headers, SSE chat to
  done (mock LLM).
- README 'Caching / deploys' section + story file.

Also fixed two prod-image defects surfaced by this phase's podman smoke
(the full app would not boot):
- Containerfile: ship the scripts/ package — app/api/sync.py (phase 32)
  imports scripts.git_sync / scripts.import_docs at module level, so the
  container crashed on boot (ModuleNotFoundError: No module named
  'scripts').
- compose.yaml: pass BOR_ADMIN_PASSWORD / BOR_SESSION_SECRET through to
  the app service (:- defaults keep 'podman compose up -d db' working;
  the app's own fail-loud gate still names missing admin auth).

Smoke: podman compose --profile prod up -d on a fresh image + a fresh
Chromium profile — /, /sources.html and /login.html all served
Cache-Control: no-cache; all 8 asset requests versioned with one shared
token (content-hash fallback inside the image — no .git there);
/assets/* immutable for a year.
2026-08-25 22:42:10 -04:00
ducoterra 52136fe307 feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status 2026-08-25 21:39:38 -04:00
ducoterra 0654b304e1 feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, <knowledge_base> section in HIGH+LOW prompts 2026-08-25 20:22:51 -04:00
ducoterra 572a4190a6 feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc 2026-08-25 17:48:37 -04:00
ducoterra 9809482a4b chore(agent): phase roadmap from TODO.md — 4 phases (lite document summaries, KB overview in prompt, admin sync button, cache busting) 2026-08-25 15:43:44 -04:00
ducoterra 3841bd5a30 feat(ui): expose Global Tuning from Chat + Sources headers (admin-only Tuning nav link) 2026-08-25 14:51:16 -04:00
ducoterra 3d044f33a1 feat(rag): git-based import sources — BOR_GIT_SOURCES repos cloned (first run, --depth 1) or pulled (--ff-only) into BOR_SOURCES_DIR/<repo>/ then indexed; --source still wins; a failed sync aborts before importing anything 2026-08-25 14:23:02 -04:00
ducoterra 589e26dbe9 feat(rag): global tuning manager — /tuning.html + PUT /api/steering/{id}: create, edit, list, delete steering notes without a chat 2026-08-25 13:46:32 -04:00
ducoterra fcde1fd37b feat(ui): documents open in an almost-fullscreen modal instead of a new page — same-page overlay on chat + Sources, /document.html kept as the no-JS/direct-link fallback 2026-08-25 13:45:57 -04:00
ducoterra 476aa0e066 chore(agent): phase roadmap from TODO.md — 3 phases (document modal, global tuning, git-based sources) 2026-08-25 10:15:42 -04:00
ducoterra 025f57beb5 fix(ui): background no longer moves — static grid, three glow spots fading in and out on their own slow cycles (owner 2026-08-25) 2026-08-25 09:43:19 -04:00
ducoterra 1e6ae360e0 feat(rag): feed whole matched documents to the LLM — no context truncation (A7 revised) 2026-08-24 23:37:44 -04:00
ducoterra d7a4064616 fix(build): Containerfile builds again — relative module imports, all four pages and shared assets in the image 2026-08-24 22:54:30 -04:00
ducoterra 0adc9b5801 fix(ui): animated background actually animates — grid drift and glow breathe per the phase-08 design 2026-08-24 22:16:41 -04:00
ducoterra 22a6121411 fix(ui): thinking window no longer scrolls — live 320px view pinned to the stream tail 2026-08-24 20:28:12 -04:00
ducoterra 986e704eb1 chore(agent): add phase 24 from TODO.md — whole-document context (no truncation, A7 revised); clear TODO.md 2026-08-24 18:53:24 -04:00
ducoterra 04a7f4c05d fix(ui): thinking window no longer scrolls — live 320px view pinned to the stream tail 2026-08-24 18:24:57 -04:00
ducoterra 76c6a01199 fix(chat): keep the in-flight answer when navigating away mid-turn — partial answer restored on return 2026-08-24 15:53:07 -04:00
ducoterra 824914ca3d chore(agent): add phases 20-23 from TODO.md — four owner-confirmed bug fixes; clear TODO.md 2026-08-24 15:23:49 -04:00
ducoterra 2afc77ee56 feat(ui): shared header — Sign in/Sign out and New Chat on every page; hide the Sources nav link from anonymous users 2026-08-24 12:32:45 -04:00
ducoterra fd7f02ce68 remove tests from pipeline
Build and Push Containers / build-and-push (push) Failing after 52s
2026-08-24 11:02:13 -04:00
ducoterra c37477d3f9 add gitea build and push
Build and Push Containers / test (push) Failing after 14s
Build and Push Containers / build-and-push (push) Skipped
2026-08-24 11:01:06 -04:00
ducoterra 076358db94 uncommit .agent 2026-08-24 10:59:45 -04:00
ducoterra bc0158f858 feat(ui): chat auto-scrolls only while pinned to the bottom — submitting reveals your message, scrolling up holds the viewport 2026-08-24 10:53:04 -04:00
ducoterra b16deb2b1d feat(chat): stream model thinking over SSE and show it in a collapsible block 2026-08-24 09:52:27 -04:00
ducoterra cbc263a4b2 feat(auth): single-admin password login (signed cookie) — gate tuning + Sources catalog, keep chat and document viewer public 2026-08-23 19:58:39 -04:00
ducoterra fc0d9a2d5c feat(rag): steering notes — tune how Brain answers, stored in Postgres and injected into every system prompt 2026-08-22 16:44:42 -04:00
ducoterra 19df7df99d feat(ui): persist the chat conversation in localStorage — survives refresh and navigation, with a New chat reset 2026-08-22 15:52:53 -04:00
ducoterra 2485b50af0 fix(ui): document viewer back button returns to the page you came from (chat or sources) 2026-08-22 15:26:43 -04:00
ducoterra 8ca564cd83 fix(ui): uniform header bar height on chat, sources, and the document viewer 2026-08-22 15:14:31 -04:00
ducoterra 0da5275eeb fix(rag): lift chat output cap to 32768 tokens — long answers no longer cut off 2026-08-22 11:30:19 -04:00
ducoterra 6ec6181c7b feat(ui): clickable document viewer — open any cited document in the browser from chat chips and the sources table 2026-08-22 02:08:49 -04:00
ducoterra 7e8d14702e feat(rag): hybrid FTS+vector retrieval and multi-format ingestion — name-your-tool questions find the right document 2026-08-22 01:27:02 -04:00
2607 changed files with 299182 additions and 2037 deletions
-399
View File
@@ -1,399 +0,0 @@
# Brain of Reese — Master Plan
> **Status:** Phase 1–3 complete (scaffolded, designed, decomposed).
> **Rule:** Every agent reads this file first. Decisions marked `LOCKED` in the
> Anchors table are settled — do not re-litigate them in a phase.
> **Revisions (2026-08-21, owner permission):** A7/A8/A9 revised (multi-format
> ingestion, hybrid FTS+vector retrieval, re-tuned honesty gate); dark tech
> theme (Phase 08); clickable document viewer (Phase 10). See roadmap §12.
---
## 1. Mission
A **knowledge base chatbot** that embeds the `~/Homelab` and `~/Deployments`
projects into a Postgres vector database and lets anyone ask *Reese* (the
bot) questions about them.
**Product feel:** a chippy, upbeat assistant that is optimistic about the
user's ability ("you've got this") and **radically honest** — if retrieval
didn't surface anything relevant it says *"I haven't done anything like
that"* and offers alternatives instead of hallucinating.
### In scope (v1)
- Chat UI (mobile-friendly, well-styled, no auth, no CDN).
- RAG over text knowledge files — `md, markdown, txt, yaml, yml, json, py`
by default (A9, revised 2026-08-21) — from `~/Homelab` + `~/Deployments`
(and any future directory the importer is pointed at).
- Self-hosted models via `https://aipi.reeseapps.com/v1` — `turbo` (chat),
`embed` (embeddings, **768 dims — verified**).
- Postgres 17 + pgvector, cosine similarity, chunk→document mapping so the
LLM receives the **entire relevant document** as context.
- Idempotent import/update script, documented in the README.
- Ample server logging + explicit UI loading/progress feedback (never a
stale submit button).
### Out of scope (v1)
- Auth / multi-user (API is stateless under `/api` so it can be added later).
- Binary / non-text content, file uploads, caching layer, message persistence.
- Real-time document watching (manual re-import for now).
---
## 2. Architectural Anchors (LOCKED DECISIONS)
| # | Component | Decision | Rationale | Status |
|---|-----------|----------|-----------|--------|
| A1 | Runtime | Python 3.12+, `uv` for all package management | Fast, reproducible envs; one language for API + tooling | LOCKED |
| A2 | Web framework | FastAPI + Pydantic v2 + Uvicorn | Async, typed, SSE-friendly for LLM streaming, free OpenAPI docs | LOCKED |
| A3 | Database | **PostgreSQL 17** (`docker.io/postgres:17`, pgvector compiled in via `db/Containerfile`) with **cosine** (`<=>`) search | One system for relational + vectors; pgvector is mature; official base image kept per project standard | LOCKED |
| A4 | Orchestration | `compose.yaml`, started with **`podman compose up -d`** | Matches Reese's toolchain | LOCKED |
| A5 | LLM backend | OpenAI-compatible `https://aipi.reeseapps.com/v1`; models **`turbo`** (chat) & **`embed`** (embeddings); `openai` async client | Self-hosted, offline from cloud; no new model management | LOCKED |
| A6 | Embedding dim | **768** (verified 2026-08-21 against live endpoint via `scripts/llm_probe.py`); configured by `BOR_EMBEDDING_DIM` | User recalled 768 — probe confirmed; dimension is fixed at table creation, so mismatch must fail loudly at import time | LOCKED |
| A7 | Retrieval→context | **Hybrid:** cosine top-30 + Postgres FTS top-30 (OR tsquery, `ts_rank`) fused with **RRF (k=60)** → map to parent documents ranked by best fused chunk score → feed the **full text of top-N=2 documents** (deduped, capped at 24k chars) to the LLM | Owner permission 2026-08-21: pure-cosine top-4 missed real docs (gitlab case — best chunk ranked 7th behind vendored-cache junk; score compression 0.41–0.84); the lexical signal finds name-your-tool questions; whole-document context contract preserved | LOCKED (revised 2026-08-21) |
| A8 | Honesty gate | **Deflection mode** (LLM must open with a variant of *"I haven't done anything like that"* and offer 2–3 alternative questions) when best cosine < `BOR_RELEVANCE_THRESHOLD` **and** no candidate chunk FTS-matches the question; threshold re-tuned for the `embed` model's compressed score range (default **0.62**, calibrated via `scripts/eval_retrieval.py`; the E2E mock uses its own 0.30 calibration via the app fixture) | Owner permission 2026-08-21: at 0.30 the gate never discriminated (measured corpus range 0.41–0.84); the FTS-OR keeps name-your-tool questions honest-positive; deflection product behavior unchanged | LOCKED (revised 2026-08-21) |
| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-21) |
| A10 | Auth | **None in v1**; all endpoints stateless under `/api` | Per user (auth later); statelessness keeps the future migration cheap | LOCKED |
| A11 | Frontend | Vanilla HTML/CSS/JS in git; **no CDN** — everything served by FastAPI `StaticFiles`; minified by esbuild in the `Containerfile` build stage; system font stack | No external deps at runtime; tiny, auditable surface; mobile-friendly by construction | LOCKED |
| A12 | Aux services | **None in v1** (no Valkey, no SeaweedFS) | No sessions/auth (no store), no uploads (no object storage); add later only if a need appears | LOCKED |
| A13 | Migrations | Alembic + SQLAlchemy 2.0 (sync) + psycopg 3 | Standard, reversible, reviewable schema history | LOCKED |
| A14 | Debugging | `debugpy` **only when `DEBUGPY=1`** (env var read directly, not via settings); listen `0.0.0.0:5678` (override `DEBUGPY_PORT`), non-blocking, attach-on-demand; **not imported at all when off** | Zero overhead by default per project standard; attach-on-demand keeps production runs clean | LOCKED |
| A15 | Chat transport | **SSE streaming** from `POST /api/chat` (deltas + final `done` event with metadata) | Local LLM latency is 10–30s; live token stream + explicit completion event power the UI's feedback states | LOCKED |
| A16 | Testing | Per phase: unit + integration (pytest, **coverage >90%** on `app/`) + **one dedicated Playwright E2E file per user story**, run in isolation; E2E uses a deterministic mock LLM by default (`E2E_REAL_LLM=1` opts into live aipi) | One story, one phase, one E2E gate — the pipeline's core invariant | LOCKED |
| A17 | Git | Conventional Commits, **always `--no-gpg-sign`**, repo-local `commit.gpgsign=false`; one atomic commit per completed phase | Subsequent agents may lack the GPG key | LOCKED |
---
## 3. High-Level Architecture
```
┌────────────────────────────────────────────┐
│ Podman Compose │
Browser │ ┌──────────────────────────────────────┐ │
┌──────────┐ HTTP │ │ brain-of-reese/app (FastAPI) │ │
│ index.html│◄──────┼─►│ • static frontend (no CDN) │ │
│ app.js │ SSE │ │ • /api/chat /api/suggestions │ │
└──────────┘ │ │ • /api/health /api/docs │ │
│ │ • RAG pipeline (embed→retrieve→gen) │ │
│ └──────┬──────────────────┬───────────┘ │
│ │ SQL (psycopg) │ OpenAI-compat│
│ ┌──────▼──────┐ ┌───────▼────────────┐ │
│ │ db: │ └─────────┬──────────┘ │
│ │ postgres:17 │ │ │
│ │ + pgvector │ │ │
│ └─────────────┘ │ │
└──────────────────────────────┼────────────┘
▼
https://aipi.reeseapps.com/v1
(self-hosted: turbo, embed)
Offline tooling (same repo, same venv):
scripts/import_docs.py → walks A9-format dirs, chunks, embeds, upserts
scripts/eval_retrieval.py → ranks hybrid results for a question (tuning)
scripts/llm_probe.py → verifies models + embedding dim
```
### Component breakdown
| Component | Responsibility | Lives in |
|-----------|----------------|----------|
| **App (FastAPI)** | Serves frontend + `/api`; RAG pipeline; logging | `app/` |
| **RAG pipeline** | `embed` → pgvector cosine top-K → doc mapping → context assembly → `turbo` (streamed) with persona/honesty prompt | `app/rag/` (added in story phases) |
| **Importer** | Directory walk (A9 formats, hidden dirs skipped, exclusions), sha256 delta detection, format-aware chunking, batched embedding, upsert/prune | `scripts/import_docs.py` (story phase) |
| **DB** | `documents`, `chunks`, `query_log` + `vector` extension | `db/` image, `alembic/` |
| **Frontend** | Chat shell, sources view, loading/feedback states | `frontend/` |
### Chat data flow
```
user question
→ POST /api/chat {message}
→ embed(question) [aipi /v1/embeddings, model=embed]
→ cosine top-30 + FTS top-30 (OR tsquery, ts_rank) [pgvector + PG FTS]
→ RRF fuse (k=60) → docs ranked by best fused chunk score
├─ best cosine >= 0.62 OR fts_hits > 0 → top-2 documents' FULL content
│ → system prompt (persona + HONESTY rules + docs)
│ → turbo, stream=True → SSE deltas
└─ else → DEFLECT_MODE system prompt (weak hits as topics)
→ turbo, stream=True → SSE deltas (honest reply)
→ query_log row (question, score, deflected, sources, latency)
→ final SSE "done" event: {deflected, sources[], suggestions[]}
```
---
## 4. API Design
All endpoints stateless (A10). Errors: standard JSON `{detail: str}`.
| Method | Path | Purpose | Story |
|--------|------|---------|-------|
| GET | `/api/health` | Liveness + db up/down + version | 01 |
| GET | `/api/suggestions` | Onboarding suggestion strings | 01 (05 refines) |
| GET | `/api/docs` | Indexed document list (source, path, title, chunks, indexed_at) | 02 |
| GET | `/api/documents/content?source=…&path=…` | One indexed document's full content (feeds the viewer page) | 10 |
| POST | `/api/chat` | RAG chat turn → **SSE stream** | 03/04 |
### SSE contract (`POST /api/chat`)
```
data: {"type":"delta","text":"Hey! "}\n\n
data: {"type":"delta","text":"Good "}\n\n
...
data: {"type":"done","deflected":false,"sources":[{"source":"Homelab","path":"kubernetes.md","title":"Kubernetes Homelab Cluster"}],"suggestions":[]}\n\n
```
Client rules: render deltas as they arrive; on `done` append source chips /
suggestion chips and clear the busy state; on HTTP/stream error show the
error banner + retry (never a stuck button).
---
## 5. Data Model (PostgreSQL 17)
Created by `alembic/versions/0001_initial_schema.py` (idempotent
`CREATE EXTENSION IF NOT EXISTS vector`).
### `documents`
| Column | Type | Notes |
|--------|------|-------|
| id | `UUID` PK | |
| source | `VARCHAR(120)` | source dir basename, e.g. `Homelab` |
| path | `VARCHAR(1000)` | relative to source dir, e.g. `ansible/roles/k3s.md` |
| full_path | `VARCHAR(2000)` | absolute path at import time (diagnostics) |
| title | `VARCHAR(500)` | first markdown H1, else file stem |
| content | `TEXT` | **full markdown — the RAG context** |
| content_hash | `VARCHAR(64)` | sha256 of content — change detection |
| indexed_at | `TIMESTAMPTZ` | |
| — | `UNIQUE (source, path)` | upsert key |
### `chunks`
| Column | Type | Notes |
|--------|------|-------|
| id | `UUID` PK | |
| document_id | `UUID` FK→documents CASCADE | **embedding→document mapping** |
| position | `INT` | 0-based order within the doc |
| content | `TEXT` | chunk text (heading-aware) |
| embedding | `VECTOR(768)` | nullable until embedded (two-phase import) |
| tsv | `TSVECTOR` | **generated** `to_tsvector('english', content) STORED` + GIN index (hybrid retrieval, A7) |
> No vector index in v1: sequential scan is fine at this corpus size
> (~100–500 docs). Revisit with an HNSW index if retrieval latency grows.
### `query_log`
`id UUID PK, question TEXT, top_score FLOAT, fts_hits INT, chunk_hits INT, deflected BOOL, sources TEXT, latency_ms INT, created_at TIMESTAMPTZ`
### Document state transitions
```
unseen ──import──▶ indexed ──hash changed + re-import──▶ reindexed
│
└──file deleted + --prune──▶ removed (chunks cascade)
```
### Chunking policy (markdown-aware)
Split on `## `/`### ` headings into sections; sub-split any section longer
than `BOR_CHUNK_TARGET_CHARS` (2000) at paragraph boundaries with
`BOR_CHUNK_OVERLAP_CHARS` (200) overlap; each chunk keeps its nearest
preceding heading in the text for retrieval quality.
**Format-aware (A9, revised):** `yaml`/`yml` split on top-level keys and
`---` separators (key line kept as anchor); `json` pretty-printed, split on
top-level keys; `py` split on top-level defs/classes (stdlib `ast`);
`txt` on paragraphs; markdown unchanged. Every format honors the 1200-char
hard cap (aipi ~1024-token request limit).
---
## 6. RAG Pipeline & Persona
### Locked system prompt (sent with every chat turn)
```
You are "Brain of Reese" — the digital brain of Reese, a self-hoster and
homelab tinkerer. Personality: chippy, upbeat, warm, and genuinely
optimistic about the user's ability to do things ("you've got this").
Rules:
1. Answer ONLY from the provided document context. Cite which document(s)
you used, by path.
2. Be concrete: names, versions, ports, hosts, schedules — the specifics in
the docs are the value.
3. HONESTY GATE: if <relevance> is "LOW", you must NOT pretend to know.
Start your answer with a variant of: "I haven't done anything like that."
Then offer 2-3 alternative questions about things you DO have notes on.
4. Never invent facts, hosts, or steps that are not in the context.
5. Keep answers tight: short paragraphs, bullets where helpful.
<relevance>{HIGH|LOW}</relevance>
```
- `HIGH` mode appends the full document text under `<documents>…</documents>`.
- `LOW` mode (deflection) appends only the **titles** of the weak hits so the
model can suggest real alternatives (marker used by the E2E mock:
`DEFLECT_MODE` appears in the system prompt).
### Retrieval (hybrid — A7/A8, revised 2026-08-21)
- Embed the question (`embed`, 768-d) → cosine top-30 candidates.
- Lexical: OR tsquery over the question's tokens → FTS top-30 by `ts_rank`.
- **Reciprocal Rank Fusion** (`Σ 1/(k+rank)`, k=60) → distinct parent docs
ranked by best chunk's fused score → top 2 → full content, concatenated,
truncated to `BOR_MAX_CONTEXT_CHARS` (24k) with a `[…truncated…]` marker.
- Honesty gate: LOW only when `best cosine < BOR_RELEVANCE_THRESHOLD`
(default 0.62, calibrated against the `embed` model's measured 0.41–0.84
distribution) **and** zero FTS hits among the candidates.
---
## 7. UI/UX Strategy
### 7.1 Layout structure
- **App frame:** sticky header (64px) + `<main>` (flex-grow) + footer.
Container: `max-width: 72rem; margin-inline: auto; padding-inline: 1.25rem`.
- **Chat:** a *centered column capped at 46rem*. This is deliberate: chat is
a vertical conversation — a centered, capped column is the correct pattern
(NOT a layout bug). The 72rem frame + header/footer ensure the column
never reads as a hairline in a sea of whitespace.
- **Sources page:** full-width responsive **table** (min 640px, horizontal
scroll wrapper on small screens) + stat cards in
`grid-template-columns: repeat(auto-fit, minmax(170px, 1fr))`.
No skinny single-column lists anywhere: lists/tables/grids use ≥80–90% of
the container width.
- **Mobile (≤640px):** suggestion chips become a horizontally scrollable row;
composer stays reachable with `safe-area-inset-bottom`; touch targets ≥44px.
### 7.2 Accessibility (WCAG 2.1 AA)
- Semantic landmarks on every page: `<header>`, `<nav aria-label>`,
`<main>`, `<footer>`; skip-link to `#main`.
- Every control labeled: visible `<label>` or `aria-label` (icon-only
buttons always get `aria-label`); form input has a (visually-hidden) label.
- Live regions: message stream `aria-live="polite"`; typing indicator
`role="status"`; banner `role="status"`; errors `role="alert"`.
- **Dark tech theme (Phase 08, 2026-08-21)** — page `#0a0e17`, surface
`#121a2e`; ink `#e8ebf4` on surface ≈14.5:1; ink-soft `#9aa4bd` on
surface ≈6.9:1; **dark ink `#0a0e17` on brand `#6d78f2` ≈5.2:1** (white
on brand ≈3.7:1 — never used for text); brand-ink `#a5b4fc` on
brand-soft `#232b52` ≈6.9:1; deflection `#fbbf24` on `#2b2110` ≈9.5:1
(border `#f59e0b`); error `#fca5a5` on `#2d1318` ≈9.1:1. All computed,
all ≥4.5:1. `prefers-reduced-motion` also stills the Phase-08 background
layer.
- `:focus-visible` outline 3px; `prefers-reduced-motion` respected by the
typing/spinner animations.
### 7.3 No external dependencies
- System font stack only (no font files to bundle, no CDN fonts).
- Zero `<script src="https://…">` / `<link href="https://…">` — enforced by
an integration test (`tests/integration/test_api.py::test_index_html_served_locally`)
and re-checked by every UI phase's verification step.
- Markdown rendering is a ~60-line local function (escape-first, then
transform) — XSS-safe, no library.
### 7.4 Visual feedback standard (the "never stale" contract)
| State | UI |
|-------|----|
| **Idle** | Send button enabled, label "Send". |
| **Thinking (pre-token)** | 3-dot typing bubble + button disabled with spinner, label "Thinking…". |
| **Streaming** | Deltas append live into the brain bubble; button stays busy. |
| **Done (answer)** | Source chips under the bubble (mono, path-based); button re-enabled. |
| **Done (deflected)** | Amber-bordered bubble + "Maybe try:" suggestion chips. |
| **Error** | Red banner (`role="alert"`) with retry hint; button re-enabled. |
| **KB offline** | Amber banner at top of chat ("start Postgres…"); chat disabled with explanation. |
| **Guard** | 120s client-side timeout → error state (a button can never sit "stuck" forever). |
### 7.5 Component inventory (ids used by tests)
`#messages` (stream), `#empty-state`, `#suggestions`, `.suggestion-chip`,
`#composer`, `#message-input`, `#send-btn` / `#send-label`, `#typing-indicator`,
`.msg.user/.msg.brain .bubble`, `.source-chip`, `.msg.brain.is-deflected`,
`#kb-banner`, `#app-version`; sources: `#stat-docs`, `#stat-chunks`,
`#stat-last`, `#docs-table`, `#docs-tbody`, `#sources-empty`; viewer
(Phase 10): `/document.html`, `#doc-title`, `#doc-meta`, `#doc-content`,
`.doc-raw`, `.format-badge`, `#doc-not-found`, `.doc-link` (Sources table
path links).
---
## 8. Debugging (debugpy protocol)
- `DEBUGPY` unset/`0` → **`debugpy` is never imported** (verified by unit test).
- `DEBUGPY=1` → listener on `0.0.0.0:${DEBUGPY_PORT:-5678}`, **non-blocking**,
app continues; IDE attaches on demand.
- Entry point: `app/core/debugging.py::configure_debugging()` called at the top
of `app/main.py` module import — so `uv run uvicorn app.main:app`,
`python -m scripts.…`, and tests all honor it.
- VS Code: `"type": "debugpy", "request": "attach", "connect": {"host": "localhost", "port": 5678}`.
---
## 9. Observability
- **App logs:** single-line `timestamp LEVEL logger :: message` on stdout;
uvicorn access logs on. INFO by default (`BOR_LOG_LEVEL`).
- **Per-chat-turn log line (required):**
`question=… embed_ms=… top_score=… fts_hits=… threshold=… deflected=… sources=… total_ms=…`
- **Importer logs:** per-file `added|updated|unchanged|pruned` + summary
(counts, embedding batches, total time).
- **`query_log` table:** durable record of every question (score, deflection,
sources, latency) for tuning the threshold and finding gaps in the docs.
---
## 10. Testing Strategy (LOCKED — A16)
| Layer | Tooling | Runs | Gate |
|-------|---------|------|------|
| Unit | pytest | `uv run pytest tests/unit` | pass |
| Integration | pytest + FastAPI TestClient | `uv run pytest tests/integration` | pass |
| Coverage | pytest-cov on `app/` | `uv run pytest --cov=app --cov-report=term-missing` | **>90%** per phase |
| E2E | Playwright (sync API), one file per story | `uv run pytest tests/e2e/test_<story>.py -v --no-cov` | passes **in isolation** |
- **E2E determinism:** `tests/e2e/mock_llm.py` serves a deterministic
OpenAI-compatible API. Embeddings are genuine L2-normalized token-overlap
vectors, so the cosine threshold behaves like production: on-topic
questions retrieve, off-topic questions deflect. `E2E_REAL_LLM=1` switches
the app fixture to live aipi (needs imported KB).
- **E2E prerequisites:** `podman compose up -d db`; Chromium installed via
`uv run playwright install chromium`.
- DB isolation: story E2E fixtures truncate `query_log` (and re-import
fixtures for import-dependent stories) per test module.
---
## 11. Import & Update Workflow (documented in README)
```
# first import (and any future refresh):
uv run python -m scripts.import_docs # defaults: ~/Homelab ~/Deployments
uv run python -m scripts.import_docs --source ~/OtherProject # extra dirs
uv run python -m scripts.import_docs --prune # drop deleted / filtered-out files
uv run python -m scripts.eval_retrieval "How did I install gitlab?"
uv run python -m scripts.llm_probe # sanity: models + dim
```
Behavior: sha256 delta per `(source, path)` — unchanged files are skipped
(no re-embedding); changed files are re-chunked + re-embedded (chunks
replaced atomically); `--prune` removes docs whose files disappeared or no
longer match the format filter. Formats per A9 (revised): `md, markdown,
txt, yaml, yml, json, py` (`BOR_IMPORT_EXTENSIONS`), hidden (dot)
directories skipped, exclusion list applied. `scripts/eval_retrieval.py`
ranks live hybrid results for a question (retrieval tuning).
---
## 12. Roadmap (one story → one phase → one Playwright gate)
| Phase | File | Story | Playwright gate |
|-------|------|-------|-----------------|
| 01 | `01_infrastructure.md` | — (foundation) | `tests/e2e/test_smoke.py` |
| 02 | `02_story_import_documents.md` | `import-documents.md` | `tests/e2e/test_import_documents.py` |
| 03 | `03_story_chat_rag.md` | `chat-rag-answer.md` | `tests/e2e/test_chat_rag.py` |
| 04 | `04_story_honest_deflection.md` | `honest-deflection.md` | `tests/e2e/test_honest_deflection.py` |
| 05 | `05_story_suggestion_chips.md` | `suggestion-chips.md` | `tests/e2e/test_suggestion_chips.py` |
| 06 | `06_story_loading_feedback.md` | `loading-feedback.md` | `tests/e2e/test_loading_feedback.py` |
| 07 | `07_story_responsive_polish.md` | `responsive-polish.md` | `tests/e2e/test_responsive_polish.py` |
| 08 | `08_story_dark_tech_theme.md` | `dark-tech-theme.md` | `tests/e2e/test_dark_tech_theme.py` |
| 09 | `09_story_retrieval_quality.md` | `retrieval-quality.md` | `tests/e2e/test_retrieval_quality.py` |
| 10 | `10_story_document_viewer.md` | `document-viewer.md` | `tests/e2e/test_document_viewer.py` |
Completion = unit+integration green, coverage >90%, story E2E green in
isolation, UI verification passed, **one `--no-gpg-sign` commit**.
---
## 13. Future (post-v1 hooks, deliberately not built)
- Auth (stateless API makes this a drop-in: sessions → Valkey).
- HNSW index on `chunks.embedding` at scale.
- Conversation persistence (messages tables).
- Watchdog auto-re-import (inotify) — until then the script is the truth.
- More sources: any directory of A9-format files via `--source`.
-61
View File
@@ -1,61 +0,0 @@
# Story: Chat RAG Answer (happy path)
**Phase:** `03_story_chat_rag.md` · **E2E:** `tests/e2e/test_chat_rag.py`
## Narrative
As **a user** (friend, colleague, future me), I want to ask Brain a question
about Reese's setup and get a grounded, chippy answer that points me at the
exact documentation — so I can actually *do* the thing.
- **Given** the knowledge base is imported and I type "How is my Kubernetes
cluster set up?"
- **When** Brain embeds the question, retrieves the top chunks by cosine
similarity, maps them to their parent documents, and feeds the **full
document text** to `turbo`
- **Then** I see a streamed, upbeat answer that cites the source
(`Homelab/kubernetes.md`), grounded in the doc's specifics (Talos,
Cilium, the node list) — and never in anything the docs don't say.
## Acceptance criteria
1. `POST /api/chat` streams SSE: `delta` events then a final `done` event
carrying `{deflected, sources[], suggestions[]}` (PLAN §4).
2. Retrieval: top-4 chunks (`BOR_TOP_K_CHUNKS`), cosine via pgvector
`<=>`, score = 1 − distance.
3. Context assembly: top-2 **distinct documents** by best-chunk score, full
content, capped at `BOR_MAX_CONTEXT_CHARS` with truncation marker.
4. System prompt = locked persona + HONESTY GATE rules (PLAN §6), with
`<relevance>HIGH</relevance>` and `<documents>…</documents>`.
5. The answer arrives **streamed** (multiple deltas), rendered live.
6. Source chips (mono, `source/path`) render under the answer bubble.
7. Per-turn log line emitted (PLAN §9) and a `query_log` row inserted
(`deflected=false`, top_score, sources, latency).
8. LLM/embedding failure → JSON/SSE error the UI turns into the error banner
(no hang, no stale button).
## UI Visualization & Structure
- Chat column centered at 46rem (PLAN §7.1); user bubble right (brand
indigo, white text ≥4.5:1), Brain bubble left (white, ink text, avatar 🧠).
- While generating: typing indicator → live-appended text (see
loading-feedback story for the full state machine — this story only needs
"deltas render as they arrive and the button is busy throughout").
- **Source chips:** pill, `font-family: mono`, `bg --brand-soft`,
`color --brand-ink` (6.3:1), `max-width` + ellipsis; each shows
`Homelab/kubernetes.md`. `aria-label` when truncated.
- Bubble content is safe-rendered markdown (escape-first local renderer —
`<script>` in an LLM answer must NOT execute).
- On mobile the bubbles expand to ~92% width; chips wrap.
## Playwright Mapping Rule
**Test Scenario → `tests/e2e/test_chat_rag.py`** (mock LLM, seeded KB):
1. `test_on_topic_question_streams_grounded_answer` — type "How is my
Kubernetes cluster set up?", submit; assert: answer bubble appears with
streamed content (mock's answer references the question), a `.source-chip`
containing `kubernetes.md` is present, send button returns to enabled
"Send".
2. `test_chat_logs_query` — after the turn, `GET /api/health` is still ok AND
(via a test-only detail: query the DB directly) a `query_log` row exists
with `deflected=false` and sources including `kubernetes.md`.
3. `test_sse_stream_shape` — raw `httpx` streaming request to `/api/chat`:
assert multiple `data:` delta events precede a `done` event with
`deflected: false` and a non-empty `sources` list.
-57
View File
@@ -1,57 +0,0 @@
# Story: Honest Deflection
**Phase:** `04_story_honest_deflection.md` · **E2E:** `tests/e2e/test_honest_deflection.py`
## Narrative
As **a user**, when I ask something Brain genuinely has no notes about, I
want it to **admit it plainly** and still be helpful — so I never walk away
with a confident-sounding hallucination.
- **Given** the knowledge base is about homelab/infra topics
- **When** I ask "How do I bake sourdough bread?"
- **Then** retrieval's best similarity is below `BOR_RELEVANCE_THRESHOLD`,
Brain switches to deflection mode, opens with a variant of
**"I haven't done anything like that"**, stays chippy, and offers 2–3
alternative questions about things it *does* know (from the weak hits).
## Acceptance criteria
1. Gate: `max(1 − cosine_distance) < BOR_RELEVANCE_THRESHOLD` ⇒
`<relevance>LOW</relevance>` + `DEFLECT_MODE` system prompt (weak-hit
**titles only**, no full docs).
2. The LLM is still called (voice stays chippy); the prompt forces the
honesty phrasing + alternative suggestions (PLAN §6).
3. `done` event carries `deflected: true` and `suggestions[]` (2–3 strings).
4. `query_log` row has `deflected=true` + the weak `top_score`.
5. UI: the deflected bubble is visually distinct (amber border/background),
and "Maybe try:" chips render below it; clicking a chip asks that
question (delegated to the suggestion-chips story for chip behavior;
here only rendering).
6. Threshold is env-tunable; lowering it to ~0 makes every question an
"answer" (documented in README troubleshooting).
7. Unit tests cover the gate boundary (score == threshold → answer mode;
just below → deflect) using a fake retriever — no LLM needed.
## UI Visualization & Structure
- Deflected brain bubble: `background: var(--accent-bg) #fff7e8`,
`border: 1px solid var(--accent-line) #f59e0b`, text stays `var(--ink)`
(or accent-ink for emphasis ≥4.5:1) — clearly "different" from a normal
answer without being alarm-red (it's honesty, not an error).
- Below the bubble: `Maybe try:` label (visually hidden for SR, `aria-label`
on the chip group) + 2–3 `.suggestion-chip` pills (same chip component as
onboarding: ≥44px height, brand-soft bg, brand-ink text).
- Bubble may include the model's alternative list in text too; chips are the
one-click affordance.
- Contrast audit: `#92400e` on `#fff7e8` ≈ 8.7:1 ✓; chip text on chip bg ≥6:1 ✓.
## Playwright Mapping Rule
**Test Scenario → `tests/e2e/test_honest_deflection.py`** (mock LLM, seeded KB):
1. `test_off_topic_question_deflects_honestly` — ask "How do I bake
sourdough bread?"; assert the answer bubble is `.is-deflected`, its text
matches /haven't done anything like that/i, and ≥2 "Maybe try:" chips
render below it.
2. `test_deflection_suggestions_are_clickable` — click the first deflection
chip; assert the input is populated/focus behavior per chip contract and a
new user bubble is created.
3. `test_threshold_gate_unit_boundary` is a **unit** test (not Playwright):
retriever returns score 0.30 → HIGH; 0.2999 → LOW (mocked components).
-62
View File
@@ -1,62 +0,0 @@
# Story: Import Documents
**Phase:** `02_story_import_documents.md` · **E2E:** `tests/e2e/test_import_documents.py`
## Narrative
As **Reese** (the owner), I want to point the importer at one or more
directories of markdown files and have them chunked, embedded, and stored in
Postgres — so that Brain's answers always reflect my *current* documentation.
- **Given** the `~/Homelab` and `~/Deployments` trees (or any `--source` dirs)
- **When** I run `uv run python -m scripts.import_docs`
- **Then** every `*.md` file (after the exclusion list) is present in the
`documents` table with its full content, a sha256 hash, and chunk rows with
768-dim embeddings; unchanged files are skipped on re-runs; and the
Sources page in the browser shows the indexed documents.
## Acceptance criteria
1. `scripts/import_docs.py` accepts repeatable `--source PATH` (default
`~/Homelab` `~/Deployments`), `--prune`, and `--limit N` (debug).
2. Only `*.md` files are imported; excluded dirs: `.venv`, `node_modules`,
`.git`, `__pycache__`, `.pytest_cache`, `dist`, `build` (PLAN A9).
3. Delta detection by sha256 on `(source, path)`: unchanged → skipped
(no re-embedding); changed → re-chunked + re-embedded, old chunks
replaced atomically.
4. Embeddings are batched (`BOR_EMBED_BATCH_SIZE`) against `aipi /v1/embeddings`
(`embed`); a dimension mismatch fails loudly with an actionable message.
5. Rich per-file logging (`added|updated|unchanged|pruned`) + summary.
6. `GET /api/docs` returns the document list; the Sources page renders it
(stat cards + table) or the designed empty state when none exist.
7. The whole flow works against the **mock LLM** in E2E (deterministic),
and against real aipi for manual runs.
## UI Visualization & Structure
- **Sources page (`/sources.html`), desktop:** header row (h1 + sub), then
stat cards in `repeat(auto-fit, minmax(170px,1fr))` (documents / chunks /
last indexed), then a **full-width table** inside a scroll wrapper
(min-width 640px → horizontal scroll, never a squeezed hairline list).
Columns: Source · Path (mono, ellipsized w/ `title`) · Title · Chunks ·
Indexed. Uses ≥85% of the 72rem container width.
- **Empty state (no docs):** centered card with 📂, "Nothing indexed yet",
and the exact import command in a `<code>` pill. No dead links, no
placeholder tables.
- **Accessibility:** `<caption class="visually-hidden">` on the table,
`scope="col"` on headers, `role="region"` + `tabindex="0"` on the scroll
wrapper (keyboard scrollable), stat values have visible labels.
- **Mobile:** stat cards stack (auto-fit), table scrolls horizontally,
no content below the fold is unreachable.
## Playwright Mapping Rule
**Test Scenario → `tests/e2e/test_import_documents.py`** (one isolated
Playwright suite for this story):
1. *Seeding:* run the import function in-process against
`tests/fixtures/docs/` (mock embeddings, temp DB state) — a fixture, not
the test's subject.
2. `test_sources_page_lists_indexed_docs` — goto `/sources.html`, assert stat
cards show the fixture counts and the table rows include
`homelab/kubernetes.md`, `homelab/backups.md`, `deployments/new-service.md`.
3. `test_sources_table_layout` — table wrapper width ≥80% of container;
`caption` present; on a 375px viewport the wrapper scrolls horizontally.
4. `test_empty_state_when_no_docs` (fresh/truncated DB) — empty state visible
with the import command; table hidden.
-61
View File
@@ -1,61 +0,0 @@
# Story: Loading Feedback & Progress
**Phase:** `06_story_loading_feedback.md` · **E2E:** `tests/e2e/test_loading_feedback.py`
## Narrative
As **a user**, local LLM answers can take 10–30+ seconds. I want to *always*
know Brain is working — a clear "thinking" state, live progress as tokens
arrive, and a definitive end — so I never stare at a stale Send button
wondering if it's stuck.
- **Given** I submit a question
- **When** the answer is in flight (pre-token, streaming, or erroring)
- **Then** the UI shows an unambiguous in-progress state, transitions
cleanly to done/error, and the send button is never left in a zombie state.
## Acceptance criteria
1. **Pre-token:** typing-indicator bubble (3 animated dots, `role="status"`,
`aria-label="Brain of Reese is thinking"`) + send button disabled with
spinner and label "Thinking…".
2. **Streaming:** first delta replaces the typing indicator; text appends
live; button stays busy until `done`.
3. **Done:** button re-enabled, label "Send", input focused back.
4. **Error paths:** (a) LLM/DB error → red banner `role="alert"` with retry
hint, button re-enabled; (b) **120s client timeout** → same error state
(guard against a hung stream); (c) page reload mid-stream loses the
stream but the composer is usable again (state is turn-local).
5. **Slow-model E2E:** the mock LLM's 3s warm-up (message containing
"pretend to think slowly") must show the typing indicator for ≥2s before
any text appears.
6. Server side: per-turn log includes `embed_ms` / total `total_ms` (PLAN
§9) so "slow" is diagnosable.
7. `prefers-reduced-motion`: dots/spinner still visible (slower/static) —
feedback is never removed, only calmed.
## UI Visualization & Structure
- State machine (single source of truth in `app.js`):
`idle → thinking → streaming → done | error → idle`.
- Typing indicator: 8px dots, `--ink-soft`, staggered 1.2s bounce; inside a
normal brain bubble (same geometry as answers) so the layout doesn't jump.
- Send button busy style: `background: #a5b4fc` (disabled contrast still
fine — it's a disabled state), 16px spinner (2.5px ring, white top
arc), label swap "Send" ↔ "Thinking…".
- Error banner: `--err-bg/--err-ink/--err-line`, top of chat shell,
`role="alert"`, includes the actionable hint ("Try again — if this
persists, check the LLM is reachable").
- Elapsed-time hint: after 10s still pre-token, the typing bubble's aria
label becomes "…still thinking (12s)" — SR users are never left guessing.
## Playwright Mapping Rule
**Test Scenario → `tests/e2e/test_loading_feedback.py`** (mock LLM):
1. `test_typing_indicator_during_slow_think` — ask "pretend to think slowly
then tell me about kubernetes"; assert `#typing-indicator` visible within
500ms of submit, still visible at ~2s, gone by the time the answer text
is present.
2. `test_button_state_machine` — during the in-flight turn: `#send-btn`
disabled + label "Thinking…"; after done: enabled + "Send".
3. `test_streaming_appends_live` — capture bubble text at two timestamps
during the stream; second length > first (progress is visible).
4. `test_error_banner_on_llm_down` (fixture stops the mock) — submit;
assert `role=alert` banner visible and button re-enabled within timeout.
-65
View File
@@ -1,65 +0,0 @@
# Story: Responsive, Polished, Accessible UI
**Phase:** `07_story_responsive_polish.md` · **E2E:** `tests/e2e/test_responsive_polish.py`
## Narrative
As **a user on any device** — phone at the coffee shop, laptop at the
desk — I want the chat to be comfortable to read and drive: no pinched
layout, no tiny tap targets, no contrast failures, no wasted whitespace —
so asking Brain feels effortless everywhere.
- **Given** any viewport from 360px to 1600px+
- **When** I use the chat and the Sources page
- **Then** the layout follows the PLAN §7 standards (containers, chat
column, full-width table), all interactive elements are reachable by
keyboard, and every color pair meets WCAG 2.1 AA.
## Acceptance criteria
1. **Layout:** container 72rem centered with side padding; chat column
capped at 46rem centered; Sources table uses full container width with
horizontal scroll below 640px (never a squeezed single hairline column).
2. **Mobile (375px):** header condenses, composer reachable above the home
indicator (`safe-area-inset-bottom`), chips scroll horizontally, bubbles
≤92% width, no horizontal page overflow (document `scrollWidth ==
clientWidth`).
3. **A11y sweep:** landmarks present on both pages (`header/nav/main/
footer`); skip link works (focus `#main`); all inputs have labels
(visible or programmatically associated); all icon-only buttons have
`aria-label`; `:focus-visible` outline on every control (Tab through).
4. **Contrast:** automated check of the key pairs (ink/surface,
ink-soft/surface, white/brand, chip-ink/chip-bg, deflection pairs) ≥4.5:1
(test computes from computed styles; PLAN §7.2 table is the baseline).
5. **No-CDN re-verification** on both pages (no `http(s)://` src/href
except same-origin `/…`).
6. **Reduced motion:** with `prefers-reduced-motion`, typing dots and
spinner do not animate (computed `animation: none` or duration ≥2s).
7. Long words/paths (e.g. a 60-char file path) wrap or ellipsize without
breaking the bubble (overflow-wrap anywhere).
## UI Visualization & Structure
- This phase is the **visual audit + fix pass**: it does not add features,
it enforces PLAN §7 end-to-end on chat + sources.
- Desktop 1440px screenshot pass: header 64px, chat centered with balanced
margins, sources table edge-to-edge within the container.
- Tablet 768px: chat column uses most of the width (≤46rem cap), no
mid-column dead zones; stat cards 3-across.
- Phone 375px: one-column flow, 44px+ targets, thumb-zone composer.
- Any deviation found → fix in `frontend/assets/styles.css` (tokens first),
re-verify with the E2E below.
## Playwright Mapping Rule
**Test Scenario → `tests/e2e/test_responsive_polish.py`**:
1. `test_no_horizontal_overflow_at_viewports` — for 360/375/768/1280/1600:
`document.documentElement.scrollWidth <= clientWidth` on both pages.
2. `test_chat_column_capped_and_centered` — at 1600px, `.chat-shell`
width ≤ 46rem (736px) + 2% and horizontally centered (±2%).
3. `test_sources_table_full_width` — at 1280px, `.table-wrap` width ≥ 80%
of `.container` width.
4. `test_a11y_landmarks_and_labels` — both pages: landmarks present,
skip link target `#main` focusable, `#message-input` has an associated
label, no `<img>`/icon buttons without accessible name.
5. `test_contrast_pairs_pass_aa` — computed-color contrast assertions for
the PLAN §7.2 pairs (helper computes WCAG relative luminance).
6. `test_reduced_motion_respected` — emulate `reducedMotion: 'reduce'`;
typing dots have no running animation (or ≥2s duration).
-58
View File
@@ -1,58 +0,0 @@
# Story: Suggestion Chips
**Phase:** `05_story_suggestion_chips.md` · **E2E:** `tests/e2e/test_suggestion_chips.py`
## Narrative
As **a user who opens the chat for the first time** (or after a deflection),
I want a few **concrete example questions** right in front of me — so I
immediately understand what Brain is good at and can start with zero
friction.
- **Given** I land on the chat page
- **When** the app is healthy
- **Then** I see 3–4 suggestion chips drawn from `GET /api/suggestions`
(defaults in settings, tuned to the real Homelab topics), and clicking one
fills the composer and submits it.
## Acceptance criteria
1. `GET /api/suggestions` returns the configured list (settings-driven,
overridable via `BOR_SUGGESTIONS` JSON env).
2. Chips render in the empty state as `<button class="suggestion-chip">`
(real buttons, not links/divs) with `role="list"` container +
`role="listitem"` items; `aria-label="Suggested questions"` on the group.
3. Click behavior: fills `#message-input`, focuses it, **and submits**
(one tap → answer). Keyboard: Tab to chip, Enter activates.
4. After the first user message the empty state (and its chips) is replaced
by the conversation; chips re-appear only on deflection (see
honest-deflection story).
5. If `/api/suggestions` fails, the chat still works (progressive
enhancement — no chips, no error spam).
6. Mobile: chips become a horizontally scrollable single row
(no wrapping into the composer's territory).
## UI Visualization & Structure
- Chips: pill (`border-radius: 999px`), `bg --brand-soft`, `text --brand-ink`
(≥6:1), 1px `--line` border, **min-height 44px**, comfortable
`padding 0.55rem 1rem`; hover deepens bg; `:active` scales 0.98.
- Desktop: `flex-wrap: wrap`, centered under the empty-state subcopy, gap 0.5rem.
- Mobile (≤640px): `flex-wrap: nowrap; overflow-x: auto` single row,
`scrollbar-width: thin`, chips `flex: 0 0 auto` (thumb-friendly, no
accidental double-tap on wrapped lines).
- Default suggestion copy (tune to real docs in this phase):
1. "How is my Kubernetes cluster set up?"
2. "What's my backup strategy?"
3. "How do I deploy a new service?"
4. "What's currently running in the homelab?"
## Playwright Mapping Rule
**Test Scenario → `tests/e2e/test_suggestion_chips.py`**:
1. `test_onboarding_chips_render` — goto `/`, assert ≥3 `.suggestion-chip`
visible inside `#suggestions` (role=list) with non-empty text.
2. `test_chip_click_submits` — click the first chip; assert a user bubble
with the chip's exact text appears and the brain reply (mock) follows.
3. `test_chips_keyboard_accessible` — Tab from the page start reaches the
first chip; Enter submits it.
4. `test_chips_mobile_row` — at 375px viewport, the chip row is
horizontally scrollable (`scrollWidth > clientWidth` or single-line
height check) and no chip is cut vertically.
-39
View File
@@ -1,39 +0,0 @@
#!/usr/bin/env bash
# .agent/validate.sh — validation gate for the phased-execution pipeline.
#
# A phase is only moved to .agent/phases/complete/ if this script exits 0.
# Gates (PLAN §10 / AGENTS.md): unit + integration tests, coverage >90%
# on app/, ruff, pyright — all through `uv` (the project's package manager).
set -uo pipefail
rc=0
if [[ -f pyproject.toml || -f pytest.ini || -f setup.py ]]; then
out="$(uv run pytest -q --cov=app --cov-report=term 2>&1)"; pytest_rc=$?
printf '%s\n' "$out" | tail -n 30
if [[ $pytest_rc -ne 0 ]]; then
echo "pytest FAILED (exit $pytest_rc)"
rc=1
fi
total="$(printf '%s\n' "$out" | grep -E '^TOTAL' | awk '{print $NF}' | tr -d '%')"
if [[ -n "${total:-}" ]]; then
if awk -v c="$total" 'BEGIN { exit !(c > 90.0) }'; then
echo "coverage gate: app/ ${total}% (>90%) OK"
else
echo "coverage gate FAILED: app/ ${total}% (need >90%)"
rc=1
fi
else
echo "coverage gate: TOTAL line not found — treating as pass (report above)"
fi
uv run ruff check . || rc=1
uv run pyright || rc=1
fi
if [[ $rc -ne 0 ]]; then
echo "validation FAILED (see output above)"
else
echo "validation OK"
fi
exit "$rc"
+363
View File
@@ -0,0 +1,363 @@
# Brain of Reese — Master Plan (minimal)
> **Status:** Minimal re-land (2026-09-10). See §12 for current state.
> Section numbers and anchor IDs match the
> previous full plan (in git history, commit `dac4a3e`) so the codebase's
> `PLAN §…` / anchor comments stay valid — consult that revision or the
> phase records in `.agents/phases/complete/` for full detail and the
> complete owner-permission revision log.
>
> **Rule:** Every agent reads this file first. `LOCKED` anchors in §2 are
> settled — a change requires explicit owner permission, recorded as a
> dated revision note under the table. Never a silent deviation
> (AGENTS.md rule 3).
>
> **This is deliberately minimal:** it captures the tech stack, the
> architecture an agent needs to work, and the locked decisions. It does
> NOT recap the shipped work — the git log, the README, and the completed
> phase directories are the history.
---
## 1. Mission
A **knowledge-base chatbot** ("Brain of Reese") over the owner's homelab
documentation — git repos, local directories, and uploaded archives
registered on the admin Sources page — embedded into Postgres + pgvector.
Product feel: chippy, upbeat, and **radically honest** — if retrieval
surfaces nothing relevant it says *"I haven't done anything like that"*
and offers alternatives instead of hallucinating. Self-hosted LLMs only.
**Deliberately out of scope:** binary/non-text ingestion; real-time file
watching (the import script / Sync button is the refresh loop); PR
tooling for docs push (the owner opens the PR); multi-admin/per-user
accounts (one admin + hand-out tokens is the model).
---
## 2. Tech Stack & Architectural Anchors (LOCKED)
| # | Component | Decision |
|---|-----------|----------|
| A1 | Runtime | Python 3.12+, `uv` for all package management (API + scripts + tests, one venv) |
| A2 | Web framework | FastAPI + Pydantic v2 + Uvicorn (async, SSE-friendly) |
| A3 | Database | **PostgreSQL 17 + pgvector** (local build of official `postgres:17`, `db/Containerfile`), cosine (`<=>`) search; one system for relational + vectors |
| A4 | Orchestration | `compose.yaml`, `podman compose up -d` (dev: `db` only; `--profile prod` adds the app container) |
| A5 | LLM backend | OpenAI-compatible self-hosted endpoint `https://aipi.reeseapps.com/v1` via the `openai` async client: **`turbo`** (chat, streams `reasoning_content` thinking), **`embed`** (embeddings), **`lite`** (one-shot: document summaries, KB overview, folder summaries) |
| A6 | Embedding dim | **768** (verified against the live endpoint); `chunks.embedding` is fixed at table creation — a dim mismatch must **fail loudly**, never silently re-embed |
| A7 | Retrieval→context | **Hybrid:** cosine top-100 ∪ Postgres FTS top-30 (OR tsquery, `ts_rank`), fused with **RRF (k=60)** → parent docs ranked by best fused chunk → **full text of top-N=2 documents, never truncated** on the retrieval path (owner 2026-08-24: "this should never happen"; the agent `read`-tool cap is a separate owner-permitted path) |
| A8 | Honesty gate | Deflect (LOW mode) **only when** best cosine < `BOR_RELEVANCE_THRESHOLD` (0.62) **and** (zero FTS hits **or** best cosine < `BOR_LEXICAL_SUPPORT_FLOOR` (0.35)); an FTS hit flips HIGH only when `best_cosine >= lexical_support_floor` — the vector signal must corroborate the lexical match (A8 revised 2026-09-14, owner-confirmed, TODO L2a: lexical-only hits without vector support deflect). LOW prompt carries weak-hit *titles only* + the `DEFLECT_MODE` marker (the E2E mock keys on its presence) + the plain-text no-tools line |
| A9 | Content scope | Default `md, markdown, txt, yaml, yml, json, py` + Podman quadlet family + `j2`; `BOR_IMPORT_EXTENSIONS` may name **any** well-formed extension or narrow the set; hidden (dot) paths + exclusion list + per-source `ignore_paths` (raw prefixes, no globs) always apply |
| A10 | State & auth | `POST /api/chat` is **stateless** (client-provided `history` only, budget-trimmed — nothing stored per conversation). Auth = single admin, signed `bor_session` cookie (Starlette `SessionMiddleware` + itsdangerous; no server-side session store); admin-issued SHA-256-hashed access tokens are the only other identity; **only** shared chats stay anonymous. Fail-loud at boot while `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` are empty |
| A11 | Frontend | Vanilla HTML/CSS/JS in git, **no CDN** — everything served by FastAPI `StaticFiles`; system font stack; the navbar views are views of ONE shell document (`frontend/index.html` + `router.js` deep-links) |
| A12 | Aux services | **None** (no Valkey/queue/SeaweedFS): sync runs in-process, the login rate limit is in-memory per-process, sessions are the signed cookie. Restart clearing in-memory state is accepted |
| A13 | Migrations | Alembic + SQLAlchemy 2.0 (sync) + psycopg 3; **every migration ships a tested downgrade** (A13 — reversible) |
| A14 | Debugging | `debugpy` imported **only when `DEBUGPY=1`** (`app/core/debugging.py`); never imported otherwise (unit-tested) |
| A15 | Chat transport | **SSE** from `POST /api/chat`; event types `thinking`, `delta`, `tool`, `retry`, `done`, `error`, `tool_result`; no proxy buffering, no client caching on the stream |
| A16 | Testing | Per phase: unit + integration (pytest) with **>90% coverage on `app/`** + one dedicated **Playwright E2E file**, run **in isolation** (`--no-cov`); E2E uses a deterministic mock LLM by default (`E2E_REAL_LLM=1` opts into live aipi) |
| A17 | Git | Conventional Commits, **always `--no-gpg-sign`**, one atomic commit per completed phase |
| A18 | Docs push | Save-a-answer-as-doc: server-side draft (`doc_drafts`, long body never in a URL — unguessable `uuid4` token is the URL credential) pushed to a **generic git remote** (`BOR_DOCS_REPO`, URL or local path; no `gh`) on a dedicated branch, `--ff-only`, re-cut per push; **no PR tooling**; inert (hidden + 409) while the repo is unset |
| A19 | Deploys & caching | HTML pages `Cache-Control: no-cache` (no etag/last-modified); `/assets/*` versioned (`?v=<token>`) + `immutable, max-age=1y`; **the token is the deploy** (git `HEAD` short SHA; content-hash fallback) — a deploy is a commit the browser sees without a hard refresh |
| A20 | Security headers | Every response: `Content-Security-Policy: default-src 'self'` (+ the theme's inline-`<style>` hash on themed pages), `frame-ancestors 'none'`, `X-Content-Type-Options: nosniff`; login rate-limited 10 attempts / 15 min (in-memory, A12) |
**UI/theming anchors (B):**
| # | Decision |
|---|----------|
| B1 | Theme `ui_settings` columns: **NULL/empty = "use the default"** — env value for the 3 strings, built-in palette for colors (colors have no env fallback) |
| B3 | The semantic state families (`--ok-*`, `--err-*`, `--accent-*`) **join** the storable palette — 17 variables total, all `NULL = built-in` (the built-in theme stays byte-identical). The 2026-09-09 lock that they are "not identity" is lifted |
| B4 | **Byte-identical contract:** with no `ui_settings` row (or a fully default theme) the served HTML is byte-identical to the built-in default — no `#bor-theme` tag, caching rewrites are rewrite-only |
| B5 | Admin-only nav views (Sources, Git sources, Tuning, History, Tokens, Theme) are hidden from non-admins; a monochrome theme must keep every state **text** label ("text + color, never color alone") |
**Key recent revisions (full log in the previous plan revision / phase
records):** A10 — single-admin cookie auth, saved chats + shares +
staleness, client history budgets, access tokens.
A15 — `thinking`, `tool`, unlimited tool rounds under
`BOR_AGENT_MAX_ROUNDS`, `retry`, harness-aligned
`ls`/`read`/`grep` surface. A7 — the **retrieval** path's
never-truncated contract is unchanged; the `read`-tool cap is
the only exception, owner-permitted 2026-09-10.
---
## 3. High-Level Architecture
```
Browser (SPA shell, no CDN)
│ HTTP pages · SSE chat · JSON API
▼
FastAPI app (app/main.py)
├── middleware: SessionMiddleware → caching (A19) → SecurityHeaders (A20)
├── /api/* chat, docs, git-sources, steering, sync, chats,
│ doc-drafts, tokens, ui-settings, auth, health, suggestions
├── RAG pipeline (app/rag/) embed → retrieve → gate → agent → turbo
└── StaticFiles (frontend/) — shell routes registered before the mount
│ SQL (psycopg) OpenAI-compat
▼ ▼ ▼
Postgres 17 + pgvector aipi.reeseapps.com/v1
(db service, podman compose) (self-hosted: turbo / embed / lite)
Sources registry (Postgres `git_sources`, admin-managed; DB rows win,
BOR_GIT_SOURCES is the empty-table git-only fallback):
git repos → clone/pull via scripts/git_sync.py (the ONLY git-invocation
site; stdlib subprocess) local dirs → walked directly
archives → unpacked under BOR_UPLOAD_DIR (zip-bomb guarded)
```
| Component | Lives in |
|-----------|----------|
| App + middlewares + auth/tokens/rate-limit/theming/caching | `app/main.py`, `app/core/` |
| RAG (retriever, prompts, agent, llm, chunker, importer, summarizer, overview, scaffolding, suggestions, …) | `app/rag/` |
| API routers | `app/api/` |
| Import / sync tooling | `scripts/import_docs.py`, `app/api/sync.py`, `scripts/git_sync.py`, `scripts/eval_retrieval.py`, `scripts/llm_probe.py` |
| Frontend (shell + standalone pages) | `frontend/` (`index.html` shell; `document.html`, `shared.html`, `login.html`, `doc-edit.html`) |
| Migrations (all reversible) | `alembic/versions/` |
---
## 4. Chat Turn & SSE Contract (§3/§4 in older comments)
```
POST /api/chat {message, history?} (auth: require_user)
→ embed(question) [retried, A15 ext.]
→ cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7]
├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
│ persona + <knowledge_base> + <tuning> + full top-2 <documents>
│ + <tools> → agent loop (ls/read/grep; round-capped
│ BOR_AGENT_MAX_ROUNDS=10, 0 = tools off) → turbo streamed
└─ LOW (A8): DEFLECT_MODE prompt (weak titles only, no tools,
byte-identical direct-stream path) → turbo streamed
→ query_log row + per-turn log line (§9)
```
SSE frames (`data: <json>\n\n`): `thinking` (before first delta) →
`tool` (grounded turns, one per model call) → `retry` (pre-first-frame
restarts) → `delta` (answer tokens) → `done` `{deflected, sources[],
suggestions[]}` (terminal). Failure: `error` (terminal — no `done`, no
`query_log` row; a pre-stream DB outage is a plain 503 JSON).
`BOR_STREAM_THINKING=0` suppresses `thinking` frames server-side (chars
still counted). LLM retries (A15 extension): retry count and delay are
configurable (defaults: 3 retries, 5 s flat delay), **only before a
request has streamed its first output frame**.
Retrieval details (A7/A8) and the persona/`<tools>` prompt contract: see
the previous plan revision §6 or `app/rag/retriever.py` /
`app/rag/prompts.py` / `app/rag/agent.py` — the module docstrings carry
the full contracts. **Persona text changes through the plan, not in code
(phase 03 convention); the `DEFLECT_MODE` marker and the mock-LLM markers
(`SUMMARY_MODE`, `KB_OVERVIEW_MODE`, …) may not change without updating
`tests/e2e/mock_llm.py`.**
---
## 5. Data Model & Chunking (§5 in older comments)
Tables (full column detail: `app/models.py` — it is the living doc;
each migration has a tested downgrade):
| Table | Purpose |
|-------|---------|
| `documents` | one row per imported file — full content, `(source, path)` unique, sha256 `content_hash`, `lite` `summary` (non-markdown) |
| `chunks` | retrieval units; `embedding VECTOR(768)`; `position −1` = the embedded summary chunk; stored `tsvector` (GIN) for FTS |
| `query_log` | every question: top score, FTS hits, deflection, sources, latency (threshold-tuning record) |
| `steering_notes` | owner tuning notes → `<tuning>` section of every turn (char-budgeted) |
| `kb_overview` | single row `id=1`: `lite`-generated KB outline → `<knowledge_base>` section (regenerated on KB change) |
| `git_sources` | source registry: `kind` `git`\|`local`, `url`/`path`, `ignore_paths` JSONB |
| `saved_chats` | owner-saved conversations; `messages` = the raw `bor.chat.v1` JSONB; `share_token` (NULL = private, `uuid4` → `/shared/<token>`); `sources_version` (stale when < current) |
| `sources_meta` | single row `id=1`: the KB generation counter — bumped once per KB-changing sync |
| `doc_drafts` | save-as-doc drafts; `token` (uuid4) is the URL credential; `draft` → `pushed` (branch + sha) |
| `api_tokens` | access tokens; only the SHA-256 of the full `bor_…` string is stored; `revoked_at` = dead |
| `ui_settings` | single row `id=1`: Theme tab persistence (3 strings + 17 color variables; NULL = default, B1) |
Single-row tables use `id = 1` (the `kb_overview` precedent).
**Chunking** (format-aware, `app/rag/chunker.py`): markdown on
`##`/`###` headings (paragraph sub-split > `BOR_CHUNK_TARGET_CHARS`=2000,
200 overlap); YAML/JSON on top-level keys; Python on top-level defs via
`ast`; txt on paragraphs; quadlet/j2 plain text. Every format honors the
**1200-char hard cap** (aipi ~1024-token request limit).
**Import workflow** (script and UI Sync share the importer): sha256
delta (unchanged files skip), two-phase upsert (one transaction per
file), `--prune` drops deleted/ignored files, non-markdown files get
`lite` summaries (best-effort/fail-soft), and a KB-changing run
regenerates the `kb_overview` + bumps `sources_meta.version` exactly
once. A failed source aborts the run — no partial junk.
---
## 6. Retrieval & Persona
(See §4's flow and the anchor rows A7/A8. The locked persona, section
order — `<relevance>` → `<knowledge_base>` → `<tuning>` → mode body —
and the `<tools>` teaching live in `app/rag/prompts.py`; the agent loop,
teaching refusals, and scaffolding filter in `app/rag/agent.py` /
`app/rag/scaffolding.py`. Empty prompt sections omit themselves — a
no-notes/no-overview prompt is byte-identical to the pre-steering text.)
Tool surface (harness-aligned): **`ls`** (lists indexed docs
`source: X | path: Y | title: Z`; `path` = a source name — drill-down
tree); **`read`** (combined `source/path` including the source name;
appends the full document — with a truncation cap); **`grep`** (case-
insensitive **fixed substring**, ≤20 `source/path:line: text` matches,
optional one-doc scope; a locator that adds no source/context — never a
regex).
Rejected calls get deterministic teaching refusals and consume a round;
`holder.tool_calls` counts executed calls only.
---
## 7. UI/UX Standards
- **Layout (§7.1):** sticky 64px header + `<main>` + footer; container
`max-width: 72rem` centered; chat is a **centered 46rem column** (2×
= 92rem at ≥1500px desktops — deliberate, not a bug); Sources uses a
**full-width responsive table** — no skinny single-column lists
(lists/tables/grids ≥80–90% of container width); document viewer is a
near-fullscreen same-page modal; mobile ≤640px: hamburger nav,
≥44px touch targets, safe-area composer.
- **Accessibility (§7.2, WCAG 2.1 AA):** semantic landmarks on every
view, labeled controls (icon buttons get `aria-label`),
`aria-live="polite"` stream, `role="status"`/`role="alert"`,
3px `:focus-visible`, `prefers-reduced-motion` respected, text pairs
≥4.5:1 (state is **text + color, never color alone**).
- **Theming (B1–B5):** built-in dark palette in
`app/core/theming.py::BUILTIN_COLORS` (the `:root`-drift unit test
parses built-ins from `styles.css` — no second copy); the admin Theme
tab persists the palette in `ui_settings`, injected pre-paint as
`<style id="bor-theme">` with a matching CSP hash (A20); **B4
byte-identical** when nothing is set.
- **No CDN (§7.3):** zero external `<script>`/`<link>` (integration test
on the index page); markdown rendering is a small local
escape-first function; esbuild minify is build-time only.
- **Never-stale feedback (§7.4):** every control state — idle /
thinking / calling tool / streaming / retrying / done (answer or
deflected) / error / KB-offline / stopped — has a defined UI, every
failure path re-enables its controls, a 300 s pre-token guard
(`TURN_TIMEOUT_MS`) turns a hung stream into the error state (counts
only visible time), and **no auto-follow during a turn**
(viewport moves only on user intent). Pinned by unit tests
on the `app.js` state machine + the story E2E suites.
- Frontend house rules: `app.js` and siblings **never build HTML
strings** (createElement + textContent); asset paths carry
`?v=<token>` (A19).
---
## 8. Debugging
`DEBUGPY` unset/`0` → `debugpy` never imported (unit-tested, A14).
`DEBUGPY=1` → non-blocking listener on `0.0.0.0:${DEBUGPY_PORT:-5678}`;
IDE attaches on demand. Wired at `app/main.py` module import
(`app/core/debugging.py`), so `uvicorn app.main:app`, `python -m
scripts.…`, and tests all honor it.
---
## 9. Observability
- Logs: single-line `timestamp LEVEL logger :: message` on stdout, INFO
default (`BOR_LOG_LEVEL`), third-party loggers capped at WARNING.
- **Per-chat-turn line (required — AGENTS.md rule 10):**
```
question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
kb_chars=N history_msgs=N threshold=… deflected=… sources=…
thinking_chars=… tool_calls=N total_ms=… retries=N scaffold_stripped=N
```
(`sources=` = retrieval + agent-read docs, deduped; `tool_calls=`
executed only; a cancelled turn writes neither line nor `query_log`
row.)
- Importer: per-file `added|updated|unchanged|pruned` + a greppable
cron-safe summary line (`import: summary files=… added=… … formats=…`).
- `query_log` is the durable tuning/gap-finding record.
---
## 10. Testing & Quality Gates (A16 — non-negotiable)
| Gate | Command |
|------|---------|
| Unit + integration | `uv run pytest` |
| Coverage **>90%** on `app/` | `uv run pytest --cov=app --cov-report=term-missing` |
| Story E2E, **in isolation** | `uv run pytest tests/e2e/test_<story>.py -v --no-cov` (DB up: `podman compose up -d db`) |
| Lint + types | `uv run ruff check . && uv run pyright` |
E2E determinism: `tests/e2e/mock_llm.py` is a deterministic
OpenAI-compatible mock (genuine L2-normalized token-overlap embeddings,
so the cosine gate behaves like production); `tests/e2e/slow_llm.py` is
the slow/dead variant; `E2E_REAL_LLM=1` opts into live aipi. Story E2E
fixtures truncate/re-import per module — suites must run in isolation.
Real-model tool-calling verification uses the controlled methodology in
`TOOL_CALLING_TESTING.md` + `scripts/agent_realmodel_check.py`
(fixture KB: `tests/fixtures/test_kb.dump.sql`; the 1,000-doc live-replica
snapshot restores via the `restore-test-db` skill).
---
## 11. Import & Update Workflow
```bash
uv run python -m scripts.import_docs # sync sources + index delta
uv run python -m scripts.import_docs --prune # also drop deleted/ignored
uv run python -m scripts.import_docs --source ~/X # extra directories
# or one click: the admin Sources page "Sync sources" (in-process, 409 while running)
uv run python -m scripts.eval_retrieval "<question>" # rank hybrid results (tuning)
uv run python -m scripts.llm_probe # models + embedding dim sanity
```
The loop for git sources is *commit → re-run*. Uploads:
`POST /api/git-sources/upload` unpacks + registers (202); the scan is
deferred to Sync. Source removal prunes on the next sync.
---
## 12. Current State & Roadmap
- **Shipped phases:** `ls .agents/phases/complete/` — read-only
history; each phase's `00_phase.md` has the full detail. The shipped-
features recap of the previous plan revision and the README cover it.
- **Migrations:** `ls alembic/versions/` — all reversible.
- **Next up (`todo/`):** Already authored via `phase-authoring`.
Read the directory listing in `.agents/phases/todo/` for the ordered
task list; each file's `00_phase.md` has the full description.
- **Next free phase number:** `max(completed phase numbers) + 1` —
computed from `.agents/phases/complete/`.
- **Post-v1 hooks (deliberately not built):** HNSW index at scale;
inotify auto-import; more providers (the OpenAI-compatible client is
the seam); multi-user accounts (the `require_user` split is the seam);
index-backed `grep`.
**Per-phase completion** (AGENTS.md rules 8/9): unit + integration
green, coverage >90%, the phase's Playwright E2E green in isolation, UI
verified against §7, one atomic `--no-gpg-sign` Conventional Commit,
phase dir moved to `complete/`.
---
## 13. House Conventions (quick catch-up)
- **Env:** every setting is a `BOR_`-prefixed env var (or gitignored
`.env`; see `.env.example`); `get_settings()` is `lru_cache`d.
Kill switches follow the `agent_max_rounds` pattern: `0` disables the
feature, a negative value fails startup **loudly** (validator names
the field).
- **Fail loud, never half-configured:** missing admin secrets, bad
extension lists, dim mismatches, unresolvable docs branches — all
refuse to start or refuse the request with a named reason.
- **Shared marker:** `TRUNCATION_MARKER = "[…truncated…]"`
(`app/rag/retriever.py`) is the only overflow marker; char budgets are
the pattern for anything sent to `lite`.
- **Byte-identical contracts are load-bearing:** deflected turns
(A8), empty prompt sections, the B4 theme no-op, and the
rewrite-only caching all have tests that assert byte-identity.
- **git is invoked only in `scripts/git_sync.py`** (A11 — stdlib
subprocess); everything else talks Postgres.
- **One story → one phase → one dedicated Playwright file** (AGENTS.md
rules 4/9); `.agents/` is tracked; only `.agents/phase-sessions/` and
`.agents/pipeline.log` are gitignored.
- **Full-history references:** the previous full plan (all revision
notes, complete API table, column-level data model) is in git —
`git show dac4a3e:.agents/PLAN.md`; per-phase decisions are in
`.agents/phases/complete/*/00_phase.md`; owner test methodologies in
`TOOL_CALLING_TESTING.md`; skills for model testing/KB restore in
`.agents/skills/`.
+92
View File
@@ -0,0 +1,92 @@
# VULNS.md — Known & Historically Found Vulnerabilities
Registry of security findings for Brain of Reese. **Full details, proofs of
concept, and the priority-ordered remediation task list live in
[`.agents/remediation_plan.md`](./remediation_plan.md)** (this file is the
quick-reference index; when a finding is fixed, update its status here and
append to the fixed log at the bottom).
Conventions:
- Statuses: `open` → `mitigated` (control in place) → `fixed` (remediated +
verified) → `accepted` (owner decision, risk consciously kept) →
`false positive` (intended behavior — never re-raise or "fix" without an
explicit owner decision).
- Prior-audit IDs keep their original numbering (SEC-0x = audit 2026-09-07,
SEC-14-xx = audit 2026-09-14).
## Open findings
| ID | Sev | Type (CWE) | Location | Status |
|----|-----|-----------|----------|--------|
| SEC-14-04 | Medium | In-flight SSE chat stream pins a DB connection for the whole turn → pool-exhaustion DoS (~15 streams, one token user) (CWE-770/400) | `app/api/chat.py` (request-scoped `get_db` across the `StreamingResponse`), `app/db.py` (default pool 5+10) | open |
**SEC-14-04 — one-line essence:** 16 slow `curl -N` streams on `/api/chat`
with any token hang every DB-backed route (default pool = 15 connections).
Remediation: short-lived sessions per DB step inside the stream (the
sync/upload house pattern), explicit pool kwargs, optional chat concurrency
cap — see `.agents/remediation_plan.md` §SEC-14-04.
## False positives — 2026-09-14 audit (owner: intended behavior, well known)
Reclassified by the owner on 2026-09-14: every 2026-09-14 finding **except
SEC-14-04** is intended, well-known behavior of a single-owner homelab app —
not a vulnerability to remediate. Kept here for the record so a future audit
(or a future agent) does not re-raise them. Do NOT "fix" these without an
explicit owner decision; the trade-offs below are deliberate.
| ID | Sev (as audited) | Type (CWE) | Location | Why it's intended |
|----|-----|-----------|----------|-------------------|
| SEC-14-01 | Medium | Unthrottled anonymous write surface → storage-exhaustion DoS (CWE-770/307) | `app/api/chats.py` (public POST/PUT `/api/chats`, POST `/{id}/share`); `app/core/rate_limit.py` covers login routes only | Public chat-save is the owner-locked phase-55 A1 design (the visitor saves their own conversation; auto-save is a normal traffic path). Homelab LAN trust model. |
| SEC-14-02 | Medium | Guest IDOR — chat-id holder can force-share / overwrite a private chat (CWE-639) | `app/api/chats.py` `share_chat`, `update_chat` | Owner-locked trust model: the unguessable uuid4 row id IS the credential (phase 51/55). Sharing/overwriting by an id holder is intended. |
| SEC-14-03 | Medium | Postgres on `0.0.0.0:5432` with weak documented creds `reese:reese` (CWE-790/668) | `compose.yaml` (db `ports`, `POSTGRES_PASSWORD`), `app/config.py` default `database_url`, `.env.example` | Deliberate homelab dev convenience; the trusted LAN is the security boundary and the creds are documented on purpose. |
| SEC-14-05 | Low | Cleartext transport + plaintext admin password (CWE-319/256) | `app/main.py` (`https_only=False`), `app/config.py` | Owner decision 2026-08-22: plaintext password and homelab HTTP are deliberate (TLS is a documented non-goal). |
| SEC-14-06 | Low | `debugpy` binds `0.0.0.0:5678` when `DEBUGPY=1` → LAN RCE if left on (CWE-1188) | `app/core/debugging.py` | Explicit opt-in dev feature (AGENTS.md rule 7); the wide bind is intentional for remote IDE attach. |
| SEC-14-07 | Low | No throttle on `/api/chat` for token users — leaked token = unlimited LLM turns (CWE-770) | `app/api/chat.py`, `app/rag/llm.py` | Tokens are capability credentials by design (phase 79); holders are trusted to use the models. |
| SEC-14-08 | Low | Importer reads source files whole, no per-file size cap (git/local sources; uploads are capped) (CWE-770) | `app/rag/importer.py:390` | Source dirs are admin-chosen trusted content; the never-truncated A7 contract implies whole-file reads. |
| SEC-14-09 | Low | Full question text + tool args in stdout logs and `query_log` (CWE-532) | `app/api/chat.py` per-turn line, `app/rag/agent.py` tool line | The full per-turn log line is a locked PLAN §9 requirement (AGENTS.md rule 10: "amply log"). |
| SEC-14-10 | Info | Admin git-source registry reaches arbitrary https/ssh/git@ hosts (admin-gated SSRF) (CWE-918) | `app/api/git_sources.py` `URL_RE` → `scripts/git_sync.py` | Admin-only surface; the admin already controls the host. Arbitrary remotes are the feature (phase 59/35). |
| SEC-14-11 | Info | No `Referrer-Policy` / `Permissions-Policy` (owner decision A4, phase 82) (CWE-538) | `app/core/security_headers.py` | Phase-82 decision A4 deliberately scoped headers to CSP + XFO + nosniff; no external requests exist (No-CDN). |
| SEC-14-12 | Info | `BOR_SESSION_SECRET` only checked non-empty — weak secret allows cookie forgery (CWE-340) | `app/core/auth.py` `ensure_admin_configured` | Operator-provisioned secret (generator command documented in `.env.example`); strength is the operator's choice. |
Note: `.agents/remediation_plan.md` (the audit report) still lists these with
remediation tasks — its task list items P1 (parts), P2, P4, P5, P6, P7
address findings now reclassified as false positives and are **dropped**
unless the owner reopens them. Only task P3 (SEC-14-04) remains actionable.
## Fixed (verified present in the 2026-09-14 audit)
| ID | Sev | What it was | Fixed by | Fixed phase |
|----|-----|-------------|----------|-------------|
| SEC-03 | Medium | No rate limit on failed sign-ins (`/api/login`, `/api/token-auth`) | Per-IP sliding window (10 fails / 15 min, shared counter, fail-open) — `app/core/rate_limit.py` | 81 |
| SEC-04 | Medium | No CSP / X-Frame-Options / nosniff → clickjacking of the admin UI | `SecurityHeadersMiddleware` — CSP `default-src 'self'; base-uri 'none'; frame-ancestors 'none'` + `X-Frame-Options: DENY` + nosniff on every response | 82 |
| SEC-05 | Medium | Unbounded anonymous payload on the public chat-save surface | Schema caps: ≤200 messages, 32 KB text/thinking, nested list caps (~13 MB worst-case) — `app/schemas.py` | 83 |
| SEC-08 | Medium | Docs-push 502 leaked git stderr with `user:pass@` credentials | `app/core/errors.sanitize_error` userinfo masker applied to all error surfaces | 84 |
| SEC-13 | Low | LLM error strings interpolated the raw base URL (possible embedded creds) | Same sanitizer on every LLM error f-string | 84 |
## Controls verified sound (2026-09-14 audit — no finding)
- SQL injection: all ORM / parameter-bound `text()`; FTS tsquery regex-constrained to `[a-z0-9.]` **and** bound as a parameter.
- XSS: escape-first `renderMarkdown()`, `textContent` for all data, strict CSP, no `unsafe-inline` (theme tag uses a sha256-hashed style-src exemption), all `innerHTML` sites audited static-or-escaped.
- Command injection: single list-argv subprocess funnel (`scripts/git_sync.run_git`), no `shell=True`; branch tokens shape-validated at boot; draft paths refuse absolute/`..`.
- Archive upload: traversal / absolute / drive-letter / symlink-hardlink escape / device-FIFO rejection, compressed + extracted byte caps (zip-bomb), atomic swap-in with rollback — `app/rag/archive_upload.py`.
- API tokens: 128-bit CSPRNG, SHA-256 hash stored (plaintext one-shot in the 201 body only), per-request live row check = revocation, generic 401s (no enumeration).
- Open redirect: login `?next=` restricted to same-origin relative paths.
- Agent tools: model-supplied `ls/read/grep` resolve against Postgres rows only (no FS/network), round cap bounds loops, fixed-substring grep (no ReDoS).
- Dependencies: `pip-audit` clean on the locked set (2026-09-14); bandit hits are false positives / dev scripts.
- Git history: no committed secrets across all 161 commits (2026-09-14 scan); `.env` gitignored.
## Re-audit triggers
Re-run the audit (or at least the relevant lens) when any of these land:
a reverse proxy or TLS in front of the app, a multi-user/auth rework,
changes to `app/api/chats.py` write gating, the compose port bindings, the
subprocess/git surface, the upload pipeline, or any dependency with a
published CVE (`uvx pip-audit`).
## Changelog
- 2026-09-14 (owner reclassification) — all 2026-09-14 findings except
SEC-14-04 reclassified as **false positives** (intended behavior, well
known). Open count: 12 → 1.
- 2026-09-14 — second audit: 12 new findings (4 Medium / 4 Low / 4 Info),
5 prior findings verified fixed. Full report: `.agents/remediation_plan.md`.
@@ -1,7 +1,7 @@
# Phase 02 — Story: Import Documents
**Story:** `.agent/user_stories/import-documents.md`
**Context:** `.agent/PLAN.md` §5 (data model), §9 (logging), §11 (import workflow)
**Story:** `.agents/user_stories/import-documents.md`
**Context:** `.agents/PLAN.md` §5 (data model), §9 (logging), §11 (import workflow)
## Goal
The importer (`scripts/import_docs.py`) + `GET /api/docs` + the Sources page
@@ -1,7 +1,7 @@
# Phase 03 — Story: Chat RAG Answer (happy path)
**Story:** `.agent/user_stories/chat-rag-answer.md`
**Context:** `.agent/PLAN.md` §3 (data flow), §4 (SSE contract), §6 (persona), §9 (logging)
**Story:** `.agents/user_stories/chat-rag-answer.md`
**Context:** `.agents/PLAN.md` §3 (data flow), §4 (SSE contract), §6 (persona), §9 (logging)
## Goal
The core product loop: question → embed → cosine top-4 → full top-2
@@ -1,7 +1,7 @@
# Phase 04 — Story: Honest Deflection
**Story:** `.agent/user_stories/honest-deflection.md`
**Context:** `.agent/PLAN.md` §4, §6 (honesty gate), §9
**Story:** `.agents/user_stories/honest-deflection.md`
**Context:** `.agents/PLAN.md` §4, §6 (honesty gate), §9
## Goal
When retrieval finds nothing relevant, Brain says so — plainly, chippily —
@@ -1,7 +1,7 @@
# Phase 05 — Story: Suggestion Chips
**Story:** `.agent/user_stories/suggestion-chips.md`
**Context:** `.agent/PLAN.md` §7 (UI/UX), story file for chip spec
**Story:** `.agents/user_stories/suggestion-chips.md`
**Context:** `.agents/PLAN.md` §7 (UI/UX), story file for chip spec
## Goal
Zero-friction onboarding: 3–4 real example questions on first load,
@@ -1,7 +1,7 @@
# Phase 06 — Story: Loading Feedback & Progress
**Story:** `.agent/user_stories/loading-feedback.md`
**Context:** `.agent/PLAN.md` §7.4 ("never stale" contract), §9
**Story:** `.agents/user_stories/loading-feedback.md`
**Context:** `.agents/PLAN.md` §7.4 ("never stale" contract), §9
## Goal
An unambiguous state machine — `idle → thinking → streaming → done |
@@ -1,7 +1,7 @@
# Phase 07 — Story: Responsive, Polished, Accessible UI
**Story:** `.agent/user_stories/responsive-polish.md`
**Context:** `.agent/PLAN.md` §7 (the whole UI/UX strategy)
**Story:** `.agents/user_stories/responsive-polish.md`
**Context:** `.agents/PLAN.md` §7 (the whole UI/UX strategy)
## Goal
The final visual + accessibility audit pass across chat and Sources. No
@@ -1,7 +1,7 @@
# Phase 08 — Story: Dark Tech Theme
**Story:** `.agent/user_stories/dark-tech-theme.md`
**Context:** `.agent/PLAN.md` §7 (UI/UX strategy), §10 (testing)
**Story:** `.agents/user_stories/dark-tech-theme.md`
**Context:** `.agents/PLAN.md` §7 (UI/UX strategy), §10 (testing)
## Goal
Re-skin the whole UI to a dark, techy, emoji-free look with a subtly
@@ -126,11 +126,11 @@ animate; reduced-motion context → `animation-name: none`; on-topic smoke
- [ ] layout metrics + chat behavior unchanged (smoke E2E)
- [ ] existing story E2E suites still green in isolation
- [ ] unit + integration green, coverage >90%, ruff + pyright green
- [ ] committed (force-add `.agent/PLAN.md` + this phase record — rule 8)
- [ ] committed (force-add `.agents/PLAN.md` + this phase record — rule 8)
## Commit
```bash
git add -A
git add -f .agent/PLAN.md .agent/phases/todo/08_story_dark_tech_theme.md
git add -f .agents/PLAN.md .agents/phases/todo/08_story_dark_tech_theme.md
git commit --no-gpg-sign -m "feat(ui): dark tech theme — emoji-free chrome, subtle animated CSS background, WCAG AA dark palette"
```
@@ -0,0 +1,164 @@
# Phase 09 — Story: Retrieval Quality — Multi-Format Ingestion + Hybrid Search
**Story:** `.agents/user_stories/retrieval-quality.md`
**Context:** `.agents/PLAN.md` §3 (data flow), §5 (data model), §6 (retrieval), §11 (import)
## Goal
Fix "RAG retrieval is terrible": ingest the full text-format set (not
just `.md`), purge vendored-cache junk from the index, and replace
pure-cosine top-4 with hybrid (vector + Postgres FTS, RRF-fused)
retrieval so name-your-tool questions find the right document.
## Owner permission (recorded per phase protocol)
> "I'm giving you explicit permission to update the locked decisions and
> proceed with writing all 3 of these phases" — Reese, 2026-08-21.
This phase revises anchors **A9** (content scope: `*.md` only →
`md, markdown, txt, yaml, yml, json, py` + hidden-dir skip), **A7**
(pure-cosine top-4 → hybrid RRF retrieval; the whole-document context
contract is preserved), **A8** (gate: LOW only when best cosine <
threshold **and** zero FTS hits; threshold re-tuned 0.30 → 0.62 default).
`PLAN.md` anchors were updated 2026-08-21 under this permission.
## Evidence (measured 2026-08-21 against the live KB + `embed` model)
- "How did I install gitlab?": the best `gitlab.md` chunk ranks **7th**
(cosine 0.804) — outside the top-4 window. Ranks 1–6: a vendored-cache
README (`.esphome/.espressif/…/esp-tflite-micro/README.md`, 0.838) and
generic templates (`project_readme_template.md`, `templates/…/foobar.
md`, 0.81–0.82). The LLM therefore answered from junk docs and honestly
reported "no notes on gitlab".
- Corpus cosine range: **0.41–0.84** — the old 0.30 gate never
discriminated.
- FTS: `plainto_tsquery('english','gitlab')` matches **exactly**
gitlab.md's 4 chunks and nothing else.
- ~470 of 672 indexed docs live under dot-prefixed path components
(vendored caches) that A9's exclusion list doesn't cover.
## Dependencies
Phases 01–07 (02 importer, 03 retriever, 04 gate especially).
Independent of 08 (backend + E2E only). Phase 10 builds on the new
multi-format corpus.
## Implementation steps
1. **Config** (`app/config.py`): `import_extensions` (csv, default
`md,markdown,txt,yaml,yml,json,py`; env `BOR_IMPORT_EXTENSIONS`),
`hybrid_vector_candidates` (30, `BOR_HYBRID_VECTOR_CANDIDATES`),
`hybrid_lexical_candidates` (30, `BOR_HYBRID_LEXICAL_CANDIDATES`),
`rrf_k` (60, `BOR_RRF_K`), `relevance_threshold` default **0.62**
(re-tuned; env override stays). In `tests/e2e/conftest.py`'s
`app_server` fixture set `BOR_RELEVANCE_THRESHOLD=0.30` — the mock's
token-overlap embeddings need their own calibration; this keeps
stories 02–07's E2E suites green.
2. **Chunker** (`app/rag/chunker.py`): add a `chunk_document(content,
path)` dispatcher by lowercased suffix + per-format functions —
**stdlib only, no new dependencies**:
- `yaml`/`yml`: split on `---` document separators and top-level keys
(indent-0 `key:` lines); every chunk keeps its key line as anchor.
- `json`: `json.dumps(obj, indent=2)` then split at top-level keys
(track brace depth); unparseable JSON → paragraph packing.
- `py`: stdlib `ast` top-level node line ranges → split at
defs/classes; oversized functions fall back to line packing.
- `txt`: paragraph packing (reuse `_paragraph_blocks`/`_pack_blocks`).
- All formats honor `HARD_MAX_CHARS` (1200 — the aipi ~1024-token
request cap) and the target/overlap settings; the `md` path stays
byte-for-byte unchanged (existing chunker tests must stay green).
3. **Importer** (`app/rag/importer.py`, `scripts/import_docs.py`):
extension filter (case-insensitive, config-driven); **skip any path
containing a dot-prefixed component** (hidden dirs); chunker dispatch
by suffix; `--prune` now also drops docs whose files **no longer
match the filter** (this is how the ~470 junk docs leave the index);
summary log gains per-format counts
(`formats=md:203,yaml:267,…`).
4. **Migration `0002_hybrid_retrieval.py`** (alembic):
- `chunks.tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english',
content)) STORED` + `CREATE INDEX … USING gin (chunks.tsv)`.
- `query_log.fts_hits INT` (nullable; pre-existing rows stay NULL).
5. **Retriever** (`app/rag/retriever.py`) — hybrid path:
- `retrieve(db, question, question_embedding)`: vector top-N (cosine,
as today) ∪ lexical top-N — `to_tsquery('english', <OR-joined
stemmed tokens of the question>)` (skip pure-stopword/no-token
questions → empty lexical list), ordered by `ts_rank` — fused with
RRF: `score = Σ 1/(k + rank)` over the lists a chunk appears in
(single-list chunks get one term; k from config).
- `RetrievedChunk` gains `cosine` (for the gate) and `fts_hit: bool`
alongside `score` (now the fused score, used for ranking);
`select_documents` / `weak_hit_titles` keep working off `score`.
- Deterministic tie-break: `(−fused, −cosine, document.path,
chunk.position)`.
6. **Chat flow + gate** (`app/api/chat.py`): pass the raw question into
`retrieve`; **LOW only when `best_cosine < threshold and fts_hits ==
0`** (`fts_hits` = count of lexical candidates matched); per-turn log
line gains `fts_hits=…` (PLAN §9); `query_log` row stores `fts_hits`.
7. **Eval script** `scripts/eval_retrieval.py`:
`uv run python -m scripts.eval_retrieval "q1" "q2" …` (or
`--from-file questions.txt`) — embeds via aipi, runs the hybrid
search, prints top-5 docs per question with cosine/fts/fused scores +
the gate verdict. Requires `AIPI_KEY` in the environment (same
convention as `llm_probe.py`).
8. **Re-import the live KB** (one-time; expect ~15–40 min of embedding
batches — the importer logs per file):
`uv run python -m scripts.import_docs --prune`. Expect ~470
hidden-dir docs pruned and ~500 docs indexed (md + new formats). Then
verify with the eval script:
- "How did I install gitlab?" → top doc `active/container_gitlab/
gitlab.md` (the compose yaml should land in the top-2).
- "How is my Kubernetes cluster set up?" → kubernetes docs.
- "sourdough starter" → LOW (deflect).
If the gitlab case isn't #1, iterate the **fusion** (k, candidate
counts, token handling) — not the threshold — until it is, and record
the final numbers in the phase report.
9. **E2E fixtures** (`tests/fixtures/docs/`): add
`homelab/container_gitlab/gitlab.md` (H1 "Gitlab", docker install
steps, "gitlab" repeated), `homelab/container_gitlab/
gitlab-compose.yaml` (`services: gitlab: …`), a `.py` note, a `.json`
note, a `.txt` note, and `.hidden/junk.md` (must never be imported).
Follow the existing in-process seeding pattern from
`tests/e2e/test_import_documents.py`.
10. **README**: import workflow section — supported formats, hidden-dir
skip, `scripts/eval_retrieval.py`, threshold tuning; note that the
Sources count drops after the prune (intended cleanup).
## Testing & Quality
- **Unit:** chunker per format (yaml top-level + `---` split, json
top-level keys + pretty-print + unparseable fallback, py ast split +
oversized-func fallback, txt paragraphs, dispatch, 1200-cap) with md
output unchanged; importer (hidden-dir skip, extension filter,
prune-when-filtered-out, per-format summary); retriever (RRF math:
both-lists / one-list / tie-break; OR tsquery construction incl.
no-token and stopword-only questions; gate: `cosine ≥ T` → HIGH;
`cosine < T` + `fts>0` → HIGH; `cosine < T` + `fts=0` → LOW; boundary
exactly `T` → HIGH).
- **Integration:** `/api/chat` hybrid against a seeded temp schema —
keyword question grounded + `fts_hits` in `query_log`; off-topic
deflected with `fts_hits=0`; migration up clean.
- **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` —
**>90%** on `app/`.
- **No regressions:** existing story E2E suites (02–07) green in
isolation after the change (the conftest threshold override is what
keeps them green — verify each one).
## Playwright Execution Phase
Run ONLY this story's suite:
```bash
uv run pytest tests/e2e/test_retrieval_quality.py -v --no-cov
```
Implements the story mapping: multi-format fixture import (hidden doc
excluded, `/api/docs` counts); "How did I install gitlab?" → grounded,
not deflected, gitlab chip, `query_log` row; keyword-only question beats
vector ranking (FTS-OR gate end to end); "sourdough" → deflected bubble +
≥2 chips.
## Success criteria
- [ ] live eval: "How did I install gitlab?" → `gitlab.md` is the top doc
- [ ] zero dot-prefixed path components in `documents` after re-import
- [ ] off-topic still deflects; on-topic still grounds (new + existing E2E)
- [ ] unit + integration green, coverage >90%, ruff + pyright green
- [ ] README documents formats / hidden-dir skip / eval / tuning
- [ ] committed
## Commit
```bash
git add -A && git commit --no-gpg-sign -m "feat(rag): hybrid FTS+vector retrieval and multi-format ingestion — name-your-tool questions find the right document"
```
@@ -0,0 +1,36 @@
# Phase 100 — Every page matches the RAG page's width (the 72rem container, at every viewport)
**Source:** Owner request (chat, 2026-09-12) — "The theme, tuning, and chat pages are still pretty narrow, I want you to match the width of the RAG page for all other pages to keep things consistent."
**Story:** n/a (owner request — supersedes the PLAN §7.1 "centered 46rem chat column (2× = 92rem at ≥1500px)" rule and the phase-58/27/91 form-column caps; the width lineage: `58_wide_desktop_column`, `86_history_page_width`).
**Context:** `frontend/assets/styles.css`: `.container` is the shared 72rem centered column (`width: 100%; max-width: 72rem; padding-inline: 1.25rem;`). The RAG view's `.sources-shell` (and `.git-sources-shell` / `.history-shell` / `.tokens-shell`) carry NO max-width — their content is the full 72rem container: **that is the width the owner wants everywhere**. The narrow pages: `:root { --chat-column: 46rem; }` caps `.chat-shell` (the chat view — the div IS `.container.chat-shell`, the later rule overriding the 72rem cap), `.shared-shell` (shared.html), `.doc-md` (the document viewer's reading column) and `.doc-summary:has(+ .doc-md)` (exactly four selectors use the token); `@media (min-width: 1500px) { :root { --chat-column: 92rem; } .doc-shell { max-width: calc(var(--chat-column) + 2 * 1.25rem); } }` (phase 58 — the wide-desktop doubling + the standalone document page's cap lift); and three hard-coded `max-width: 46rem` form columns: `.tuning-shell`, `.theme-shell` (phase 91), `.doc-edit-shell` (phase 59). Width pins live in: `tests/unit/test_wide_column_css.py` (the phase-58 contract file — 46rem base, the 92rem media block, the four-token count, the three-hard-coded-46rem negative pin, the provenance comments), `tests/unit/test_markdown_tables.py`, `test_shared_page.py`, `test_pinned_composer.py`, `test_doc_edit_screen.py`, `test_remove_confirm_modal.py`, `test_source_ignore_paths.py` (grep `46rem|92rem|chat-column` under `tests/` for the authoritative list — some hits are fixture text, judge per pin), and E2E measured-width suites: `tests/e2e/test_wide_desktop_column.py` (the measured 1472px/736px contract), `test_document_viewer.py`, `test_save_share_ux.py`, `test_sticky_navbar.py`, `test_markdown_tables.py`, `test_responsive_polish.py` (mobile no-overflow).
## Objective
Every view and page renders its content at the RAG page's width — the full 72rem container at every viewport (centered, with the container's 1.25rem gutters): the chat column, the Tuning and Theme form columns, the doc-edit screen, the shared page, and the document viewer all widen to 72rem; the phase-58 ≥1500px 92rem doubling and the 46rem form caps are retired; the mobile (≤640px) and tablet layouts are unchanged (everything was already full-width below the caps).
## Dependencies
- `86_history_page_width` / `58_wide_desktop_column` (complete) — the width lineage being revised (their suites are updated IN THIS PHASE, the phase-97 task-07/08 precedent — asserted behavior preserved, only the width contract changes).
- No code dependencies beyond `frontend/assets/styles.css` (CSS-only — `app/`, `frontend/*.html`, and every JS module are untouched).
## Decisions recorded here (owner instruction 2026-09-12 — PLAN.md is being redone by the owner)
- **D1 — one width for everything:** the reading/form columns ALL ride the 72rem container. `:root { --chat-column: 72rem; }` (the token stays — the four reading-column selectors keep `max-width: var(--chat-column)`, so `.doc-md`/`.shared-shell`/`.chat-shell`/the doc-summary panel follow automatically); the `@media (min-width: 1500px)` block is DELETED entirely (the 92rem doubling AND the `.doc-shell` cap lift — with the token at 72rem the standalone document page's `.container.doc-shell` simply keeps the 72rem cap); the three hard-coded `max-width: 46rem` caps (`.tuning-shell`, `.theme-shell`, `.doc-edit-shell`) are REMOVED (plus their `margin-inline: auto` centering — the `.container` ancestor centers; the shells keep their flex-column + gap rules, matching `.sources-shell` exactly). Net effect: ZERO literal `max-width: 46rem` rules remain in the file (the negative pin flips). This SUPERSEDES PLAN §7.1's 46rem/92rem rule and the phase-58 owner instruction of 2026-08-31 (recorded here per the phase-94/96/97 convention — the owner's new instruction wins; the PLAN.md anchor table gets the owner's sign-off separately).
- **D2 — the document viewer's modal is untouched:** the same-page doc modal's 1100px panel stays its effective ceiling (`.doc-md` at 72rem is wider than the panel's inner width, so the panel binds — no panel resize, no modal change). The standalone `document.html` page rides the 72rem container like every other page.
- **D3 — CSS-only, contracts intact:** no HTML change (the B4 byte-identical theme contract is unaffected — the served HTML is unchanged; only `styles.css` bytes change), no JS change, no endpoint change; A19's `?v=<deploy-token>` versioning busts the old CSS automatically. The `≤640px` mobile block is UNCHANGED (at 360px the container is already 100% — the caps never bound there; the mobile squeeze rules for chat/shared/tokens/theme keep working).
- **D4 — the comment sweep:** every stale "46rem base, 92rem at >=1500px" / "46rem column contract" / phase-58-provenance comment in `styles.css` is updated to the new contract (the owner instruction 2026-09-12 as provenance at the token + the note that the wide-desktop doubling was retired) — the house rule that comments carry the contract (and `test_wide_column_css.py` pins the comment states).
## Tasks
1. `01_width_css.md` — the CSS change: the token to 72rem, the media block deleted, the three form caps removed, the comment sweep
2. `02_width_unit_pins.md` — the unit pin updates: `test_wide_column_css.py` rewritten to the new contract + the grep-sweep of the other unit files
3. `03_e2e_width_sweep.md` — `test_wide_desktop_column.py` rewritten to the measured 72rem contract + the other E2E width pins + the regression sweep + the atomic commit
## Testing & Quality
- Unit: `tests/unit/test_wide_column_css.py` REWRITTEN to the new contract (the pins below); every other unit file whose pins reference the old widths updated (the grep sweep — `46rem|92rem|chat-column` under `tests/unit/`, judged per pin: a width assertion updates, fixture text does not).
- E2E (mandatory, A16): `uv run pytest tests/e2e/test_wide_desktop_column.py -v --no-cov` in isolation — the phase's dedicated width suite, REWRITTEN to the measured 72rem contract (the phase-97 task-07/08 precedent for updating a completed phase's suite when its contract changes); the other measured-width suites updated in place.
- Coverage: **>90%** on `app/` (untouched — the full-suite gate still runs).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] at any viewport ≥ ~1200px, the chat, shared, document-viewer, Tuning, Theme, and doc-edit content columns measure the 72rem container width (E2E-measured, ±4 px) — identical to the RAG page's; below that, everything is full-width as today; 360px has no horizontal overflow
- [ ] `styles.css` has ZERO literal `max-width: 46rem` rules and NO `@media (min-width: 1500px)` block; the four reading-column selectors still cap with `var(--chat-column)` (now 72rem)
- [ ] the B4 byte-identical theme contract holds (no HTML change — the theme no-op test suite green); the mobile squeeze layouts green
- [ ] `uv run pytest` green; coverage >90%; ruff + pyright clean
- [ ] one atomic Conventional Commit, `--no-gpg-sign` (e.g. `feat(ui): match every page to the RAG page's 72rem width`)
@@ -0,0 +1,27 @@
# Task 01 — The CSS: one 72rem column everywhere (token, media-block deletion, form caps, comment sweep)
**Phase:** `100_page_width_consistency` · **Story:** n/a (owner request)
## Objective
`styles.css` carries the new width contract (D1–D4 of `00_phase.md`): `--chat-column` is 72rem, the phase-58 ≥1500px block is gone, the three 46rem form caps are removed, and every stale width comment tells the new story.
## Work
1. `frontend/assets/styles.css`:
- `:root` — `--chat-column: 72rem;` (was 46rem). The token's provenance comment updates: the owner instruction 2026-09-12 ("match the width of the RAG page for all other pages") + the note that it now EQUALS the `.container`'s 72rem cap (the reading columns ride the same width as every other page; the phase-58 2026-08-31 instruction is superseded).
- DELETE the entire `@media (min-width: 1500px) { … }` block at the bottom of the file (the `:root { --chat-column: 92rem; }` doubling + the `.doc-shell { max-width: calc(var(--chat-column) + 2 * 1.25rem); }` lift — D1). Its phase-58 block comment goes with it; the deletion is total (the `prefers-reduced-motion` block that follows it stays).
- `.tuning-shell` — remove `max-width: 46rem;` and `margin-inline: auto;` (keep `display: flex; flex-direction: column; gap: 1.25rem; flex: 1;` — the shell now matches `.sources-shell` exactly); update its phase-27 comment (the "Same width discipline as the chat column — a centered, capped column" wording) to the full-container-width contract (owner instruction 2026-09-12).
- `.theme-shell` — the same removal (phase-91's "form column" rationale is superseded — the owner wants it wide); comment updated.
- `.doc-edit-shell` — the same removal (phase-59's form-column cap is superseded); comment updated.
- The comment sweep (D4): every remaining reference to "46rem base", "92rem at >=1500px", "the 46rem chat-column width", "the >=1500px 92rem override", and the stale `≤640rem`/`≤640px`-era "46rem column contract" wording in the reading-column regions (`.chat-shell`'s block comment, `.shared-shell`'s inline comment + the shared-page region comment, `.doc-md`'s comment, the doc-summary panel comment, the `≤640px` shared-page note "the shell keeps its base 46rem column (the >=1500px 92rem override never applies here…)", and any others the grep `46rem|92rem|1500px` finds) — rewrite each to the new contract (72rem at every width; the wide-desktop doubling retired 2026-09-12). The `.shared-shell` comment must keep the "the PLAN §7 centered chat column" lineage sentence only if it still reads true — it does NOT (PLAN §7.1's 46rem rule is superseded): reword to "the shared page reads exactly like the chat page — both at the 72rem container width".
- Do NOT touch: the `.container` rule itself, the `≤640px` mobile block (the squeeze rules), the doc modal's 1100px panel, any non-width CSS, any `app/` or HTML/JS file (D3 — CSS-only).
2. Verify with a grep: `grep -n "46rem\|92rem\|1500px" frontend/assets/styles.css` returns NOTHING (or only the new provenance comment's mention of the retired 2026-08-31 instruction, if phrased without the literal values — prefer phrasing it without the literals so the grep is clean).
## Testing & Quality
- Unit: the pin updates land in task 02 — this task's own check is the grep verification above + a visual sanity pass (`uv run uvicorn app.main:app --reload`, compare the chat/tuning/theme pages to the RAG page at 1280px and 1920px — the content columns match; the doc modal looks unchanged).
- Coverage: **>90%** on `app/` (untouched — the full-suite gate runs at the phase's end).
## Completion Criteria
- [ ] `styles.css`: `--chat-column: 72rem` in `:root`; no `@media (min-width: 1500px)` block; zero literal `max-width: 46rem` rules; the three shells (tuning/theme/doc-edit) structurally match `.sources-shell` (flex column, no cap)
- [ ] the width grep is clean (task 02's pins will enforce the comment states — do not leave a stale "46rem base / 92rem" claim anywhere)
- [ ] no HTML/JS/app change (D3)
- [ ] the full unit suite is green EXCEPT the width pins task 02 rewrites (run it to enumerate the breakage — the list must be exactly the files named in task 02)
@@ -0,0 +1,27 @@
# Task 02 — The unit pins: `test_wide_column_css.py` rewritten + the grep-sweep of the other width pins
**Phase:** `100_page_width_consistency` · **Story:** n/a (owner request)
## Objective
The unit layer pins the NEW width contract — `test_wide_column_css.py` becomes the phase-100 contract file, and every other unit pin that asserted the old widths is updated (fixture text that merely mentions a width is left alone).
## Work
1. `tests/unit/test_wide_column_css.py` — REWRITE (keep the file's balanced-brace `_rule_block` helpers + its module purpose — "pin the styles.css markers the width contract depends on"):
- **The token:** `:root` declares `--chat-column: 72rem` with the provenance comment citing the owner instruction 2026-09-12 (replaces the 2026-08-31 pin).
- **The retired doubling:** NO `@media (min-width: 1500px)` block anywhere in the file (`css.count("@media (min-width: 1500px)") == 0`), and no `--chat-column: 92rem` assignment (`css.count("--chat-column: 92rem") == 0`).
- **The four reading columns:** `.chat-shell`, `.shared-shell`, `.doc-md`, `.doc-summary:has(+ .doc-md)` each cap with `max-width: var(--chat-column)` — and exactly those four rules use the token (`css.count("max-width: var(--chat-column)") == 4` — the token count pin survives, the VALUE changes).
- **The flipped negative pin:** ZERO literal `max-width: 46rem` rules remain (`css.count("max-width: 46rem") == 0`) — the three former form columns (`.tuning-shell`, `.theme-shell`, `.doc-edit-shell`) must NOT carry a max-width at all (each `_rule_block` free of `max-width`), and they carry no `margin-inline: auto` (the `.container` centers).
- **The comment pins:** the stale claims are gone (`"≤46rem" not in css`, `"92rem at >=1500px" not in css`, `"46rem base" not in css`); the provenance appears at the token (owner instruction 2026-09-12); the chat-shell / shared-shell comments name the 72rem-everywhere contract (pin the key phrases, not whole paragraphs — the house "pin the contract words" style).
- Update the module docstring to describe the new pins.
2. The grep sweep — `grep -rln "46rem\|92rem\|chat-column" tests/unit/` (the authoritative list; `test_wide_column_css.py` is done above). For each hit, judge per pin: a pin that ASSERTS the old width (a literal in an `assert`, a `_rule_block` expectation, a counted occurrence) is updated to the new contract (72rem / no cap / the new comment wording); a hit that is fixture text, a docstring example, or an unrelated measurement (e.g. a fixture document mentioning "46rem", or a pin about an UNRELATED rule's max-width) is left untouched. Expected candidates (from the phase-100 `00_phase.md` context): `test_markdown_tables.py`, `test_shared_page.py`, `test_pinned_composer.py`, `test_doc_edit_screen.py`, `test_remove_confirm_modal.py`, `test_source_ignore_paths.py` — the actual set is whatever the grep + the run reveal.
3. Run `uv run pytest tests/unit/ -q` — green.
## Testing & Quality
- Unit: the rewritten `test_wide_column_css.py` + the swept pins ARE this task's test layer (they guard the CSS bytes — the house source-pin pattern).
- Coverage: **>90%** on `app/` (untouched — the full-suite gate runs at the phase's end).
## Completion Criteria
- [ ] `test_wide_column_css.py` pins the phase-100 contract (72rem token + 2026-09-12 provenance, no 1500px block, no 92rem, the four token selectors, ZERO literal 46rem rules, the three shells cap-free, the comment states)
- [ ] the grep sweep is complete — every remaining `46rem|92rem|chat-column` reference under `tests/unit/` is either the new contract's pin or non-width fixture text (list the judgment in the commit's test-run notes / the final summary)
- [ ] `uv run pytest tests/unit/ -q` green
- [ ] no behavior change in completed work (the non-width pins in the swept files still pass unchanged)
@@ -0,0 +1,31 @@
# Task 03 — The E2E width sweep: measured 72rem everywhere + updated suites + regressions + commit
**Phase:** `100_page_width_consistency` · **Story:** n/a (owner request)
## Objective
The browser proves the owner's ask: at every desktop viewport the chat, shared, document-viewer, Tuning, Theme, and doc-edit content columns measure the SAME width as the RAG page (the 72rem container) — the phase's dedicated width suite is rewritten to that contract and the other measured-width suites are updated in place.
## Work
1. `tests/e2e/test_wide_desktop_column.py` — REWRITE (the phase-58 measured-width suite becomes the phase-100 one — the phase-97 task-07/08 precedent: a completed phase's suite updates in place when its contract changes; keep its seeding helpers — the fixture markdown doc for the viewer, the shared-chat helper):
- **`test_all_columns_match_the_rag_page`** (the core pin): viewport 1280×800 — measure the `.chat-shell` bounding-box width on `/`, the `.tuning-shell` width on `/tuning.html` (admin), the `.theme-shell` width on `/theme.html` (admin), and the RAG page's `.sources-shell` width on `/sources.html` (admin): all four EQUAL (± 4 px) — the owner's "match the width of the RAG page" as one assertion. Then viewport 1920×1080: the same four still EQUAL each other (± 4 px) and each ≈ 1152 px (72rem at the 16px root, ± 4 px — the box includes the container's 2×1.25rem padding, border-box).
- **`test_reader_columns_wide`** (the token's consumers): viewport 1920×1080 — `.doc-md` in the opened fixture doc ≈ 1058 px (the 1100px border-box modal panel minus its 1px borders and `.doc-modal-content`'s 2×1.25rem padding is the binding ceiling there — 1100 − 2 − 40; D2: pin that the modal is UNCHANGED by asserting the modal panel's own width is still ~1100 px, not that `.doc-md` is 1152); `/shared/<token>`'s `.shared-shell` ≈ 1152 px; the standalone `document.html?source=…&path=…` page's `.doc-md` ≈ 1112 px (its `.container.doc-shell` is the 72rem container → inner content = 1152 − 40 px padding).
- **`test_narrow_unchanged`** (the no-regression leg): viewports 360×800 and 900×600 — `.chat-shell` width == viewport width (100%, no overflow: `document.documentElement.scrollWidth <= viewport width` — the `test_responsive_polish.py` assertion style), and at 900px the chat/tuning/theme shells are all 900px wide (equal to each other — the cap never bound below 72rem anyway).
2. The other measured-width E2E suites — update in place (each in isolation, `--no-cov`, DB up): `test_document_viewer.py` (its `≤ 736 px` standalone `.doc-md` pin → the ≈ 1112 px contract; keep its non-width assertions byte-identical), `test_save_share_ux.py` (any 92rem/46rem measured or source-pinned width references), `test_sticky_navbar.py` (its width-related pins only), `test_markdown_tables.py` (the e2e table-width pins against the reading column), `test_responsive_polish.py` (the mobile pins — expected UNCHANGED; run to confirm). Judge per assertion: a width measurement updates, a layout/behavior assertion never does.
3. The theme no-op contract (D3): run `test_ui_customization.py` + `test_admin_theme_tab.py` in isolation — no HTML changed, so the B4 byte-identical pins must pass UNCHANGED (if one fails, the CSS change touched something it should not — fix the cause, not the pin).
4. Regression sweep (each in isolation, `--no-cov`): the five suites from step 2 (updated) + `test_chat_rag.py` (the chat layout smoke) + `test_theme_semantic_completion.py` (the phase-93 page-head panels at the new width).
5. Full gates + commit: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean; move `100_page_width_consistency` → `.agents/phases/complete/`; one atomic commit:
```bash
git add -A .agents/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(ui): match every page to the RAG page's 72rem width"
```
## Testing & Quality
- E2E (mandatory, A16): `uv run pytest tests/e2e/test_wide_desktop_column.py -v --no-cov` green in isolation — the phase's dedicated suite.
- Coverage: **>90%** on `app/` (the full-suite gate).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] the rewritten width suite passes in isolation: chat == tuning == theme == RAG at 1280 AND 1920 (± 4 px, ≈ 1152 px at 1920); the doc modal unchanged (~1100 px); shared + standalone document pages ≈ 1152/1112 px; 360px overflow-free
- [ ] the other measured-width suites pass in isolation (updated in place); the theme no-op suites pass UNCHANGED
- [ ] the regression sweep passes in isolation
- [ ] full suite green, coverage >90%, ruff + pyright clean
- [ ] phase dir moved to `complete/`, one atomic `--no-gpg-sign` Conventional Commit
@@ -0,0 +1,78 @@
# Phase 101 — Tokens page: active/revoked table split, per-table search, required name, one-click regenerate
**Source:** Owner request (chat, 2026-09-12) — "The tokens page should move revoked tokens to a separate table below the active ones. Generating a token should not be possible without giving it a name. Both active and revoked token tables should be searchable. I should be able to regenerate active tokens with the click of a button."
**Story:** n/a (owner request — extends `79_api_tokens`: the model/service/API on phase 79 tasks 01–03, the single-table admin view on task 06, the once-block + two-step revoke UI, the E2E `test_api_tokens.py`).
**Context:** `api_tokens` (migration 0012): `id` UUID, `label` (1–120, NOT unique), `token_hash` (sha256 of the full `bor_…` string, unique), `created_at`, `last_used_at`, `revoked_at` (NULL = active). `app/core/tokens.py` service: `generate_token`, `hash_token`, `create_token(db, label) -> (row, plaintext)`, `find_active_by_token`, `mark_used`, `revoke(db, token_id) -> bool` (stamps `revoked_at` only when unset; False when the row is missing). Admin API (`app/api/tokens.py`, router-wide `require_admin`): `POST /api/tokens` (201 `TokenCreated{id, label, token, created_at}` — the ONLY response carrying the plaintext, owner-locked A4), `GET /api/tokens` (200 `TokenList{tokens: [TokenListItem{id, label, created_at, last_used_at, revoked: bool}]}` newest-first — `revoked_at` is NOT returned today), `POST /api/tokens/{id}/revoke` (204 idempotent, 404 unknown). The view (`frontend/index.html` `#view-tokens` + `frontend/assets/tokens.js`, phase-76 fold): the create row (`#token-label` + `#token-generate` — a BLANK label currently sends the fallback `"token"`), the shown-once block (`#token-once` — plaintext in the read-only field's value ONLY, hidden + wiped on the next `loadTokens()` / re-show), ONE full-width table (`#tokens-table` / `#tokens-tbody`, columns Label | Created | Last used | Status (active em-dash vs the rose `.stale-pill` Revoked) | Actions (the two-step inline Revoke — the `history-confirm-*` pattern; revoked rows carry no action)), `#tokens-empty-row`, the `#tokens-status` live region, the `bor:view-refresh` re-show contract (re-entrant `loadTokens()`), the whoami gate (anonymous → the gate, NO fetch). The phase-79 E2E `tests/e2e/test_api_tokens.py` pins the single-table layout — including REVOKED rows living in `#tokens-tbody` (e.g. `test_revocation_closes_the_door` asserts the `.stale-pill` there) — and `auth_helpers.login_with_token(page, app_url, token)` drives the real gate (the regenerate proof reuses it).
## Objective
The Tokens view splits its list into an ACTIVE table and a REVOKED table below it (each independently searchable), a token can no longer be generated without a name, and every active token carries a one-click **Regenerate** (a house two-step confirm, then the new plaintext shown once) that rotates the credential — the old token is revoked and lands in the revoked table, the new one (same label) takes its place in the active table.
## Dependencies
- `79_api_tokens` (complete) — the model, the service, the admin API, the A4 plaintext-once contract, the view module, the E2E helper (`login_with_token`).
- Queue order only: `98_sync_summary_visibility`, `99_kb_tree_table_and_back_nav`, `100_page_width_consistency` (todo) — no code dependency (different views; the tokens table sits in the full-width `.container` already — phase 100 changes nothing here).
## Decisions recorded here (owner review — PLAN.md is being redone by the owner)
- **D1 — the split + the column contracts:** the active table keeps the phase-79 shape MINUS the Status column (a table that is all-active needs no status) — **Label | Created | Last used | Actions** (Actions visually-hidden header, as today). The revoked table (below, its own `.table-wrap` + a visible `Revoked tokens` heading + its own search input) is **Label | Created | Last used | Revoked** (the `revoked_at` date — locale date+time, full ISO on hover, the house `tokens-date-cell` language). The table IS the status — the em-dash / `.stale-pill` column disappears from BOTH tables (the pill's meaning is now the table's position). The revoked section (heading + search + table) is HIDDEN when there are no revoked tokens (an empty table is noise); the active table keeps its empty-state row. Newest-first in both (the server order).
- **D2 — regenerate = rotation, atomic:** new endpoint `POST /api/tokens/{token_id}/regenerate` (admin, 201 `TokenCreated`): in ONE transaction — stamp `revoked_at` on the old row (the `revoke` service primitive) and `create_token(db, old.label)` (same label — the hand-out name persists) — returning the NEW row + the new plaintext exactly once (A4 preserved: the plaintext is per-token, shown once, never re-shown; the old plaintext was already gone). Unknown id → 404 `token not found`; the row is ALREADY revoked → 409 `token already revoked` (a dead token cannot be rotated — the button never offers it, the API stays honest). UI: the active row's Actions cell gains **Regenerate** (before Revoke) — the house two-step inline confirm (first click swaps to `Regenerate? The current token is revoked. [Yes] [No]`, focus to Yes — a destructive rotation deserves the same confirm weight as Revoke; ONE button starts it, per the owner's "click of a button"). On 201: the re-entrant `loadTokens()` runs (the old row moves to the revoked table, the new row lands in the active one), the once-block reveals with the new plaintext + Copy, the live region reads `Regenerated "<label>" — copy the new token now; it won't be shown again.` Failure (409 → the row re-renders in the revoked table + a line; other → neutral retry copy, the button restores — the phase-55 convention).
- **D3 — a name is required:** `generateToken()` trims the label and, when empty, does NOT send — the live region reads `Give the token a name first.`, the label input re-focuses, the Generate button's §7.4 lifecycle is untouched (it was never disabled — the request simply doesn't happen). The `|| "token"` fallback is DELETED (the create row's comment + placeholder update: the name is the hand-out identity, not an optional decoration). The server's 422 on blank/over-long labels stands unchanged (defense in depth — the UI just stops volunteering the fallback).
- **D4 — search = client-side, per table, label-scoped:** one `type="search"` input per table (`#token-search-active` above the active table, `#token-search-revoked` above the revoked table — inside the revoked section, so it hides with it), placeholder `Search active tokens…` / `Search revoked tokens…` (labeled via `aria-label` — the house visually-hidden-label language for single-purpose inputs). Filtering: case-insensitive SUBSTRING over the label, applied on the `input` event to the CURRENTLY rendered rows (no fetch — toggle each row's `hidden`), and RE-APPLIED after every `loadTokens()` (the query state survives re-renders / re-shows — the phase-77 contract). Zero visible matches with a non-empty query → a per-table no-match row (`No tokens match "<query>".` — the query text is textContent-filled; distinct from the active table's true empty-state row `No tokens yet — …`). Empty query → all rows visible, no-match row hidden.
- **D5 — the list API gains `revoked_at`:** `TokenListItem` adds `revoked_at: datetime | None = None` (wire-additive — null for active tokens; the revoked table's Revoked column needs the actual timestamp, and nothing else changes on the wire). The existing `revoked: bool` stays (the client's table split key).
## Design (shared by all tasks — the executor reads this, not the chat)
### The API (task 01)
- `app/core/tokens.py` — `regenerate_token(db, token_id) -> tuple[ApiToken, str] | None` (the service surface, unit-testable without FastAPI): fetch the row (`db.get`) — missing → return `None`; `revoked_at` set → raise `TokenAlreadyRevoked` (a new module-level exception, one sentence docstring: a dead token cannot be rotated); else `revoke(db, row.id)` (stamp) + `create_token(db, row.label)` (new row + plaintext) + `db.flush()` (the caller commits — the `create_token` convention: the service flushes, the endpoint owns the commit). ONE commit covers both writes (the rotation is atomic — a create failure rolls the revoke back with it).
- `app/api/tokens.py` — `POST /{token_id}/regenerate` (201, `response_model=TokenCreated`, the router-wide `require_admin` covers it): `None` → 404 `token not found` (the revoke endpoint's exact message); `TokenAlreadyRevoked` → 409 `token already revoked`; success → commit + refresh + `TokenCreated(id, label, token, created_at)` of the NEW row. The list endpoint passes `revoked_at=row.revoked_at` into `TokenListItem` (D5). Docstrings: the rotate-once semantics (the docstring-carries-the-contract rule).
- `app/schemas.py` — `TokenListItem.revoked_at: datetime | None = None` (the docstring notes D5).
- Unit (`tests/unit/test_tokens.py`): the `regenerate_token` matrix — active row → (new row, new plaintext) with the SAME label, the old row stamped (its original `revoked_at` semantics: first stamp), the new plaintext well-formed (`bor_` + 32 hex, ≠ the old hash); missing id → None; revoked id → raises; the new row's hash is the sha256 of the new plaintext (the lookup round-trips via `find_active_by_token`); the old token no longer authenticates (`find_active_by_token(old_plaintext)` → None).
- Integration (`tests/integration/test_tokens_api.py`): the endpoint matrix — 201 (the 201 body carries the NEW plaintext exactly once; the old row `revoked: true` + `revoked_at` non-null in the next list; the new row active, same label, newer `created_at`); 404 unknown id; 409 already-revoked; 403 anonymous + 403 token-user (the router-wide gate — the existing pattern); the list shape now carries `revoked_at` (null active / timestamp revoked, ISO-8601).
### The view (tasks 02 + 03)
`frontend/index.html` `#view-tokens` (the static skeleton — tokens.js fills the tbodys):
- The create row: the input's `aria-label` → `Token name`, the placeholder → `e.g. alice — required` (D3); the comment updated (no fallback — the name is required).
- A search row above the active table: `<input type="search" id="token-search-active" aria-label="Search active tokens" placeholder="Search active tokens…">` (the house input language — a new `.token-search` class in the CSS, full width, the `#tune-note`/archive-upload input styling family).
- The active table: thead drops the Status `<th>` (four columns now: Label | Created | Last used | the visually-hidden Actions); `#tokens-empty-row` → `colspan="4"`, copy unchanged; a NEW hidden `#tokens-no-match-row` (`colspan="4"`, its text JS-filled — ships empty).
- A NEW revoked section BELOW the active table's wrap (ships `hidden`): `<h2 id="tokens-revoked-heading" class="tokens-revoked-heading">Revoked tokens</h2>` (a visible sub-heading — house typography: `font-family: var(--mono); font-size: 1rem; color: var(--brand-ink);` on the surface-less page background, AA pair, matching the phase-97 `.kb-level h2` voice) + the search input `#token-search-revoked` + `<div class="table-wrap tokens-table-wrap" id="tokens-revoked-wrap" role="region" aria-label="Revoked tokens" tabindex="0">` + `<table class="tokens-table" id="tokens-revoked-table">` (visually-hidden caption `Revoked tokens — newest first`; thead Label | Created | Last used | Revoked; `tbody#tokens-revoked-tbody` + the hidden `#tokens-revoked-no-match-row` `colspan="4"`).
- `frontend/assets/tokens.js` (task 02 — the split + search + required name):
- `loadTokens()` — the fetched list splits: `revoked === false` → `#tokens-tbody`, `true` → `#tokens-revoked-tbody` (the server's newest-first order kept per table); the revoked SECTION (`#tokens-revoked-heading` + search input + wrap) is shown iff the revoked table has ≥ 1 row; the active empty-state row logic stands (0 active rows → the empty row — a 0-row fetch with revoked tokens shows the empty active table + the populated revoked section: honest states); AFTER rendering, both search filters re-apply (D4 — the queries persist in module state `activeQuery` / `revokedQuery`, initialized `""`, never reset by a load).
- `makeRow(tok, { table })` — the active variant: Label | Created | Last used | Actions (Regenerate — task 03 — + Revoke, the existing two-step, unchanged); the revoked variant: Label | Created | Last used | Revoked (the `revoked_at` locale date+time, full ISO in `title` — the `tokens-date-cell` class) + NO actions. The `title` hover attributes carry over (label, created, last-used, revoked).
- The search: per-table `applyFilter(tbody, noMatchRow, query, tableLabel)` — case-insensitive `label.toLowerCase().includes(query.toLowerCase())` over the data rows (the no-match/empty rows excluded); rows toggle `hidden`; `noMatchRow` visible ⟺ query non-empty AND zero data rows visible (its `<td>` textContent = `No tokens match "<query>".` — the query inside the quotes is textContent, never HTML). The `input` listeners set the module query + apply (no fetch). The active search input is revealed with the create row (admin branch); the revoked one with the revoked section.
- `generateToken()` (D3): `const label = (labelInput ? labelInput.value : "").trim();` — `if (!label) { announce("Give the token a name first."); if (labelInput) labelInput.focus(); return; }` — the `|| "token"` fallback deleted; the rest of the function (the 201 once-block reveal, the list re-load FIRST, the label clear, the error lines) unchanged.
- `frontend/assets/tokens.js` (task 03 — regenerate): `makeRegenerateControl(tok, row)` mirroring `makeRevokeControl` (the `history-confirm-*` swap, focus to Yes) — Yes → `POST /api/tokens/<id>/regenerate` → on 201: `loadTokens()` (the re-entrant load — the old row relocates, the new row lands) → the once-block reveal (`onceValue.value = created.token`, `onceBlock.hidden = false` — the same once-block, the A4 value-only contract) → `announce("Regenerated \"<label>\" — copy the new token now; it won't be shown again.")`; 404 → the row re-renders revoked-in-place + `That token was already revoked.` (the revoke control's existing 404 handling, mirrored); 409 → the row relocates via `loadTokens()` + the same line; other failure/network → neutral retry copy + the confirm restores (the `restoreRegenerate` pattern — focus returns). The active Actions cell order: **Regenerate, then Revoke** (the rotation is the primary lifecycle action).
- `frontend/assets/styles.css` — `.token-search` (the input row: `width: 100%;` + the house input surface — reuse the `#tune-note` font/padding/border family, `:focus-visible` via the global rule, ≥ 44px target height), `.tokens-revoked-heading` (the sub-heading, the AA pair noted above), the no-match rows' `<td>` (the `tokens-empty-row` styling family — `color: var(--ink-soft)`), the Regenerate button (the `.token-revoke` family's neutral sibling — or a new `.token-regenerate` class styled identically to `.token-revoke` minus the hover-error: keep it simple — REUSE `.token-revoke`'s classes for the confirm machinery, one extra class for the label if needed; NO new hue — the phase-92 monochrome invariant). The `≤640px` block: the search inputs + the two tables squeeze as today (verify the `tokens-table-wrap` mobile rules cover the second table — they are class-based, so they do).
- Source pins: extend the existing tokens.js full-file source-pin unit file (the phase-79 task-06 pin — locate it under `tests/unit/`, likely `test_tokens_ui.py` or the `test_api_tokens_model.py` sibling; if no dedicated file exists, CREATE `tests/unit/test_tokens_ui.py`): the two tbodys + the section show/hide, the split in `loadTokens`, the filter logic (the case-insensitivity, the re-apply-after-load, the no-match row text), the required-name block (the exact announce copy + NO `|| "token"` anywhere in the file), the regenerate wiring (the POST path, the once-block reveal, the 404/409 branches, the confirm copy) + the styles.css class pins.
### The E2E (task 04)
`tests/e2e/test_tokens_page.py` (new; `app_server` + `mock_llm` + `db_ready`; admin via `auth_helpers.login`; a token-user context via a FRESH browser context + `auth_helpers.login_with_token` — the phase-79 pattern; the e2e-`-labeled` token cleanup autouse fixture from `test_api_tokens.py`):
1. **`test_revoked_tokens_move_to_their_own_table`** — create two tokens (the UI create row, labels `e2e-act` / `e2e-rev`); Revoke `e2e-rev` (the two-step): the active table shows ONLY `e2e-act` (the active thead has NO Status column — four `<th>`); the revoked section is visible BELOW the active table: its heading `Revoked tokens`, the `e2e-rev` row with a Revoked date cell (non-empty, ≠ created) and NO action buttons.
2. **`test_both_tables_are_searchable`** — with ≥ 2 active tokens (`e2e-a1` / `e2e-a2` — labels chosen so one is a substring of the other's neighbor: type `e2e-a1` → only that row visible; type `zzz` → the no-match row reads `No tokens match "zzz".`; clear → both rows back). Same on the revoked search (with the revoked row from test 1's flow or a fresh revoke).
3. **`test_a_token_cannot_be_generated_without_a_name`** — clear the label, click Generate: NO once-block, the live region reads `Give the token a name first.`, the active table's row count is unchanged (and `GET /api/tokens` via the admin cookie shows no new row — the request never happened).
4. **`test_regenerate_rotates_the_token`** — create `e2e-rot`; Regenerate → Yes: the once-block shows a NEW plaintext (≠ the original, `bor_` + 32 hex); the active table has the `e2e-rot` row (the NEW one — its created timestamp is newer); the revoked table now holds the ORIGINAL `e2e-rot` (same label, Revoked date set); the live region carries the D2 line; a fresh token-user context signs in with the NEW token (the gate closes → the app opens — `login_with_token`), and the ORIGINAL token is refused at the gate (the `#auth-gate-error` alert — it was revoked by the rotation).
5. Update `tests/e2e/test_api_tokens.py` in place (the phase-97 task-07/08 precedent): the revoked-row assertions relocate to `#tokens-revoked-tbody` (`test_admin_generates_token_in_ui`'s post-revoke state, `test_revocation_closes_the_door`'s `.stale-pill` → the revoked table's row + its Revoked date cell); the Status-column assertions (if any) drop; every non-layout assertion (the auth flows, the cached token, the 403 walls, the wrong-token error) stays byte-identical.
6. Regression sweep (each in isolation, `--no-cov`, DB up): the updated `test_api_tokens.py` + `test_admin_auth.py` (the gate contract) + `test_shared_header.py` (the header/whoami plumbing — expected unchanged).
7. Full gates + commit: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean; move `101_tokens_page_overhaul` → `.agents/phases/complete/`; one atomic commit:
```bash
git add -A .agents/ app/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(tokens): split active and revoked tables with search, require a name, and add one-click regenerate"
```
## Tasks
1. `01_regenerate_api.md` — the service `regenerate_token` + `POST /api/tokens/{id}/regenerate` (rotation, atomic) + `TokenListItem.revoked_at` + unit/integration
2. `02_two_tables_search_required_name.md` — the view skeleton (two tables + the revoked section + the search inputs) + the client split/filter + the required-name block
3. `03_regenerate_ui.md` — the Regenerate control (two-step confirm, the once-block reveal, the 404/409 branches) + the CSS + the source pins
4. `04_e2e_tokens_page.md` — `tests/e2e/test_tokens_page.py` + the `test_api_tokens.py` update + the regression sweep + the atomic commit
## Testing & Quality
- Unit: the `regenerate_token` matrix (`tests/unit/test_tokens.py`); the source pins (task 03's file) for the view logic.
- Integration: the endpoint matrix + the list shape (`tests/integration/test_tokens_api.py`); the existing phase-79 auth-enforcement pins stay green (the new route rides the router-wide gate).
- E2E (mandatory, A16): `uv run pytest tests/e2e/test_tokens_page.py -v --no-cov` in isolation; the updated `test_api_tokens.py` + `test_admin_auth.py` + `test_shared_header.py` green in isolation.
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] the Tokens view (admin) shows the active tokens in the top table (Label | Created | Last used | Actions) and the revoked ones in a separate table below (Label | Created | Last used | Revoked) — the section hidden while no token is revoked; both tables filter live by label with a per-table no-match row that survives re-renders
- [ ] a blank/whitespace name generates NOTHING (the live-region line, no request, no row); the `token` fallback is gone
- [ ] Regenerate (one button + the house confirm) rotates: the old token is revoked (lands in the revoked table, refuses new sign-ins immediately) and the new token (same label) is active with its plaintext shown exactly once; 404/409 are pinned at the API level
- [ ] the A4 contract holds end to end (the list never carries plaintext/hashes; the once-block re-shows nothing — the phase-79 pins green)
- [ ] `uv run pytest` green; coverage >90%; ruff + pyright clean
- [ ] `uv run pytest tests/e2e/test_tokens_page.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`); the regression suites green in isolation
- [ ] one atomic Conventional Commit, `--no-gpg-sign`
@@ -0,0 +1,35 @@
# Task 01 — `POST /api/tokens/{id}/regenerate`: the atomic rotation + `TokenListItem.revoked_at`
**Phase:** `101_tokens_page_overhaul` · **Story:** n/a (owner request)
## Objective
The admin can rotate an active token server-side: one atomic call revokes the old row and creates its successor (same label, new plaintext returned exactly once) — plus the list API exposes `revoked_at` so the revoked table can show the revocation date.
## Work
1. `app/core/tokens.py`:
- `class TokenAlreadyRevoked(Exception)` — module-level, one-sentence docstring (a dead token cannot be rotated — the 409's reason).
- `regenerate_token(db, token_id: uuid.UUID) -> tuple[ApiToken, str] | None`:
- `row = db.get(ApiToken, token_id)` — `None` → return `None` (the endpoint maps it to the 404).
- `row.revoked_at is not None` → `raise TokenAlreadyRevoked`.
- `revoke(db, row.id)` (stamps the original revocation time — the existing primitive; it returns True here by construction) + `new_row, plaintext = create_token(db, row.label)` (the SAME label — D2).
- `db.flush()` and return `(new_row, plaintext)` — the CALLER commits (the `create_token` convention: the service flushes, the endpoint owns the commit). ONE commit covers stamp + create (atomicity: a create failure rolls the revoke back with it — note this in the docstring).
- Module docstring: the rotation's contract (the old plaintext was already one-shot; the new one is too — A4 never weakens).
2. `app/api/tokens.py` — `@router.post("/{token_id}/regenerate", response_model=TokenCreated, status_code=201)`:
- `result = token_service.regenerate_token(db, token_id)`; `None` → 404 `token not found` (the revoke endpoint's exact message); `except TokenAlreadyRevoked` → 409 `token already revoked`; success → `db.commit()` + `db.refresh(new_row)` (the server-default `created_at`) + `TokenCreated(id, label, token, created_at)`.
- `list_tokens` — pass `revoked_at=row.revoked_at` into each `TokenListItem` (D5).
- The new route's docstring: rotate-once semantics + the A4 note (this 201 is the new token's ONLY plaintext moment).
3. `app/schemas.py` — `TokenListItem` gains `revoked_at: datetime | None = None` (the docstring: null while active; the ISO-8601 timestamp the revoked table renders — the D5 note).
4. Unit (`tests/unit/test_tokens.py` extensions) — the `regenerate_token` matrix (the existing file's DB-session pattern): active row → a NEW row (different id) with the SAME label + a well-formed new plaintext (`bor_` + 32 hex) whose sha256 is the stored hash; the OLD row stamped with a `revoked_at`; the new plaintext round-trips through `find_active_by_token`; the old plaintext no longer authenticates (`find_active_by_token(old)` → `None`); missing id → `None`; an already-revoked id → `TokenAlreadyRevoked`; the rotation is atomic (a create failure — e.g. simulate by exhausting… if the existing harness cannot fail `create_token`, pin the flush-only/commit-caller contract by asserting the session is NOT committed by the service: the caller's rollback undoes both writes).
5. Integration (`tests/integration/test_tokens_api.py` extensions): 201 (body carries the new plaintext — `bor_`-prefixed, ≠ the original; the follow-up list shows the old row `revoked: true` + non-null `revoked_at` and the new row active, same label, newer `created_at`); 404 unknown id (one message); 409 an already-revoked id (revoked via the endpoint first); 403 anonymous + 403 a token-user session (the router-wide gate — the existing 403 pattern in the file); the list response shape now includes `revoked_at` (null for active, timestamp for revoked — the phase-79 list pins updated in place).
## Testing & Quality
- Unit: the matrix above (`tests/unit/test_tokens.py`).
- Integration: the endpoint matrix + the list shape (`tests/integration/test_tokens_api.py`).
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
## Completion Criteria
- [ ] `POST /api/tokens/{id}/regenerate` → 201 with the new token's plaintext (same label); the old row is revoked in the SAME transaction; 404 / 409 pinned; the admin gate pins hold (403 anonymous + token-user)
- [ ] `GET /api/tokens` items carry `revoked_at` (null active / timestamp revoked); the existing phase-79 list assertions (no plaintext, no hashes, newest-first) still pass
- [ ] the A4 contract: the list never gains a credential field (unit + integration pinned)
- [ ] full test suite green, coverage >90%
- [ ] no behavior change in completed work (the phase-79 auth-enforcement + token-gate suites green)
@@ -0,0 +1,58 @@
# Task 02 — The view: active/revoked tables, per-table search, the required name
**Phase:** `101_tokens_page_overhaul` · **Story:** n/a (owner request)
## Objective
The Tokens view splits its list into an active table and a revoked table below it (each with its own live label search), and a blank name can no longer generate a token — the `token` fallback is gone.
## Work
1. `frontend/index.html` `#view-tokens` (the static skeleton — the D1/D4 shapes from `00_phase.md`; keep every existing id that survives):
- The create row: the input's `aria-label` → `Token name`, the `placeholder` → `e.g. alice — required`; the row's comment updated (a blank name is REFUSED client-side — D3 — the server's 422 is defense in depth).
- A search row between the create row/once-block area and the active table's wrap: `<input type="search" id="token-search-active" aria-label="Search active tokens" placeholder="Search active tokens…" class="token-search">` (ships hidden — revealed in the admin branch with the create row).
- The active table (`#tokens-table`): thead loses the Status `<th>` — four columns now: Label | Created | Last used | `<th scope="col"><span class="visually-hidden">Actions</span></th>` (unchanged); `#tokens-empty-row` → `colspan="4"` (copy unchanged); a NEW `<tr class="tokens-empty-row" id="tokens-no-match-row" hidden><td colspan="4"></td></tr>` (its text JS-filled — the no-match copy, D4).
- The NEW revoked section AFTER the active table's wrap (ships `hidden` — tokens.js shows it when ≥ 1 revoked token):
```html
<h2 id="tokens-revoked-heading" class="tokens-revoked-heading">Revoked tokens</h2>
<input type="search" id="token-search-revoked" aria-label="Search revoked tokens" placeholder="Search revoked tokens…" class="token-search">
<div class="table-wrap tokens-table-wrap" id="tokens-revoked-wrap" role="region" aria-label="Revoked tokens" tabindex="0">
<table class="tokens-table" id="tokens-revoked-table">
<caption class="visually-hidden">Revoked tokens — newest first</caption>
<thead><tr>
<th scope="col">Label</th><th scope="col">Created</th>
<th scope="col">Last used</th><th scope="col">Revoked</th>
</tr></thead>
<tbody id="tokens-revoked-tbody">
<tr class="tokens-empty-row" id="tokens-revoked-no-match-row" hidden><td colspan="4"></td></tr>
</tbody>
</table>
</div>
```
(the view's HTML comments carry the D1/D2/D4 provenance — the house comment style; the section's ids are all new, nothing collides).
2. `frontend/assets/tokens.js` (the split + search + required name):
- Module lookups: the revoked section's heading / search input / wrap / tbody / no-match row + the active no-match row (scoped to `root`, the phase-76 contract).
- Module state: `activeQuery = ""` / `revokedQuery = ""` (the D4 persistent queries — initialized once, NEVER reset by a load).
- `makeRow(tok, table)` — `table` is `"active"` | `"revoked"`: shared Label/Created/Last-used cells (the `title` hovers carry over); the active variant's Actions cell (Regenerate — task 03 — + the existing two-step Revoke, UNCHANGED in this task); the revoked variant's last cell = the `revoked_at` date (locale date+time via the existing `fmtDate`, full ISO in `title`, class `tokens-date-cell`) and NO actions.
- `loadTokens()` — the re-entrant core is preserved (once-block hidden + field wiped, data rows dropped, the fetch, the error lines, the return value — all UNCHANGED); the render step now SPLITS the fetched list by `tok.revoked` (server order kept per table: active rows → `#tokens-tbody`, revoked rows → `#tokens-revoked-tbody`); the active empty-state row shows iff there are zero ACTIVE rows (a 0-active fetch with revoked rows shows the empty-state row AND the populated revoked section — both honest); the revoked SECTION (heading + search input + wrap — a small `setRevokedSectionVisible(n)` helper) shows iff n ≥ 1; FINALLY both filters re-apply (`applyFilter` below — D4: a re-render never loses the queries).
- `applyFilter(tbody, noMatchRow, query, )` — pure DOM: trim + lowercase the query; over the tbody's data rows (everything EXCEPT the no-match row), `row.hidden = query && !label.toLowerCase().includes(query.toLowerCase())` (read the label from the row's label cell textContent — the rows are the data source of truth after a render); `noMatchRow` hidden unless `query && zeroDataRowsVisible` — its `<td>` textContent = `No tokens match "${query}".` (the ORIGINAL, untrimmed user query inside the quotes — textContent only).
- The `input` listeners (armed in the admin branch, after the whoami gate — the anonymous branch arms nothing): set the module query + `applyFilter` (NO fetch — D4).
- `generateToken()` (D3): `const label = (labelInput ? labelInput.value : "").trim();` — `if (!label) { announce("Give the token a name first."); if (labelInput) labelInput.focus(); return; }` — the `|| "token"` DELETED; the placeholder/comment (step 1) and the function's comment updated; the rest of the flow (the disabled/"Generating…" lifecycle, the 201 reveal, the list re-load FIRST, the label clear, the error lines) byte-identical in behavior.
- Module docstring: the phase-101 section (the split, the search contract, the required name, the revoked column).
- The `bor:view-refresh` listener is UNCHANGED (it calls `loadTokens()`, which now splits + re-applies the filters — the queries survive a re-show for free).
3. `frontend/assets/styles.css` (the token region):
- `.token-search { width: 100%; min-height: 44px; padding: 0.55rem 0.8rem; border: 1px solid var(--line); border-radius: var(--radius-sm); background: var(--surface); color: var(--ink); font: inherit; font-size: 0.9rem; }` + `:focus-visible` via the global 3px outline rule (no custom focus rule — the house pattern) + a comment (the search inputs' surface — the `#tune-note`/archive-upload input family, AA pairs, no new hue).
- `.tokens-revoked-heading { margin: 1.5rem 0 0.5rem; font-family: var(--mono); font-size: 1rem; color: var(--brand-ink); }` (the sub-heading — the phase-97 `.kb-level h2` voice, AA on `--bg`).
- The no-match rows reuse the existing `.tokens-empty-row` styling (the class is shared — verify it covers both tables; if the empty-row rule is `#tokens-empty-row`-specific, generalize it to `.tokens-empty-row` — the styling is already class-based, so likely a no-op).
- The `≤640px` block: verify the existing `.token-create` / `.tokens-table-wrap` squeeze rules cover the new inputs + the second table (class-based — expected no change; add a search-input width rule only if the mobile layout needs it).
4. Source pins: locate the phase-79 tokens.js full-file source-pin unit file under `tests/unit/` (grep `tokens.js` in `tests/unit/` — if a dedicated file exists, EXTEND it; if the pins live inside `test_api_tokens_model.py` or similar, extend there; if no dedicated pin file exists, CREATE `tests/unit/test_tokens_ui.py`): the two-tbody split in `loadTokens`, the section show/hide helper, `applyFilter` (case-insensitivity, the re-apply-after-load call, the no-match text with the original query, the data-row exclusion), the required-name block (the exact announce copy `Give the token a name first.`, the early return BEFORE any fetch, and a negative pin: the string `"token"` fallback pattern `|| "token"` is GONE from the file), the search inputs' armed-in-admin-branch wiring + the styles.css class pins.
## Testing & Quality
- Unit: the source pins above are this task's test layer for the JS (`app/` untouched this task — task 01's API pins already cover the server side).
- Coverage: **>90%** on this task's new/modified code (frontend pins cover the JS; the browser proof is task 04's E2E).
- Note: do NOT add an E2E file here — the split/search/name proofs land in task 04.
## Completion Criteria
- [ ] the view skeleton carries the four-column active table, the revoked section (heading + search + the four-column table) shipping hidden, and both search inputs (ids per the D1/D4 contract)
- [ ] `loadTokens` splits by `revoked`, hides the section when empty, keeps the once-block / error / return-value contracts byte-identical, and re-applies BOTH persistent queries after every render (source-pinned)
- [ ] a blank/whitespace name announces `Give the token a name first.`, re-focuses, and sends NO request (source-pinned; the `|| "token"` pattern is gone from the file)
- [ ] full test suite green, coverage >90%
- [ ] no behavior change in completed work (the phase-79 gate / whoami / once-block pins green; the Regenerate button is NOT built yet — task 03 adds it)
@@ -0,0 +1,32 @@
# Task 03 — The Regenerate control: one button, the house confirm, the once-block reveal
**Phase:** `101_tokens_page_overhaul` · **Story:** n/a (owner request)
## Objective
Every active token row carries a **Regenerate** button (before its Revoke): one click opens the house two-step confirm, Yes rotates the token server-side, and the new plaintext appears in the shown-once block — the old token relocates to the revoked table.
## Work
1. `frontend/assets/tokens.js`:
- `makeRegenerateControl(tok, row)` — a structural mirror of `makeRevokeControl` (the D2 contract from `00_phase.md`): a `.token-regenerate` button labeled `Regenerate` (`aria-label: "Regenerate token: <label>"`); first click swaps the cell's contents to the confirm pair — `span.history-confirm-text` `Regenerate? The current token is revoked.` + Yes (`.history-confirm-yes`) + No (`.history-confirm-no`), focus to Yes; No / failure → `restoreRegenerate()` (the button back, focus restored — the revoke control's restore pattern, copied).
- `confirmRegenerate(tok, row, yesBtn, restoreRegenerate)`: `yesBtn.disabled = true`; `POST /api/tokens/<id>/regenerate` (JSON — no body):
- **201** → parse `{ id, label, token, created_at }` → `await loadTokens()` FIRST (the re-entrant load: the old row moves to the revoked table, the new row lands in the active one — the D2 relocation) → the once-block reveal (`if (onceValue) onceValue.value = data.token;` + `if (onceBlock) onceBlock.hidden = false;` — the SAME once-block, the A4 value-only contract: the plaintext lives in the field's value, never a data attribute) → `announce(`Regenerated "${tok.label}" — copy the new token now; it won't be shown again.`)`.
- **404** → the row is gone (revoked elsewhere / regenerated elsewhere): `row.replaceWith(makeRow({ ...tok, revoked: true }, "revoked"))` — wait: the row is an ACTIVE-table row; a 404 means the id no longer exists at all. Mirror the revoke control's 404 handling: re-render the row in the REVOKED table's state is wrong if the row is gone — instead `row.remove()` + `announce("That token no longer exists.")`… **Decision (pinned here):** a 404 on regenerate means the row vanished (revoked AND deleted by another admin, or a stale render) — `row.remove()` + `await loadTokens()` (the re-fetch reconciles both tables) + `announce("That token was already revoked.")` (the revoke control's existing 404 line — one house message for the one common case).
- **409** → the row was revoked between render and click: `await loadTokens()` (reconciles) + `announce("That token was already revoked.")`.
- **other non-2xx / network** → `announce("Couldn't regenerate \"<label>\" — try again.")` (network: the `is the app reachable?` variant — the house two-line convention) + `restoreRegenerate()` (retryable, the button back).
- The active `makeRow`'s Actions cell: `actionsTd.append(makeRegenerateControl(tok, tr), makeRevokeControl(tok, tr))` — Regenerate FIRST (the primary lifecycle action, D2); the two controls each own their own `<span class="tokens-actions">` cell-content swap (verify the two swap-scopes don't interfere — the revoke control swaps `cell`'s children; give each control its OWN wrapper span (the regenerate control gets its own, the revoke control keeps its existing one) so a confirm in one never clobbers the other).
- The `#tokens-status` live-region contract is unchanged (every outcome lands a line — D2's copy is pinned).
- Module docstring: the regenerate section (the rotation semantics + the 404/409 copy).
2. `frontend/assets/styles.css` (the token region): `.token-regenerate` — the `.token-revoke` button's visual language (same size/border/radius/focus; color `var(--ink-soft)` on transparent, hover `var(--brand-soft)` / `var(--brand-ink)` — the neutral action's hover, NOT the revoke's error hover; a `.token-regenerate:hover:disabled` state consistent with the revoke button's disabled rule) + the comment (the rotation button — no new hue, phase-92 invariant; the confirm pair reuses the `history-confirm-*` classes unchanged).
3. Source pins (the file established in task 02): `makeRegenerateControl` (the button label + aria-label, the confirm text EXACTLY `Regenerate? The current token is revoked.`, the focus-to-Yes, the restore path), `confirmRegenerate` (the POST path `/api/tokens/` + `regenerate`, the 201 sequence — load FIRST then once-block reveal then announce (the D2 line, exact), the 404/409 copy, the neutral retry copy, the disabled-while-in-flight), the Actions cell order (Regenerate before Revoke), the per-control wrapper spans, + the `.token-regenerate` CSS pin.
## Testing & Quality
- Unit: the source pins above are this task's test layer (`app/` untouched — task 01's API pins cover the endpoint).
- Coverage: **>90%** on this task's new/modified code (frontend pins cover the JS; the browser proof is task 04's E2E).
- Note: do NOT add an E2E file here — the rotation proof lands in task 04.
## Completion Criteria
- [ ] an active row shows Regenerate (before Revoke); the confirm copy/focus/restore are pinned; Yes → 201 → the old row is in the revoked table, the new row active (same label), the once-block shows the new plaintext, the D2 line is announced (source-pinned; task 04 proves it in a browser)
- [ ] the 404 / 409 / failure branches are pinned (the house copy, the retryable restore, the load-based reconciliation)
- [ ] the revoke control's behavior is byte-identical (its pins green — the shared cell hosts two independent confirm scopes)
- [ ] full test suite green, coverage >90%
- [ ] no behavior change in completed work (the phase-79 once-block / copy / gate pins green)
@@ -0,0 +1,34 @@
# Task 04 — The dedicated E2E: the split tables, the search, the required name, the rotation + regressions + commit
**Phase:** `101_tokens_page_overhaul` · **Story:** n/a (owner request)
## Objective
Pin the owner-visible contract in a browser: revoked tokens live in their own table below the active ones, both tables search, a nameless token cannot be created, and Regenerate rotates the credential end to end (the new token signs in, the old one is refused) — then the phase-79 suite moves its revoked-row pins to the new table.
## Work
1. `tests/e2e/test_tokens_page.py` (new — the phase's dedicated A16 suite, run in isolation). Fixtures: `app_server` + `mock_llm` + `db_ready` (the `conftest` pattern); admin via `tests/e2e/auth_helpers.login(page, app_url, next="/tokens.html")`; a token-user context via a FRESH browser context + `auth_helpers.login_with_token(page, app_url, token)` (the phase-79 helper — drives the real gate). An autouse fixture deletes `e2e-`-labeled tokens (the `test_api_tokens.py` cleanup pattern — `DELETE FROM api_tokens WHERE label LIKE 'e2e-%'`).
- **`test_revoked_tokens_move_to_their_own_table`** — create two tokens through the UI create row (`e2e-act`, `e2e-rev` — label → Generate → the once-block appears each time); Revoke `e2e-rev` (the existing two-step: click Revoke → click Yes): `#tokens-tbody` shows ONLY `e2e-act`; the active table's `<thead>` has exactly four `<th>` (NO Status column); `#tokens-revoked-heading` reads `Revoked tokens` and is visible; `#tokens-revoked-tbody` shows the `e2e-rev` row with a non-empty Revoked date cell (locale format, ≠ the created date) and NO buttons in its row; `#tokens-status` reads `Revoked "e2e-rev".`.
- **`test_both_tables_are_searchable`** — with `e2e-a1` + `e2e-a2` active and one revoked token present: type `e2e-a1` in `#token-search-active` → only the `e2e-a1` row visible (`e2e-a2` hidden); type `zzz` → the no-match row visible reading `No tokens match "zzz".`; clear the input → both rows back, no-match hidden. The same three-beat on `#token-search-revoked` (filter the revoked row in/out). The queries SURVIVE a re-show: type a query, leave via the Tuning nav link, come back (the phase-77 refresh re-renders) → the filter is still applied (the visible set unchanged).
- **`test_a_token_cannot_be_generated_without_a_name`** — with the label empty, click Generate: `#token-once` stays hidden, `#tokens-status` reads `Give the token a name first.`, the active table's data-row count is unchanged, and `GET /api/tokens` (admin cookie via `page.request.get`) shows no new row (the request never happened). Whitespace-only label → the same refusal.
- **`test_regenerate_rotates_the_token`** — create `e2e-rot` (capture its plaintext from `#token-once-value`); on its row click Regenerate → the confirm pair appears (`Regenerate? The current token is revoked.`, focus on Yes) → click Yes: `#token-once` re-appears with a NEW plaintext (≠ the original, matches `bor_[0-9a-f]{32}`); `#tokens-status` reads `Regenerated "e2e-rot" — copy the new token now; it won't be shown again.`; the active table has exactly ONE `e2e-rot` row (the new one — its created cell is newer than the old row's); the revoked table now holds an `e2e-rot` row (the original — its Revoked date is set); a fresh token-user context signs in with the NEW token (the gate closes, the chat view usable — `login_with_token` + an assertion on the signed-in state) and the ORIGINAL token is refused in another fresh context (the `#auth-gate-error` role=alert visible after submitting it — the rotation killed it, the phase-79 revocation semantics).
2. `tests/e2e/test_api_tokens.py` — UPDATE IN PLACE (the phase-97 task-07/08 precedent — the layout changed, the auth contracts did not):
- The revoked-row assertions relocate to `#tokens-revoked-tbody`: `test_admin_generates_token_in_ui` (its post-revoke expectations — the row's NEW home; the `.stale-pill` assertion drops with the Status column — assert the revoked row + its Revoked date cell instead) and `test_revocation_closes_the_door` (the two-step revoke now lands the row in the revoked table — update the locator + the pill assertion the same way; the `Revoked "e2e-revoke".` live-region line is UNCHANGED).
- Any Status-column / 5-`<th>` / `colspan="5"` references update to the four-column contract; `#tokens-empty-row`'s colspan is 4.
- EVERYTHING else — the anonymous lockout, the shared-chats-stay-open, the token-user app usage, the cached-token reload, the 403 walls, the sign-out clears, the wrong-token generic error — stays byte-identical (this suite is the phase-79 auth contract; only its layout locators move).
3. Regression sweep (each in isolation, `--no-cov`, DB up): the updated `test_api_tokens.py` + `test_admin_auth.py` (the gate contract — expected untouched) + `test_shared_header.py` (the whoami/header plumbing — expected untouched).
4. Full gates + commit: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean; move `101_tokens_page_overhaul` → `.agents/phases/complete/`; one atomic commit:
```bash
git add -A .agents/ app/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(tokens): split active and revoked tables with search, require a name, and add one-click regenerate"
```
## Testing & Quality
- E2E (mandatory, A16): `uv run pytest tests/e2e/test_tokens_page.py -v --no-cov` green in isolation.
- Coverage: **>90%** on `app/` (the full-suite gate).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] the four E2E tests pass in isolation (the split + the column contract, both searches incl. the re-show persistence, the nameless refusal, the full rotation with the new-token-signs-in / old-token-refused proof)
- [ ] the updated `test_api_tokens.py` passes in isolation (the phase-79 auth contract intact, the layout pins relocated)
- [ ] the regression suites pass in isolation
- [ ] full suite green, coverage >90%, ruff + pyright clean
- [ ] phase dir moved to `complete/`, one atomic `--no-gpg-sign` Conventional Commit
@@ -0,0 +1,61 @@
# Phase 102 — Extensionless filenames: sync `Dockerfile` / `Containerfile` when their name is in `BOR_IMPORT_EXTENSIONS`
**Source:** owner report (chat): "files without extensions never get synced, so 'Dockerfile' and 'Containerfile' get skipped even if I put them in BOR_IMPORT_EXTENSIONS."
**Story:** n/a (defect — the import pipeline of `38_local_directory_sources` / `56_import_extensions_env`; the badge display rides on the phase-10 viewer).
**Context:** the importer's format filter is `path.suffix.lower() in extensions` (`app/rag/importer.py`, `iter_importable_files`) and `Settings.import_extension_set` (the dotted set, `app/config.py`) feeds it. `Path("Dockerfile").suffix` is `""`, so **extensionless files never match — no value of `BOR_IMPORT_EXTENSIONS` can import them**. The validator already accepts `dockerfile` as a well-formed token (lowercase alphanumerics, 1-16 chars, phase 56 — "any extension"), so today the user can *type* the format and it still silently imports nothing. The viewer's format badge (`doc_format`, `app/api/docs.py`, rendered by `frontend/assets/document.js`) falls back to `text` for any suffix-less path; the import `formats=` counter (`ImportSummary.formats`) keys extensionless files under `unknown`.
## Objective
An extensionless file is imported **iff its lowercased full filename is a token in `BOR_IMPORT_EXTENSIONS`** — exact name, case-insensitive. With `dockerfile,containerfile` in the env (the owner's current config), `Dockerfile` and `Containerfile` sync like any other A9 file: chunks, `lite` summary, Sources tree, viewer with a truthful format badge.
## Dependencies
- `56_import_extensions_env` (the env-driven extension scope this phase extends) — complete
- `38_local_directory_sources` / `97_kb_tree_catalog` (import pipeline + drill-down tree the E2E asserts on) — complete
## Design (shared by all tasks — the executor reads this, not the chat)
### D1 — the matching rule (task 01)
A file under a source root is in scope **iff**:
1. its lowercased dotted suffix is a member of `import_extension_set` (the existing rule — `kubernetes.md` → `.md`), **or**
2. it has **no suffix** and its lowercased **full filename** equals a bare token of `import_extension_set` (`Dockerfile` → `dockerfile`).
Consequences (pinned by tests): `DOCKERFILE` matches (case-insensitive name); `mydockerfile` and `dockerfile`-suffixed lookalikes do **not** (exact name only — the list is user-controlled and exact is the only predictable rule); `Dockerfile.dev` is governed by its `.dev` suffix as today; dot-prefixed files (`.dockerfile`) stay skipped by the pre-existing hidden-component rule; the rule applies at any depth (`services/api/Dockerfile` matches).
### D2 — one choke point (task 01)
`app/rag/importer.py` gains `match_extension(path: Path, extensions: frozenset[str]) -> str | None` — returns the matched **bare lowercased token** (``md``, ``dockerfile``) or `None` when out of scope; `extensions` keeps the existing dotted form (``{'.md'}``), bare tokens are derived by stripping the leading dot. `iter_importable_files` filters with `match_extension(path, extensions) is not None` (the walk and the phase-64 progress pre-walk both call this function — one change covers both, `total` stays honest). The `summary.formats` counter in `import_sources` uses `match_extension(path, …) or "unknown"`, so an imported `Dockerfile` counts under `dockerfile:1` in the PLAN §9 line, not `unknown:1`.
### D3 — the badge (task 02)
`doc_format(path, extensions: frozenset[str] = frozenset())` in `app/api/docs.py`: a **non-empty suffix still returns the suffix token unconditionally** (display must never depend on the import list — an out-of-scope `readme.rst` still badges `rst`); a suffix-less path returns the name token when it matches per D1, else `text` (today's fallback, byte-identical for every existing row). The content endpoint (`GET /api/documents/content`) passes `get_settings().import_extension_set` (`app.config`, lru-cached). `frontend/assets/document.js` needs no change: only `md`/`markdown` trigger markdown rendering, so a `dockerfile` badge renders the raw `<pre>` view — correct for a build file.
### D4 — no changes (pinned)
Title stays `full_path.stem` (`Dockerfile` — already correct); the phase-30 `lite` summary applies (non-markdown); prune semantics unchanged — a file that stops matching (file renamed, or token removed from the env) leaves `seen` and is deleted by the next `prune=True` run (the A9 junk precedent).
### D5 — docs
`app/config.py` `import_extensions` docstring + the `.env.example` "Import scope" comment gain the extensionless rule, with `dockerfile,containerfile` as the example (no default-list change — the built-in A9 default stays exactly as phase 56 locked it).
## Tasks
1. `01_name_token_matching.md` — `match_extension` in the importer + the walk + the `formats` counter (unit-pinned).
2. `02_format_badge_and_docs.md` — the viewer badge via `doc_format` + config/`.env.example` docs.
3. `03_integration_and_e2e.md` — the dedicated integration suite + the Playwright story suite + regressions + atomic commit.
## Testing & Quality
- Unit: `tests/unit/test_importer.py` — the matching matrix (D1), walk inclusion/exclusion, the `formats` counter; `tests/unit/test_document_viewer.py` — `doc_format` (with/without the token set, suffix precedence, `text` fallback).
- Integration: `tests/integration/test_import_extensionless.py` (new) — in-process `import_sources` against a story-dedicated fixture dir with the deterministic mock LLM (the phase-02/56 seeding pattern): positive import (counts, per-format `dockerfile:1,containerfile:1,md:1`, lite summary generated for the extensionless files), negative import (no token → not walked), prune-on-token-removal; `tests/integration/test_document_content.py` — the `format` field carries `dockerfile` for an extensionless row and `text` for an unknown-name row.
- E2E (mandatory, house rule 4): `tests/e2e/test_extensionless_import.py`, run in isolation (DB up) — admin Signs in, the drill-down tree lists `Dockerfile`/`Containerfile`, the viewer modal shows badge `dockerfile` + title + summary, `Makefile` (no token) is absent, rows cleaned up in `finally`.
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — the phase is "Complete" only when the full suite passes and the gate holds.
## Completion Criteria
- [ ] `BOR_IMPORT_EXTENSIONS` containing `dockerfile,containerfile` (plus the usual formats) makes a `Dockerfile` and a `Containerfile` under a source sync end-to-end: imported, chunked, `lite`-summarized, listed in the Sources drill-down tree, openable in the viewer.
- [ ] Case-insensitivity: `DOCKERFILE` imports under the `dockerfile` token; `mydockerfile` does **not**; `Dockerfile.dev` follows its `.dev` suffix; `.dockerfile` stays hidden-skipped.
- [ ] The import `formats=` counter reports `dockerfile:1` (not `unknown:1`) for an imported `Dockerfile`.
- [ ] The viewer format badge shows `dockerfile`/`containerfile`; a suffix-less file whose name is not in the env still badges `text`; every suffixed row's badge is unchanged.
- [ ] Removing a token from the env prunes the previously-imported extensionless documents on the next `prune=True` run.
- [ ] The existing suites stay green in isolation: `test_import_documents.py`, `test_import_extensions_env.py`, `test_quadlet_jinja_import.py`, `test_document_viewer.py`, `test_kb_tree.py`.
- [ ] `uv run pytest` green; `app/` coverage >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` green in isolation (DB up).
- [ ] No behavior change to completed phases for any suffixed path (the existing walk rule is untouched — only the suffix-less branch is new).
- [ ] One atomic Conventional Commit, `--no-gpg-sign` (e.g. `fix(import): sync extensionless files (Dockerfile, Containerfile) named in BOR_IMPORT_EXTENSIONS`), `.agents/` phase files moved to `complete/` with the phase.
## Commit
```bash
git add app/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "fix(import): sync extensionless files (Dockerfile, Containerfile) named in BOR_IMPORT_EXTENSIONS"
```
@@ -0,0 +1,42 @@
# Task 01 — The name-token match: `match_extension`, the walk, the `formats` counter
**Phase:** `102_extensionless_filenames` · **Source:** owner report — "files without extensions never get synced … 'Dockerfile' and 'Containerfile' get skipped even if I put them in BOR_IMPORT_EXTENSIONS."
**Story:** n/a (defect).
## Objective
One pure helper decides import scope; the walk and the per-format counter use it; extensionless files whose lowercased full name is a `BOR_IMPORT_EXTENSIONS` token are finally imported.
## Work
1. `app/rag/importer.py` — new module-level pure function, placed near `iter_importable_files`:
```python
def match_extension(path: Path, extensions: frozenset[str]) -> str | None:
"""The bare lowercased token *path* imports under, or ``None``.
1. Non-empty lowercased dotted suffix in *extensions* (the A9 rule —
``kubernetes.md`` → ``md``).
2. No suffix: the lowercased FULL filename equals a bare token of
*extensions* (``Dockerfile`` → ``dockerfile``) — the phase-102
extensionless rule. Exact name only: ``mydockerfile`` never
matches the ``dockerfile`` token.
"""
```
*extensions* keeps the existing dotted form (``{'.md', '.dockerfile'}``); derive the bare token set by stripping the leading dot. Rule 1 first; rule 2 only when ``path.suffix`` is empty.
2. `app/rag/importer.py` — `iter_importable_files`: replace the `path.suffix.lower() not in extensions` skip with `match_extension(path, extensions) is None`. The hidden-component check and the phase-89 ignore-prefix check stay before it, byte-identical.
3. `app/rag/importer.py` — the `summary.formats` line in `import_sources` (`ext = path.suffix.lower().lstrip(".") or "unknown"`): use `match_extension(path, llm.settings.import_extension_set) or "unknown"` so an imported `Dockerfile` counts under `dockerfile`, never `unknown`.
4. `app/rag/importer.py` — module docstring: one sentence for the extensionless rule in the scope paragraph (tokens may also name extensionless files by exact lowercased name — phase 102).
ASSUMPTIONS (pinned in `00_phase.md` D1/D2): exact-name, case-insensitive matching; no partial name matching; dotted set kept as the parameter form so no `app/config.py` change is needed in this task.
## Testing & Quality
- Unit (`tests/unit/test_importer.py`, new tests next to the existing `iter_importable_files` ones):
- `match_extension` matrix: `kubernetes.md` + `{'.md'}` → `md`; `Kubernetes.MD` → `md` (case-insensitive suffix, as today); `Dockerfile` + `{'.md', '.dockerfile'}` → `dockerfile`; `DOCKERFILE` → `dockerfile`; `Dockerfile` without the token → `None`; `mydockerfile` with the token → `None`; `Dockerfile.dev` with only `dockerfile` → `None` (with `.dev` it → `dev`); `readme.rst` + `{'.md'}` → `None`.
- walk: a fixture tree with `Dockerfile`, `Containerfile`, `mydockerfile`, `Dockerfile.dev`, `.dockerfile`, `notes.md` — with tokens `md,dockerfile,containerfile` the walk yields exactly `Dockerfile`, `Containerfile`, `notes.md` (sorted); the hidden `.dockerfile` stays skipped.
- counter: in-process `import_sources` (the file's existing `db` + fake-LLM pattern) over a tree with one `Dockerfile` + one `.md` → `summary.formats == {"dockerfile": 1, "md": 1}` and the PLAN §9 log line carries `dockerfile:1` (not `unknown`).
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
## Completion Criteria
- [ ] `match_extension` exists, pure, dotted-set parameter, rule order pinned by the matrix above
- [ ] `iter_importable_files` + the progress pre-walk import extensionless name-token files (both go through the helper)
- [ ] the `formats` counter reports the matched token, never `unknown`, for in-scope files
- [ ] full test suite green, coverage >90%, `uv run ruff check . && uv run pyright` clean
- [ ] no behavior change for any suffixed path (existing `test_importer.py` cases byte-identical)
@@ -0,0 +1,32 @@
# Task 02 — The truthful badge: `doc_format` learns the name-token rule + config/env docs
**Phase:** `102_extensionless_filenames` · **Story:** n/a (defect, display half).
## Objective
The viewer's format badge shows `dockerfile`/`containerfile` for extensionless documents that matched by name (instead of the generic `text`), and the config/`.env.example` docs tell the user the extensionless rule exists.
## Work
1. `app/api/docs.py` — `doc_format(path: str, extensions: frozenset[str] = frozenset()) -> str`:
- non-empty suffix → the suffix token, **unconditionally** (display never depends on the import list — an out-of-scope `readme.rst` still badges `rst`);
- no suffix → reuse the task-01 helper semantics from `app/rag/importer.py` (import `match_extension` — `app.api.docs` already imports from `app.rag`): the lowercased full name returns the token when it is one of `extensions` (dotted form, as passed), else `text`.
- Default `extensions=frozenset()` ⇒ today's exact behavior for every caller that passes nothing (byte-identical).
- Docstring: update the fallback line — ``text`` when the path has no suffix **and its name is not a configured token (phase 102)**.
2. `app/api/docs.py` — the content endpoint (`GET /api/documents/content`, `format=doc_format(doc.path)` at ~L143): pass `get_settings().import_extension_set` (`from app.config import get_settings` — the lru-cached accessor; no per-request Settings construction).
3. `app/config.py` — `import_extensions` docstring: one sentence — a token also matches **extensionless files whose lowercased full filename equals it exactly** (`dockerfile` → `Dockerfile`), case-insensitive, no partial names (phase 102).
4. `.env.example` — "Import scope" comment: add the same rule with the example `dockerfile,containerfile` (e.g. `add your own (e.g. md,sh,toml,dockerfile,containerfile)`); the default `BOR_IMPORT_EXTENSIONS` line is unchanged.
5. No frontend change: `frontend/assets/document.js` renders `doc.format` into the `format-badge` span and only `md`/`markdown` get markdown rendering — a `dockerfile` badge already lands in the raw `<pre>` view, which is the correct treatment for a build file. Verify, don't edit.
## Testing & Quality
- Integration (`tests/integration/test_document_content.py`, extend the `test_content_format_from_suffix` neighborhood — seed via the file's existing `_seed_doc` + TRUNCATE-in-`finally` pattern):
- seeded `path="services/api/Dockerfile"` (source `Homelab`) → `body["format"] == "dockerfile"` under the default settings (the endpoint reads the real `get_settings()` — if the test needs the token, set `BOR_IMPORT_EXTENSIONS` for the client/session via the house env-override pattern used by other API tests; the default list does **not** contain `dockerfile`, so the assertion must run with a settings carrying it);
- seeded `path="README"` → `format == "text"` (unchanged — `readme` is not a token; the existing line at ~L554 stays green);
- seeded `path="notes/README.dev"` → `format == "dev"` (suffix precedence — display ignores the import list);
- `doc_format` directly: default-args call on an extensionless path → `text` (the no-arg contract pinned).
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
## Completion Criteria
- [ ] the content endpoint's `format` field carries the name token for an extensionless document whose name is in `BOR_IMPORT_EXTENSIONS`, `text` otherwise, and every suffixed value is unchanged
- [ ] `doc_format` with no second argument behaves byte-identically to today
- [ ] `app/config.py` docstring + `.env.example` document the extensionless rule with the `dockerfile,containerfile` example
- [ ] full test suite green, coverage >90%, `uv run ruff check . && uv run pyright` clean
- [ ] `frontend/assets/document.js` verified untouched (raw `<pre>` rendering is correct for the new badge values)
@@ -0,0 +1,39 @@
# Task 03 — The proof: dedicated integration suite, the Playwright story suite, regressions, commit
**Phase:** `102_extensionless_filenames` · **Story:** n/a (defect — phase-level E2E per house rule 4).
## Objective
End-to-end proof that `Dockerfile`/`Containerfile` sync when their names are in `BOR_IMPORT_EXTENSIONS` — import → chunks → `lite` summary → drill-down tree → viewer badge — plus the negative and prune proofs, regressions, and the atomic commit.
## Work
1. Fixture `tests/fixtures/extensionless_kb/` (new, story-dedicated, distinctive source name = the dir name, the phase-56 pattern):
- `Dockerfile` — a few lines of plausible compose/build content carrying a unique sentinel (e.g. `DOCKERFILE-PROBE-SENTINEL-7a3e`);
- `Containerfile` — same shape, its own sentinel;
- `Makefile` — a few lines, **no token in the env under test** (the negative control);
- `notes.md` — one short markdown doc (the control row + the `md` count).
2. Integration `tests/integration/test_import_extensionless.py` (new; the `tests/integration/test_import_extensions_env.py` pattern — in-process `import_sources` against a fixture/tmp tree, `Settings(_env_file=None, import_extensions=…)` + the house mock LLM, `db` fixture, TRUNCATE/delete rows in `finally`):
- positive: extensions `md,dockerfile,containerfile` over the fixture → `added == 3`, `errors == 0`, `formats == {"dockerfile": 1, "containerfile": 1, "md": 1}` (NO `unknown` key), the `Dockerfile` row has chunks and a `lite` summary (non-markdown → phase-30 path);
- case: a `DOCKERFILE` file + token `dockerfile` imports (its row's path is `DOCKERFILE`);
- negative: extensions `md` only → `notes.md` imported, `Dockerfile`/`Containerfile`/`Makefile` absent from `documents`;
- prune: import with the tokens, then re-run `prune=True` with extensions `md` → the two extensionless rows are deleted, the `md` row survives (mirrors `test_prune_removes_files_now_excluded_by_format_filter`).
3. E2E `tests/e2e/test_extensionless_import.py` (new; **copy the skeleton of `tests/e2e/test_import_extensions_env.py`** — the module-docstring conventions, the in-thread seeding pattern, `_drill`/`_go_top`, the distinctive-source row cleanup in `finally`):
- boot/app with the house `mock_llm` + `db_ready` fixtures; seed via `import_sources([FIXTURES], LLMClient(settings))` with `import_extensions="md,dockerfile,containerfile"` (`Settings(_env_file=None, llm_base_url=…, import_extensions=…)`);
- test 1 (admin): summary assertions as in the integration positive case, then `login(page, app_url)`, `_drill(page, SOURCE)` — rows for `Dockerfile`, `Containerfile`, `notes.md` each count 1; `Makefile` count 0 (negative control); click the `Dockerfile` path link → same-page modal: `#doc-modal-meta .doc-source-badge` = source, `.format-badge` = `dockerfile` (the D3 badge), title `Dockerfile`, `pre.doc-raw` contains the sentinel, the phase-30 summary line renders (house assertion style — `test_summary_in_viewer.py`); still on `/sources.html`;
- test 2 (anonymous): the sign-in gate renders and `GET /api/docs` 403s (the phase-16 contract, same shape as the phase-56 suite's second test).
4. Regressions (isolated runs, per AGENTS.md rule 9): `tests/e2e/test_import_documents.py`, `test_import_extensions_env.py`, `test_quadlet_jinja_import.py`, `test_document_viewer.py`, `test_kb_tree.py`.
5. Commit: full suite + `--cov=app` >90% + `uv run ruff check . && uv run pyright` clean, then move this phase dir `102_extensionless_filenames/` to `.agents/phases/complete/` and commit **everything together** (AGENTS.md rule 8 — `.agents/` is tracked), one atomic commit:
```bash
git add app/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "fix(import): sync extensionless files (Dockerfile, Containerfile) named in BOR_IMPORT_EXTENSIONS"
```
## Testing & Quality
- The suites above ARE this task's tests; coverage gate **>90%** on `app/` (the fixture files and the E2E file need no coverage, `app/` is the measured package).
- E2E runs in isolation: `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` (DB up: `podman compose up -d db`).
## Completion Criteria
- [ ] `tests/fixtures/extensionless_kb/` exists with `Dockerfile`, `Containerfile`, `Makefile`, `notes.md` (sentinels unique to this suite)
- [ ] integration suite green: positive (counts + `dockerfile:1,containerfile:1,md:1` + summary), case, negative, prune
- [ ] E2E green in isolation: tree rows (incl. `Makefile` absent), the `dockerfile` format badge in the modal, the raw-content sentinel, the anonymous gate
- [ ] all five regression suites green in isolation
- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean
- [ ] one atomic Conventional Commit with `--no-gpg-sign` (message above); phase dir moved to `.agents/phases/complete/` in the same commit
@@ -0,0 +1,46 @@
# Phase 103 — Onboarding chips suggest only session-opening questions, never follow-ups
**Source:** Owner request (chat, 2026-09-12) — "The suggested questions on the chat page should not include follow-up questions from a session. The problem is this: Users will ask 'What are the correct arguments for qwen 3.8 27b on llama.cpp?' and BOR will answer. Then, users will ask 'What about qwen 3.6 35b?'. That second question … shows up as a suggested question to *start* a conversation but that question makes no sense without the context behind it. The only questions that should show up as a suggested question are those at the very beginning of a session."
**Story:** n/a (owner request — tightens the onboarding-chips contract of `80_history_suggestion_chips`; the story file `suggestion-chips.md` cited by phases 05/80 no longer exists in `.agents/user_stories/`).
**Context:** `app/api/suggestions.py` — `last_questions(db, limit=3)` walks each saved chat's `messages` in REVERSE (newest-first) and collects every user question (the contract this phase replaces); `GET /api/suggestions` (the `require_user`-gated endpoint, phase 79) with the seed fallback (`get_settings().suggestions` — `BOR_SUGGESTIONS` / built-in) when the walk yields nothing. `app/models.py` — `SavedChat.messages` is the raw `bor.chat.v1` JSONB list in CONVERSATIONAL order (oldest→newest); `SavedChat.title` is the first user question truncated to 120 chars + whitespace-collapsed at save time (`app/api/chats.py` `_auto_title`, `_AUTO_TITLE_MAX = 120`) and user-editable on re-Save. `tests/integration/test_suggestions_api.py` — the phase-80 state matrix (REWRITTEN in task 01). `tests/e2e/test_suggestion_chips.py` — the phase-80 story suite (REWRITTEN in task 03, the phase-76/80 precedent). Docs carrying the "last 3 questions asked" wording: `app/config.py` ~L373-377 (the `suggestions` seed docstring), `.env.example` ~L60 (the `BOR_SUGGESTIONS` comment), `README.md` ~L74.
## Objective
A suggested question must make sense on its own: the onboarding chips are the **first user question of each saved chat** (the session's opening question) — follow-up questions ("What about …?") can never appear, because they are unanswerable without the session behind them. Walk order, dedup, the cap of 3, and the seed fallback are unchanged.
## Owner decisions (chat, 2026-09-12 — recorded per AGENTS.md rule 3)
- **A1 — openers only:** "The only questions that should show up as a suggested question are those at the very beginning of a session." Each saved chat contributes AT MOST ONE chip: its first user question. In the owner's example, "What are the correct arguments for qwen 3.8 27b on llama.cpp?" (the opener) may chip; "What about qwen 3.6 35b?" (the follow-up) may not.
- **A2 — everything else unchanged:** chats still walked newest-`updated_at` first (`created_at` tiebreak), EXACT (case-sensitive) de-dup, cap 3 applied AFTER dedup, seed fallback when the walk yields zero openers (`BOR_SUGGESTIONS` override or built-in default) — all phase-80 contracts survive. The deflection "Maybe try" chips (`app/rag/suggestions.py` `derive_suggestions`, carried in the chat response) are a separate contract and untouched. The FRONTEND is untouched — the chip row renders whatever the endpoint returns (chip sizing/truncation is phase 104's job).
- **A3 — the defensive opener rule:** a chat's opener is its first `who == "user"` message whose trimmed `text` is non-blank. A LEADING blank user entry (the UI cannot produce one — `handleSend` trims and guards `!text`) does not disqualify the chat; a record with no non-blank user message (brain-only, or blank-user-only) contributes nothing.
- **A4 — read `messages`, not `title`:** `SavedChat.title` is truncated to 120 chars + whitespace-collapsed at save time and is user-editable on re-Save — the chips must carry the EXACT full opener text from the raw `bor.chat.v1` record (the phase-80 precedent: no SQL JSON ops, the deserialized list).
## Design (shared by all tasks — the executor reads this, not the chat)
- **`app/api/suggestions.py` — the ONLY file changed in `app/`:**
- `last_questions` is RENAMED `opening_questions` (the old name would lie about the semantics; the helper is module-internal — its only caller is the endpoint, the tests hit the endpoint). Signature unchanged: `(db: Session, limit: int = 3) -> list[str]`.
- **The new walk:** for each chat in `updated_at DESC, created_at DESC` order, walk `chat.messages or []` FORWARD (oldest→newest — `bor.chat.v1` conversational order), take the first entry with `who == "user"` whose trimmed `text` is non-blank (the opener, per A3); if found and not already `seen` (exact, case-sensitive), append it; stop once `limit` UNIQUE openers are collected. Result in encounter order (newest chat first). No other endpoint change: `qs = opening_questions(db)` → `SuggestionList(suggestions=qs if qs else get_settings().suggestions)`.
- **Docstrings (the house dense-docstring style):** the module docstring's phase-80 paragraph becomes the opener contract — WHY follow-ups are excluded (a follow-up like "What about X?" is meaningless as a conversation starter — the owner's llama.cpp/qwen example); the function docstring documents the forward walk, the A3 rule, the A4 why-not-title note, and that dedup/cap/order are the phase-80 contracts; the endpoint docstring says "the opening questions of the 3 most recent saved chats — or, before any question has ever been saved, the seed list".
- **Not touched:** schemas, models, migrations, `app/rag/suggestions.py` (deflection), all of `frontend/` (the chips render the endpoint's list — truncation of long chips is phase 104), the auth gate.
- **Docs (task 02):** `app/config.py` seed docstring, `.env.example` `BOR_SUGGESTIONS` comment, `README.md` chat-features line — "the last 3 questions asked" → "the opening questions of the 3 most recent saved chats (the session openers, newest first)".
## Dependencies
- `80_history_suggestion_chips` (complete) — the endpoint, the seed fallback, the dedup/cap/order contracts, the story E2E suite this phase rewrites.
- `79_api_tokens` (complete) — the `require_user` gate; the tests sign in first (unchanged).
- `102_extensionless_filenames` (todo) — queue order only (numeric); no code dependency (different subsystem).
## Tasks
1. `01_opener_extraction.md` — the `opening_questions` rewrite (rename + forward walk + docstrings) + the integration matrix rewrite.
2. `02_openers_docs.md` — the "last 3 questions" → "session openers" wording in config / `.env.example` / README.
3. `03_e2e_suite_commit.md` — the story-suite rewrite to the opener semantics + regression E2Es + full gate + atomic commit.
## Testing & Quality
- Integration — REWRITTEN `tests/integration/test_suggestions_api.py`: the full opener matrix (one chat's follow-ups never surface; the cap now binds ACROSS chats; opener dedup; case variants; the A3 leading-blank rule; brain-only → seed; auth 401 — full detail in task 01).
- E2E — REWRITTEN `tests/e2e/test_suggestion_chips.py` (the phase-76/80 precedent: a semantic change rewrites the story suite in place), run in isolation: the opener-only core state (a 3-turn chat yields EXACTLY its opener as the single chip), the three-openers state, partial, seed, refetch-on-New-chat, plus the carried-over phase-05 behavior (one-tap submit, keyboard walk, the mobile single horizontal-scroll row).
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — the changed file is `app/api/suggestions.py`, every branch of the new walk covered by the matrix.
## Completion Criteria
- [ ] One saved chat with 4 user questions → the chip row holds EXACTLY its first question; none of the 3 follow-ups appears (integration + E2E pins).
- [ ] Four saved chats (each multi-turn) → exactly the 3 newest chats' OPENERS; the oldest opener is dropped by the cap; no follow-up text anywhere.
- [ ] Seed fallback, dedup, case-variant, partial (2 chips), brain-only/blank (A3), and 401 pins all green.
- [ ] `uv run pytest tests/e2e/test_suggestion_chips.py -v --no-cov` green in isolation; `test_responsive_polish.py` + `test_chat_persistence.py` green in isolation (regressions).
- [ ] The deflection "Maybe try" chips are UNCHANGED (`derive_suggestions` + its suites green).
- [ ] `uv run pytest` green; coverage >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One atomic `--no-gpg-sign` Conventional-Commits commit (e.g. `fix(chat): onboarding chips are the session-opening questions, never follow-ups`); phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,34 @@
# Task 01 — The `opening_questions` extraction + the integration matrix rewrite
**Phase:** `103_suggestions_session_openers` · **Story:** n/a (owner request)
## Objective
The endpoint's helper collects each saved chat's OPENING question (the first non-blank user message) instead of every user question — and the integration matrix pins the new semantics end to end.
## Work
1. `app/api/suggestions.py`:
- Rename `last_questions` → `opening_questions` (update the endpoint's call site; the helper is module-internal — no other importers; verify with `grep -rn "last_questions" app/ tests/`).
- Replace the REVERSE walk with the FORWARD opener walk (the 00_phase.md Design): for each chat in `updated_at DESC, created_at DESC` order, scan `chat.messages or []` oldest→newest; the first entry with `who == "user"` and a non-blank trimmed `text` is the chat's opener (A3 — a leading blank user entry does NOT disqualify the chat; keep scanning until a non-blank user message is found or the list is exhausted). Append the opener if not in `seen` (EXACT, case-sensitive — the phase-80 contract, the docstring keeps the why: case-insensitive dedup would drop a legitimately differently-cased re-ask). Stop once `limit` UNIQUE openers are collected; result in encounter order (newest chat first).
- Docstrings (house dense style): the module docstring's phase-80 paragraph → the opener contract + WHY (a follow-up like "What about X?" is meaningless as a conversation starter — cite the owner's qwen/llama.cpp example); the function docstring → the forward walk, the A3 rule, the A4 why-not-`SavedChat.title` note (120-char truncation + whitespace collapse at save, user-editable on re-Save — `app/api/chats.py` `_auto_title`), dedup/cap/order unchanged from phase 80; the endpoint docstring → "the opening questions of the 3 most recent saved chats — or, before any question has ever been saved, the seed list".
2. `tests/integration/test_suggestions_api.py` — REWRITE the matrix (keep the module's fixture/helper/auth scaffolding: the `clean_chats` autouse truncate, `_user`/`_brain`/`_add_chat`/`_chips`, the fixed `Q_*` texts — extend with a follow-up-flavored text such as `FOLLOW_UP = "What about qwen 3.6 35b?"` for the core pin):
- Module docstring → the new contract (openers only; everything else phase 80).
- `test_empty_db_returns_seed` — unchanged.
- REPLACE `test_cap_three_and_newest_first_within_a_chat` → `test_a_chats_follow_ups_never_surface`: ONE chat with 4 user questions (brain replies between, the 4th a follow-up-flavored text) → the chips are EXACTLY `[Q_ONE]` — the chat's opener only; Q_TWO/Q_THREE/Q_FOUR (the follow-ups) absent.
- NEW `test_cap_three_across_chats`: FOUR chats, each a multi-turn conversation (opener + at least one follow-up), DISTINCT explicit `updated_at` stamps → the chips are exactly the 3 NEWEST chats' openers, newest first; the oldest chat's opener is dropped (the cap now binds across chats); none of the four chats' FOLLOW-UPS appears anywhere.
- REWRITE `test_newer_chat_walked_first` → two multi-turn chats with DISTINCT `updated_at` → `[newer chat's opener, older chat's opener]`; the older chat's LAST (newest-looking) question is NOT in the chips.
- REWRITE the dedup pair → opener-flavored: the SAME opener text as the first question of two chats → exactly once (a verbatim re-ask as a FOLLOW-UP in the newer chat stays deduped too); a differently-cased OPENER variant → both kept (exact dedup).
- Keep the partial pins (2 chats → 2 openers; 1 chat → 1 chip — the follow-ups in those chats do not pad the row).
- Keep `test_brain_and_blank_user_texts_contribute_nothing` (all-brain + blank user → nothing).
- NEW (A3) `test_leading_blank_user_entry_does_not_disqualify`: a chat whose messages start `[user " ", user Q_TWO, …]` → contributes `Q_TWO` (the first NON-BLANK user message is the opener).
- Keep `test_all_brain_deployment_returns_seed` and `test_anonymous_is_401` unchanged.
3. Run `uv run pytest tests/integration/test_suggestions_api.py -v` (DB up: `podman compose up -d db`) — green.
## Testing & Quality
- Integration: the rewritten matrix IS this task's test layer — every branch of the new walk is exercised: zero chats (seed via the empty test), opener found / no non-blank user message (brain-only), leading-blank skip (A3), dedup hit, case variant kept, cap stop (across chats), encounter order.
- Coverage: **>90%** on this task's modified code — `app/api/suggestions.py` is fully covered by the matrix (the helper's every branch; the full-suite gate runs at the phase's end).
## Completion Criteria
- [ ] `opening_questions` walks FORWARD, takes ONE opener per chat (first non-blank user message), keeps the phase-80 order/dedup/cap/seed contracts; the old `last_questions` name is gone from `app/` and `tests/`
- [ ] The docstrings state the WHY (follow-ups are context-free) and the A3/A4 rules
- [ ] `uv run pytest tests/integration/test_suggestions_api.py -v` green (DB up)
- [ ] No other `app/` or `tests/` file changed (the E2E rewrite is task 03; docs are task 02)
@@ -0,0 +1,21 @@
# Task 02 — The "session openers" wording: config / `.env.example` / README
**Phase:** `103_suggestions_session_openers` · **Story:** n/a (owner request)
## Objective
The three doc surfaces that still say "the last 3 questions asked" (the phase-80 wording) describe the NEW contract — the chips are the session-OPENING questions; the seed's meaning ("shown only before any question has ever been saved") is unchanged.
## Work
1. `app/config.py` (~L373-377, the `suggestions` field comment): the seed docstring currently reads "shown ONLY while no saved chat has ever asked a question — after that, ``GET /api/suggestions`` serves the last 3 questions asked (deployment-wide, newest first)" → "serves the opening questions of the 3 most recent saved chats (the session openers — a chat's first user question; follow-ups never chip — phase 103; deployment-wide, newest first)". Keep the `BOR_SUGGESTIONS` override sentence.
2. `.env.example` (~L60, the `BOR_SUGGESTIONS` comment): "…# JSON seed chips — shown only before any question has been saved (phase 80)" → append "; after that the chips are the 3 newest chats' opening questions (phase 103)".
3. `README.md` (~L74, the chat-features line): "follow the last 3 questions asked — on a fresh deployment they seed from …" → "follow the opening questions of the 3 most recent saved chats (each chat's first question — a follow-up never chips, phase 103) — on a fresh deployment they seed from …" (keep the surrounding sentence intact; match the README's voice).
4. Grep sweep: `grep -rn "last 3 questions" app/ .env.example README.md` → ZERO hits (the phase-80 wording is fully retired; the phase records in `.agents/phases/complete/` may keep the historical phrasing — they are read-only and out of scope).
## Testing & Quality
- No new logic — doc/comment changes only; `uv run pytest tests/unit/test_config.py -q` green (the config docstring change is inert; `tests/unit/test_config.py` pins the seed list shape, untouched).
- Coverage: **>90%** on `app/` (untouched — the gate runs at the phase's end).
## Completion Criteria
- [ ] All three surfaces say "opening questions / session openers" (the WHY — follow-ups are meaningless without their session — appears at least in the `config.py` docstring)
- [ ] `grep -rn "last 3 questions" app/ .env.example README.md` → zero hits
- [ ] `uv run pytest tests/unit/test_config.py -q` green
@@ -0,0 +1,33 @@
# Task 03 — The story-suite rewrite to the opener semantics + full gate + atomic commit
**Phase:** `103_suggestions_session_openers` · **Story:** n/a (owner request)
## Objective
The dedicated Playwright suite proves the new contract in the browser (a multi-turn chat yields EXACTLY its opener as the single chip — the owner's "What about …?" follow-up can never surface), then the phase closes with every gate green and one atomic commit.
## Work
1. `tests/e2e/test_suggestion_chips.py` — REWRITE in place (the phase-76/80 precedent: a semantic change rewrites the story suite in place). Keep the module scaffolding (the `SEED` literal pin, the `auth_helpers.login` sign-in, the autouse `saved_chats` TRUNCATE fixture, the fixture-KB import, the mock-LLM marker constant, the run-in-isolation header). New module docstring: the opener contract + the four states. The states:
- **seed** (unchanged): fresh DB (no saved chats) → the chip texts equal the built-in default seed list EXACTLY, rendered as accessible buttons in the `role="list"` group (the phase-05 component contract).
- **opener-only** (the NEW core state): ONE saved chat with a 3-turn conversation — Q1 (the opener, e.g. "What are the correct arguments for qwen 3.8 27b on llama.cpp?") → brain → Q2 (a follow-up, e.g. "What about qwen 3.6 35b?") → brain → Q3 → brain — saved via the API (`POST /api/chats`) → a fresh page load shows EXACTLY ONE chip: Q1. Assert the chip count == 1 AND the exact text Q1; Q2/Q3 are absent (the owner's scenario, pinned).
- **three-openers** (replaces the old "last-3" state): THREE saved chats, each multi-turn (opener + at least one follow-up), DISTINCT `updated_at` (the API stamps them on save — save oldest→newest) → exactly 3 chips = the three openers, newest `updated_at` first; none of the chats' FOLLOW-UPS appears.
- **partial** (kept, re-scoped): exactly 2 saved chats → exactly 2 chips (the two openers — NO seed top-up; the follow-ups in those chats do not pad the row).
- **refetch** (kept): boot with the seed chips → save a multi-turn chat (opener Q) via the API → click New chat (`#new-chat-btn`) → the chips now are exactly Q, and the request log shows a SECOND `GET /api/suggestions` (the boot fetch was the first).
- Carried-over story behavior (unchanged semantics from the phase-05/80 suites): one-tap submit (chip click → composer filled → submitted → the mock-LLM brain bubble with the `MOCK_ANSWER_MARKER`), Tab+Enter keyboard reachability of the chips (the keyboard-walk assertion), and the mobile single horizontal-scroll row (the ≤640px viewport assertion).
2. Regression E2Es — run EACH in isolation (`--no-cov`, DB up), must stay green WITHOUT edits:
- `tests/e2e/test_responsive_polish.py` (chip visibility + the chip AA-contrast pair — it waits on `#suggestions .suggestion-chip` and measures colors; the seed chips on a truncated DB are still rendered),
- `tests/e2e/test_chat_persistence.py` (the deflection "Maybe try" chips + the empty-state chips after New chat — the separate contract, untouched).
If one of them asserts the OLD chip CONTENT semantics (not visibility/contrast/flow), update ONLY that assertion to the opener contract and note it in the commit message.
3. Full gate: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
4. `git diff --stat` — limited to `app/api/suggestions.py`, `tests/integration/test_suggestions_api.py`, `tests/e2e/test_suggestion_chips.py`, `app/config.py`, `.env.example`, `README.md`, and the phase files (no migrations, no `frontend/` diff, no other `app/` diff).
5. Move the phase dir to `.agents/phases/complete/` and make ONE atomic `--no-gpg-sign` Conventional-Commits commit (e.g. `fix(chat): onboarding chips are the session-opening questions, never follow-ups`).
## Testing & Quality
- E2E: the rewritten `tests/e2e/test_suggestion_chips.py` IS the phase's story suite — the opener-only state is the load-bearing pin (the owner's exact scenario); run in isolation per AGENTS.md rule 9.
- Coverage: **>90%** on `app/` — `app/api/suggestions.py` is the only `app/` delta and every branch of the new walk is covered (the integration matrix + the endpoint's seed fallback).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_suggestion_chips.py -v --no-cov` green in isolation (DB up) — including the opener-only state (1 chip for a 3-turn chat) and the three-openers state
- [ ] `test_responsive_polish.py` + `test_chat_persistence.py` green in isolation (with edits only if their assertions measured the old content semantics — noted in the commit)
- [ ] The deflection "Maybe try" chips are UNCHANGED (`app/rag/suggestions.py` untouched — `git diff` shows no delta there)
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean
- [ ] `git diff --stat` scoped per Work step 4; phase dir moved to `.agents/phases/complete/`; one atomic `--no-gpg-sign` commit
@@ -0,0 +1,106 @@
# Phase 104 — Suggestion chips stay single-line (hover reveals the full text) + the composer enforces the 4,000-char question cap
**Source:** Owner request (chat, 2026-09-12) — "…tweak the size of the suggestion chips on the chat page. Some users submit truly massive queries and the 'chips' become more like 'chonks'. Hovering over the chips should still show the full message so users still have a way to see it. In fact, I would suggest you put a character cap on the chat submission box, to prevent ultra-long context overflowing queries. Users should still be able to paste code example of a few dozen lines, but nothing much longer than that."
**Story:** n/a (owner request — extends the suggestion-chips component contract of `05_story_suggestion_chips` / `80_history_suggestion_chips`; the question-cap backstop pattern of `83_chat_save_payload_limits`).
**Context:** `frontend/assets/styles.css` — the base `.suggestion-chip` rule (~L1225: the pill, `min-height: 44px`, text WRAPS inside today — a 400-char question becomes a tall multi-line block), the `.maybe-try .suggestion-chip` override (~L702: `min-width: 0; max-width: 100%` — the phase-07 overflow fix), the mobile row (~L4331: `.suggestions { flex-wrap: nowrap; overflow-x: auto; }` + `.suggestion-chip { flex: 0 0 auto; }`). `frontend/assets/app.js` — `renderChips` (~L1003-1040: the shared chip component, onboarding row + "Maybe try" row; NO `title`/`aria-label` today), the source-chip `title` + `aria-label`-when-truncated pattern (~L1331-1338 — the house precedent to copy), `autoGrow` (~L1258, 192px max), the four `input.value` mutation sites (the `input` listener ~L2484, `submitSuggestion` ~L1013, the `handleSend` clear ~L2118, the `startNewChat` clear ~L2015), the out-of-turn `showErrorBanner` precedent (`saveAsDoc` ~L731). `frontend/index.html` — the composer (~L271-285: `#message-input` textarea with NO maxlength; the `.chat-bottom` sticky unit wraps the chat-actions row + the form, ~L205-215); the theme inputs' `maxlength`-mirrors-the-server comment precedent (~L1054). `app/schemas.py` — `ChatRequest.message: Field(min_length=1, max_length=4000)` (L75 — the server backstop ALREADY in force: a >4000-char question 422s today with zero UI feedback — the banner just shows "Brain's API answered with HTTP 422."); `HistoryTurn.text` stays 32_000 (HISTORY turns may be long answers — only the CURRENT question is capped at 4000). `tests/unit/test_schemas.py` — no 4000/4001 `message` boundary pin today. `tests/unit/test_pinned_composer.py` — pins `.chat-bottom` as the LAST CHILD of `.chat-shell` and the `.composer` sticky CSS (a new child INSIDE `.chat-bottom` breaks neither).
## Objective
Two coupled fixes on the chat page. (1) Long questions no longer balloon the suggestion chips into multi-line "chonks": every chip is single-line, ellipsized at the row edge, and the FULL text is one hover away (native `title` tooltip) plus the accessible name. (2) The composer makes the question-length cap VISIBLE — the server already hard-caps the current question at 4,000 chars (`ChatRequest.message`); the textarea gains `maxlength="4000"`, a counter appears near the cap, and a guard covers the one path that bypasses `maxlength` (the chip one-tap fill) — so no user ever meets the 422 blind.
## Owner decisions (chat, 2026-09-12 — recorded per AGENTS.md rule 3)
- **A1 — single-line chips, never chonks:** chips never wrap — `white-space: nowrap`, ellipsized at the row edge, never taller than the one-line 44px pill, at every viewport width.
- **A2 — hover shows the full text:** "Hovering over the chips should still show the full message so users still have a way to see it" — a native `title` tooltip carrying the FULL text on every chip (the house source-chip pattern), plus `aria-label` = full text when the chip is visually truncated (screen readers).
- **A3 — the question cap is 4,000 chars, mirroring the existing server cap:** the cap must allow "code example of a few dozen lines, but nothing much longer" — 4,000 chars ≈ a 50-line block at 80 chars/line, and the server ALREADY rejects >4,000 (`ChatRequest.message max_length=4000`, pre-existing, untouched). **NO schema change:** the server cap stays the backstop (this phase pins it at the boundary, task 03); the UI becomes the visible contract (`maxlength` + counter + guard).
- **A4 — counter behavior:** hidden while the RAW length < 3,200 (80% of the cap — no noise on normal use); shows `len/4000` from 3,200; at/over the cap shows `len/4000 — character limit` in the `--err-*` semantic family (B3: the COPY change carries the state — text + color, never color alone; the executor verifies + records the AA ratio of the chosen `--err-*` pairing in the CSS comment). Count the RAW value (no trim): raw ≤ 4,000 ⟹ trimmed ≤ 4,000, so a raw count is a safe superset of what the server validates.
- **A5 — the over-cap guard:** `maxlength` constrains typing + pastes, but a programmatic `input.value = …` bypasses it — the one reachable path is `submitSuggestion` (a chip >4,000 chars; possible only via an admin-authored `BOR_SUGGESTIONS` seed — history chips are ≤4,000 by construction, having passed the same cap when asked). `handleSend` guards: trimmed text > the cap → the out-of-turn error banner (the `saveAsDoc` precedent), NO turn, the input KEEPS the text (the user trims it) — never stale (PLAN §7.4).
- **A6 — shared page:** `frontend/assets/shared.js` renders "Maybe try" chips as plain non-interactive spans (`pointer-events: none`, scoped `.shared-shell` — owner-locked phase 51: a guest tapping a chip has nowhere to go; a tooltip could never show) — UNTOUCHED; the CSS sizing applies to those pills automatically (shared stylesheet).
## Design (shared by all tasks — the executor reads this, not the chat)
### Chip sizing (task 01) — `frontend/assets/styles.css`
- The base `.suggestion-chip` rule gains: `white-space: nowrap; overflow: hidden; text-overflow: ellipsis; max-width: 100%; min-width: 0;`
- `overflow: hidden` (≠ visible) zeroes the flex item's automatic minimum size, so `max-width: 100%` actually binds: in the desktop wrap row (`.suggestions`, the 46rem column) a long chip clips at the column edge; in the ≤640px row (`flex-wrap: nowrap; overflow-x: auto`) it clips at the VISIBLE width and the row scrolls (the phase-07 contract).
- One line per chip at every width — the 44px `min-height` pill stays a pill.
- The `.maybe-try .suggestion-chip` override (`min-width: 0; max-width: 100%`) is fully subsumed by the new base rule → **delete it**, folding its phase-07 overflow-fix provenance into the base rule's comment (house: provenance lives with the contract).
- The mobile rule `.suggestion-chip { flex: 0 0 auto; }` (≤640px block) stays.
- No other CSS in this task.
### Tooltip + a11y (task 02) — `frontend/assets/app.js` `renderChips`
- After `btn.textContent = text;`: `btn.title = text;` — the FULL text, always (A2; the house source-chip pattern, ~L1331).
- After `container.appendChild(btn)`: `if (btn.scrollWidth > btn.clientWidth) btn.setAttribute("aria-label", text);` — the source-chip truncation pattern (~L1336-1338): the screen-reader name is the full text when the visible text is clipped; attribute absent when not clipped (textContent already carries the full text).
- No other `renderChips` change (one-tap submit, `role="listitem"`, the container contract).
### The composer cap (task 03) — `frontend/index.html` + `frontend/assets/app.js`
- `index.html`:
- `#message-input` gains `maxlength="4000"`, with the provenance comment (house pattern — the theme inputs' "maxlength=300 mirrors the server's 300-char"): "maxlength=4000 mirrors ChatRequest.message max_length=4000 (app/schemas.py) — the server 422s beyond; the #char-count line makes the cap visible (app.js updateCharCount)".
- The counter element, INSIDE `.chat-bottom` between the chat-actions row and `<form class="composer">` (a new child of the sticky unit — the sticky contract + `test_pinned_composer.py`'s last-child-of-`.chat-shell` pin are untouched):
```html
<p class="char-count" id="char-count" hidden></p>
```
with the phase-104 comment: hidden until 80% of the cap (app.js); the `.is-max` state uses the `--err-*` family + a copy change (B3); NOT a live region — per-keystroke feedback is decorative, the over-cap failure announces through the error banner (`role=alert`).
- `app.js`:
- Constants next to `autoGrow` (provenance comment: mirrors the schema cap; the threshold is 80% — owner 2026-09-12 A4):
```js
const MAX_QUESTION_CHARS = 4000;
const CHAR_COUNT_SHOW_AT = 3200;
```
- Element grabber alongside the existing ones (`~L301`): `const charCountEl = document.querySelector("#char-count");`
- `updateCharCount()` (next to `autoGrow`): `len = input.value.length` (RAW); `len < CHAR_COUNT_SHOW_AT` → `hidden = true` + drop `.is-max`; else unhide, `textContent = "${len}/${MAX_QUESTION_CHARS}"` and, when `len >= MAX_QUESTION_CHARS`, append ` — character limit` + add `.is-max`. Over-cap (the chip path) shows the honest `len` (e.g. `5123/4000 — character limit`).
- Call `updateCharCount()` at the EXACT four `input.value` mutation sites (each already calls `autoGrow()` there): the `input` listener, `submitSuggestion`, the `handleSend` clear, the `startNewChat` clear.
- `handleSend` guard, immediately AFTER `if (!text || sendBtn.disabled) return;` (BEFORE the clear — the input keeps the text):
```js
if (text.length > MAX_QUESTION_CHARS) {
showErrorBanner("Questions are limited to 4,000 characters — trim the question and try again.");
return;
}
```
with the A5 comment (maxlength caps typing + pastes; the programmatic chip fill bypasses it — this guard is the never-stale backstop).
- `styles.css` (this task): `.char-count` — `margin: 0; text-align: right; font-size: 0.75rem; line-height: 1.2; color: var(--ink-soft);` (the counter sits on the app background behind `.chat-bottom` — the executor verifies the chosen pairing is ≥4.5:1 there and records the ratio in the comment, house style); `.char-count.is-max { color: var(--err-ink); }` (verify + record the ratio on the same background; the copy change already carries the state — B3).
- `tests/unit/test_schemas.py`: the `ChatRequest.message` boundary pin — exactly 4,000 chars validates; 4,001 → a pydantic `ValidationError` naming `message` (the A2/A3 backstop — the cap the UI now mirrors).
### Unit source pins (house pattern — one new file, `tests/unit/test_chip_sizing_question_cap.py`, extended per task)
- Task 01 (CSS): the `.suggestion-chip` rule block contains `white-space: nowrap`, `overflow: hidden`, `text-overflow: ellipsis`, `max-width: 100%`, `min-width: 0`; `css.count(".maybe-try .suggestion-chip") == 0` (the subsumed override is gone); the ≤640px block keeps `.suggestion-chip { flex: 0 0 auto; }`.
- Task 02 (renderChips): the `renderChips` block contains `btn.title = text` and the `scrollWidth > btn.clientWidth` → `setAttribute("aria-label"` pattern.
- Task 03: `index.html` — the `#message-input` block carries `maxlength="4000"`; `#char-count` exists, is `hidden` by default, and sits inside `.chat-bottom` before `#composer` (source order). `app.js` — `MAX_QUESTION_CHARS = 4000` + `CHAR_COUNT_SHOW_AT = 3200`; the guard `text.length > MAX_QUESTION_CHARS` + the banner copy "4,000 characters"; `updateCharCount()` defined AND called at all four mutation sites (pin each site's context). **Single-source cross-file pin:** the HTML `maxlength` value == the JS `MAX_QUESTION_CHARS` value (regex-parse both files and compare — the cap lives in one place conceptually).
### E2E (task 04) — new dedicated suite `tests/e2e/test_chip_sizing_question_cap.py`
House scaffolding (DB up, mock LLM, the fixture-KB module import, admin login — the chips are `require_user`, the phase-80 autouse `saved_chats` truncate):
1. **Truncated chip + tooltip (A1/A2 core):** save via the API a chat whose FIRST user question is LONG (300+ chars — a readable repeated phrase) with a short follow-up; reload → exactly ONE onboarding chip (phase-103 opener semantics):
- computed style `white-space: nowrap`, `overflow: hidden`, `text-overflow: ellipsis`;
- `scrollWidth > clientWidth` (visually clipped);
- single line: `44 <= clientHeight <= 60` (a one-line pill is the 44px min-height; a wrapped two-liner is ≥ ~76px);
- `title` attribute == the full long text; `aria-label` == the full long text.
Fresh-DB contrast pin: a short SEED chip has `title` set and NO `aria-label` (not truncated).
2. **Counter threshold (A4):** `#char-count` hidden at boot; 100 chars typed → still hidden; exactly 3,500 chars in the box (a dispatched `input` event) → visible, text `3500/4000`, NO `.is-max`.
3. **Hard cap through the input path (A3):** `keyboard.insert_text("x" * 6000)` (CDP `Input.insertText` = the paste path — `maxlength` applies) → the textarea holds EXACTLY 4,000 chars; the counter reads `4000/4000 — character limit` + `.is-max`. Submit → the 4,000-char question passes the server cap (NO 422 error state) → the mock answer streams to `done` → the input is cleared and the counter hidden again.
4. **The over-cap guard (A5):** `page.evaluate` sets `#message-input.value = "x".repeat(5000)` + dispatches an `input` event (the programmatic path `maxlength` cannot stop) → counter `5000/4000 — character limit` + `.is-max` → click Send → the error banner shows the "4,000 characters" copy; NO brain bubble appended; the input STILL holds the 5,000 chars (kept for trimming — never stale).
5. **Short-flow regression:** a short question submits cleanly; the counter never becomes visible.
**Regressions (run in isolation, must stay green):** `tests/e2e/test_suggestion_chips.py` (the chip contract — phase 103's rewrite), `tests/e2e/test_pinned_composer.py` (the sticky cluster now hosts the counter), `tests/e2e/test_responsive_polish.py` (the mobile chip row + the chip AA-contrast pairs — the chip colors are unchanged), `tests/e2e/test_chat_history.py` (the send flow). Unit: `tests/unit/test_pinned_composer.py` (the `.chat-bottom` pins — untouched structure) + any `tests/unit/` source pin that conflicts with the added `app.js` lines (the executor runs `uv run pytest tests/unit/ -q` and fixes only genuine conflicts — the additions live INSIDE existing functions, so substring/context pins should survive).
## Dependencies
- `103_suggestions_session_openers` (todo) — the onboarding chips become session openers; this phase's long-chip E2E state (a 300+ char opener chip) builds on that contract and runs AFTER it. Queue order only at the code level (different files), but the E2E fixtures assume the phase-103 semantics.
- `80_history_suggestion_chips` (complete) — the chip component + the E2E fixture pattern.
- `83_chat_save_payload_limits` (complete) — the boundary-cap pin pattern (`tests/unit/test_schemas.py`).
## Tasks
1. `01_chip_ellipsis_css.md` — the base chip rule (single-line ellipsis) + the subsumed-override deletion + the CSS unit pins.
2. `02_chip_tooltip_aria.md` — `renderChips` full-text `title` + `aria-label`-when-truncated + the unit pins.
3. `03_composer_question_cap.md` — `maxlength` + the counter (HTML/JS/CSS) + the `handleSend` guard + the unit pins + the `test_schemas.py` boundary pin.
4. `04_e2e_suite_commit.md` — the dedicated E2E suite + regression E2Es + full gate + atomic commit.
## Testing & Quality
- Unit — `tests/unit/test_chip_sizing_question_cap.py` (new; per-task pins above) + the `ChatRequest.message` 4,000/4,001 boundary in `tests/unit/test_schemas.py`.
- E2E (mandatory, A16) — `tests/e2e/test_chip_sizing_question_cap.py` green in isolation (the five states above); the four named regression suites green in isolation.
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — the `app/` delta is nil-to-trivial (no `app/` code changes; the gate stays green).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] A 300+ char onboarding chip renders single-line (`clientHeight` ≤ 60px, ellipsized, `scrollWidth > clientWidth`) with `title` + `aria-label` == the full text (E2E).
- [ ] `#message-input` hard-caps at 4,000 through the paste path; submitting at the cap streams a mock answer (no 422); the counter is hidden again after send (E2E).
- [ ] The over-cap guard: a programmatic 5,000-char fill + Send → the cap banner, no turn, the text kept (E2E).
- [ ] The `ChatRequest.message` 4,000/4,001 boundary is pinned in `tests/unit/test_schemas.py`; the HTML `maxlength` == the JS constant (cross-file pin).
- [ ] `uv run pytest tests/e2e/test_chip_sizing_question_cap.py -v --no-cov` green in isolation; `test_suggestion_chips.py`, `test_pinned_composer.py`, `test_responsive_polish.py`, `test_chat_history.py` green in isolation.
- [ ] `uv run pytest` green; coverage >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `git diff --stat` limited to `frontend/`, the new unit file, `tests/unit/test_schemas.py`, the new E2E file, phase files (NO `app/` code diff, no migration, no `shared.js` diff).
- [ ] One atomic `--no-gpg-sign` Conventional-Commits commit (e.g. `feat(chat): single-line suggestion chips with full-text tooltips + the visible 4,000-char question cap`); phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,33 @@
# Task 01 — Single-line ellipsized chips: the base `.suggestion-chip` rule + the subsumed override deletion
**Phase:** `104_chip_sizing_question_cap` · **Story:** n/a (owner request)
## Objective
A suggestion chip is ONE line at every viewport width — long text ellipsizes at the row edge instead of wrapping the pill into a multi-line "chonk" (owner A1).
## Work
1. `frontend/assets/styles.css` — the base `.suggestion-chip` rule (~L1225) gains four declarations (keep the existing ones — font/weight/color/background/border/radius/padding/min-height/transition):
```css
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
max-width: 100%;
min-width: 0;
```
Extend the rule's comment (house provenance style): phase 104 (owner 2026-09-12) — chips never wrap; `overflow: hidden` zeroes the flex item's automatic minimum size so `max-width: 100%` binds (desktop wrap row: 100% = the 46rem column; the ≤640px row: 100% = the visible width, the row scrolls — the phase-07 overflow contract now lives here).
2. DELETE the `.maybe-try .suggestion-chip { min-width: 0; max-width: 100%; }` override (~L702) — fully subsumed by the new base rule; fold its phase-07 provenance sentence into the base rule's comment (the step-1 comment) so the history is not lost.
3. Leave the ≤640px block's `.suggestion-chip { flex: 0 0 auto; }` (~L4331) and `.suggestions` row rule untouched.
4. `tests/unit/test_chip_sizing_question_cap.py` — CREATE the phase's unit-pins file (module docstring: pins the phase-104 chip-sizing contract in the static frontend — the house source-pin pattern) with the CSS pins:
- the `.suggestion-chip` rule block contains `white-space: nowrap`, `overflow: hidden`, `text-overflow: ellipsis`, `max-width: 100%`, `min-width: 0` (reuse the `_rule`-style block extraction from `tests/unit/test_pinned_composer.py` / `test_wide_column_css.py` — read one and match its helpers);
- `css.count(".maybe-try .suggestion-chip") == 0` (the subsumed override is gone);
- the ≤640px mobile block still contains `.suggestion-chip { flex: 0 0 auto; }`.
5. Run `uv run pytest tests/unit/ -q` — green (the new file + no existing pin broken).
## Testing & Quality
- Unit: the new pins ARE this task's test layer (they guard the CSS bytes — the house source-pin pattern).
- Coverage: **>90%** on `app/` (untouched — the full-suite gate runs at the phase's end).
## Completion Criteria
- [ ] `.suggestion-chip` is single-line + ellipsized + capped at the row edge (the five declarations pinned); the `.maybe-try .suggestion-chip` override deleted with its provenance folded into the base comment
- [ ] `tests/unit/test_chip_sizing_question_cap.py` created and green; `uv run pytest tests/unit/ -q` green
- [ ] no other file changed (JS/HTML/E2E are later tasks)
@@ -0,0 +1,30 @@
# Task 02 — Full-text hover tooltip + accessible name on every suggestion chip
**Phase:** `104_chip_sizing_question_cap` · **Story:** n/a (owner request)
## Objective
The full text of a (possibly ellipsized) chip is always recoverable: a native `title` tooltip on hover (owner A2) and the full text as the accessible name when the visible text is clipped (the house source-chip pattern).
## Work
1. `frontend/assets/app.js` — `renderChips` (~L1024-1040), inside the per-chip loop:
- after `btn.textContent = text;` add `btn.title = text;` — the FULL text, always (the hover contract; the source-chip precedent is `chip.title = label` at ~L1331).
- after `container.appendChild(btn);` add the truncation-aware accessible name (the source-chip precedent at ~L1336-1338):
```js
if (btn.scrollWidth > btn.clientWidth) btn.setAttribute("aria-label", text);
```
(When NOT clipped the attribute stays absent — the `textContent` already carries the full text, so screen readers read it; the attribute is belt-and-suspenders for the clipped case, exactly like the source chips.)
- One short comment on the pair: phase 104 (owner 2026-09-12) — the single-line chip clips long questions; `title` is the hover reveal, `aria-label` the clipped-case accessible name (the source-chip pattern).
- Touch NOTHING else in `renderChips` (one-tap submit via `submitSuggestion`, `role="listitem"`, the container-replace contract, the `onSelect` hook).
2. `tests/unit/test_chip_sizing_question_cap.py` — extend with the `renderChips` pins (extract the `renderChips` function body from `app.js` the way the neighboring unit files do):
- the body contains `btn.title = text`;
- the body contains the `btn.scrollWidth > btn.clientWidth` guard setting `aria-label` (pin the `setAttribute("aria-label"` call inside that guard — a small slice of the function text, the house "pin the contract words" style).
3. Run `uv run pytest tests/unit/ -q` — green (including the existing `tests/unit/test_shared_page.py` pin that `renderChips` does NOT leak into `shared.js` — `shared.js` is untouched by this task).
## Testing & Quality
- Unit: the new pins guard the `app.js` bytes; the full E2E hover/tooltip behavior is task 04.
- Coverage: **>90%** on `app/` (untouched).
## Completion Criteria
- [ ] every chip rendered by `renderChips` (onboarding row AND "Maybe try" row) carries `title` = the full text, and `aria-label` = the full text when clipped
- [ ] the unit pins are green; `uv run pytest tests/unit/ -q` green
- [ ] no other file changed
@@ -0,0 +1,91 @@
# Task 03 — The visible 4,000-char question cap: `maxlength` + counter + the over-cap guard + the server-boundary pin
**Phase:** `104_chip_sizing_question_cap` · **Story:** n/a (owner request)
## Objective
The question-length cap the server ALREADY enforces (`ChatRequest.message max_length=4000` — a >4,000-char question 422s today with zero UI feedback) becomes VISIBLE in the composer: the textarea hard-caps input/paste, a counter appears near the cap, and a guard covers the one path that bypasses `maxlength` (the chip one-tap fill). The server cap is untouched and pinned at the boundary.
## Work
1. `frontend/index.html`:
- `#message-input` (~L273-279) gains `maxlength="4000"`, with the provenance comment (house pattern — the theme inputs' "maxlength=300 mirrors the server's 300-char", ~L1054): `maxlength=4000 mirrors ChatRequest.message max_length=4000 (app/schemas.py) — the server 422s beyond; the #char-count line makes the cap visible (app.js updateCharCount)`.
- The counter element — INSIDE `.chat-bottom` (~L215), between the closing `</div>` of `.chat-actions` and the composer comment/`<form class="composer">`:
```html
<!-- Phase 104 (owner 2026-09-12): the question-length counter —
hidden until 80% of the 4,000-char cap (app.js
updateCharCount); the .is-max state uses the --err-* family
PLUS a copy change (B3: text + color, never color alone). NOT
a live region: per-keystroke feedback is decorative — the
over-cap failure path announces through the error banner
(role=alert). -->
<p class="char-count" id="char-count" hidden></p>
```
(A new child of the sticky unit — the unit stays the LAST child of `.chat-shell` and the `.composer` sticky CSS is untouched, so `tests/unit/test_pinned_composer.py` + `tests/e2e/test_pinned_composer.py` stay green; a `hidden` `<p>` adds zero height.)
2. `frontend/assets/app.js`:
- Constants directly above `autoGrow` (~L1258) with the provenance comment (mirrors the schema cap — the executor must NOT change `app/schemas.py`; the threshold is 80% of the cap, owner A4):
```js
const MAX_QUESTION_CHARS = 4000;
const CHAR_COUNT_SHOW_AT = 3200; // 80% of the cap — visible only when it matters
```
- Element grabber alongside the existing ones (next to `const suggestionsEl = …` ~L301): `const charCountEl = document.querySelector("#char-count");`
- `updateCharCount()` next to `autoGrow`:
```js
function updateCharCount() {
// RAW length (no trim): raw ≤ cap ⟹ trimmed ≤ cap, so the raw
// count is a safe superset of what the server validates.
const len = input.value.length;
if (len < CHAR_COUNT_SHOW_AT) {
charCountEl.hidden = true;
charCountEl.classList.remove("is-max");
return;
}
charCountEl.hidden = false;
const atMax = len >= MAX_QUESTION_CHARS;
charCountEl.classList.toggle("is-max", atMax);
charCountEl.textContent = atMax
? `${len}/${MAX_QUESTION_CHARS} — character limit`
: `${len}/${MAX_QUESTION_CHARS}`;
}
```
(Over-cap — reachable only via the programmatic chip-fill path — shows the honest `len`, e.g. `5123/4000 — character limit`.)
- Call `updateCharCount()` at the EXACT four `input.value` mutation sites (each already calls `autoGrow()` — add the call right after it, or fold both into the listener body):
a. the `input` listener (~L2484: `input.addEventListener("input", autoGrow)` — e.g. `input.addEventListener("input", () => { autoGrow(); updateCharCount(); })`);
b. `submitSuggestion` (~L1013-1015, after `input.value = text; autoGrow();`);
c. `handleSend` (~L2118-2119, after the post-send clear `input.value = ""; autoGrow();`);
d. `startNewChat` (~L2015-2016, after `input.value = ""; autoGrow();`).
- The `handleSend` guard — immediately AFTER `if (!text || sendBtn.disabled) return;` (~L2114) and BEFORE the clear (the input keeps the text for trimming):
```js
// maxlength caps typing + pastes, but a programmatic fill (the chip
// one-tap path) bypasses it — this guard is the never-stale backstop
// (PLAN §7.4): no turn, no clear, the user trims the kept text.
if (text.length > MAX_QUESTION_CHARS) {
showErrorBanner("Questions are limited to 4,000 characters — trim the question and try again.");
return;
}
```
(Out-of-turn banner = the `saveAsDoc` precedent, ~L731; the banner is cleared by the next user action — the existing `clearErrorBanner` call sites.)
3. `frontend/assets/styles.css` — the counter rules (near the composer styles, ~L1242+):
```css
/* Phase 104: the question-length counter — right-aligned above the
composer, hidden until 80% of the 4,000-char cap (app.js).
[executor: verify + record the ratio] --ink-soft on the app
background behind .chat-bottom is ≥4.5:1 (WCAG AA). */
.char-count { margin: 0; text-align: right; font-size: 0.75rem; line-height: 1.2; color: var(--ink-soft); }
.char-count.is-max { color: var(--err-ink); }
```
Verify BOTH pairings against the actual background the counter sits on (the app `--bg` behind `.chat-bottom` — the chat column area) and record each ratio in the comment (house style). The `.is-max` state pairs the color with the "— character limit" COPY change (B3 — never color alone).
4. `tests/unit/test_chip_sizing_question_cap.py` — extend with:
- `index.html` pins: the `#message-input` textarea block carries `maxlength="4000"`; `#char-count` exists, is `hidden` by default, and appears INSIDE `.chat-bottom` before `#composer` (source order).
- `app.js` pins: `MAX_QUESTION_CHARS = 4000` and `CHAR_COUNT_SHOW_AT = 3200`; the guard `text.length > MAX_QUESTION_CHARS` with the banner copy "4,000 characters"; `updateCharCount` defined AND its call present in each of the four mutation-site contexts (the listener, `submitSuggestion`, `handleSend`, `startNewChat` — pin each site's slice).
- **Single-source cross-file pin:** regex-parse the HTML `maxlength="(\d+)"` on the `#message-input` block and the JS `MAX_QUESTION_CHARS = (\d+)` and assert they are EQUAL (the cap lives in one place conceptually — the schema is the source, both mirror it).
5. `tests/unit/test_schemas.py` — the `ChatRequest.message` boundary pin (currently unpinned): exactly 4,000 chars validates; 4,001 → a pydantic `ValidationError` naming `message` (the backstop the UI now mirrors — the phase-83 boundary-pin pattern).
6. Run `uv run pytest tests/unit/ -q` — green.
## Testing & Quality
- Unit: the new pins (HTML/JS/CSS bytes + the cross-file constant match + the schema boundary) ARE this task's test layer; the behavioral E2E is task 04.
- Coverage: **>90%** on `app/` (no `app/` code changes — `app/schemas.py` is untouched, only test-pinned).
## Completion Criteria
- [ ] `#message-input` has `maxlength="4000"` + the provenance comment; the counter element sits in `.chat-bottom` above the composer, hidden by default
- [ ] `updateCharCount` fires at all four mutation sites; the `handleSend` guard (banner + no turn + no clear) is in place; the counter CSS is AA-verified with recorded ratios
- [ ] The HTML `maxlength` == the JS `MAX_QUESTION_CHARS` (cross-file pin); the 4,000/4,001 `ChatRequest.message` boundary is pinned in `test_schemas.py`
- [ ] `uv run pytest tests/unit/ -q` green; no `app/` file changed
@@ -0,0 +1,41 @@
# Task 04 — The dedicated E2E suite + regression E2Es + full gate + atomic commit
**Phase:** `104_chip_sizing_question_cap` · **Story:** n/a (owner request)
## Objective
The browser proves the whole contract — single-line ellipsized chips with full-text tooltips, the hard 4,000-char cap through the paste path, the counter states, and the over-cap guard — then the phase closes with every gate green and one atomic commit.
## Work
1. `tests/e2e/test_chip_sizing_question_cap.py` — NEW dedicated suite (house scaffolding: module docstring stating the phase-104 contract + the run-in-isolation command; DB up `podman compose up -d db`; the deterministic mock LLM — the fixture-KB import pattern from a sibling chat suite, e.g. `tests/e2e/test_suggestion_chips.py`; admin login via `e2e.auth_helpers.login` — the chips are `require_user`; the phase-80 autouse `saved_chats` TRUNCATE fixture so each test starts from — and leaves — an empty deployment):
- **`test_long_chip_is_single_line_ellipsized_with_full_text_tooltip`** (A1/A2 core): save via the API a chat whose FIRST user question is LONG (300+ chars — a readable repeated phrase, e.g. `"What are the correct arguments for " * 20 + "qwen on llama.cpp?"`) with one short follow-up turn; reload the chat page → exactly ONE onboarding chip (the phase-103 opener semantics — the follow-up never surfaces):
- computed style: `white-space: nowrap`, `overflow: hidden`, `text-overflow: ellipsis`;
- `scrollWidth > clientWidth` (visually clipped — 300+ chars of ~0.5rem/char far exceeds the 46rem column);
- single line: `44 <= clientHeight <= 60` (a one-line pill sits at the 44px `min-height`; a wrapped two-liner is ≥ ~76px — the chonk);
- `get_attribute("title")` == the full long text (the hover reveal);
- `get_attribute("aria-label")` == the full long text (the clipped-case accessible name).
- **`test_short_seed_chip_has_tooltip_but_no_aria_label`**: fresh DB (seed chips) → a short chip has `title` set AND no `aria-label` (not truncated — the attribute is absent by design).
- **`test_counter_hidden_below_threshold_and_visible_above`** (A4): on the empty-state chat page, `#char-count` is hidden; type 100 chars → still hidden; put exactly 3,500 chars in `#message-input` (a dispatched `input` event — `locator.fill` does this) → `#char-count` visible, text `3500/4000`, NO `.is-max` class.
- **`test_paste_path_hard_caps_at_the_cap_and_sends`** (A3): `page.keyboard.insert_text("x" * 6000)` (CDP `Input.insertText` = the paste path — `maxlength` applies) → `#message-input` holds EXACTLY 4,000 chars; the counter reads `4000/4000 — character limit` + `.is-max`. Click Send → NO 422 error state (the 4,000-char question passes the server cap) → the mock answer streams to `done` (the brain bubble + the "Deterministic mock answer for E2E" marker) → `#message-input` cleared and `#char-count` hidden again.
- *Executor note:* if `insert_text` proves not to respect `maxlength` on the pinned Chromium build (it goes through the browser's input pipeline, like a paste — expect it to work), fall back to pinning the attribute (`maxlength == "4000"`) + a `fill`-based counter check, and record the deviation in the phase record — the guard test below still covers the bypass path.
- **`test_over_cap_programmatic_fill_hits_the_guard`** (A5): `page.evaluate` sets `#message-input.value = "x".repeat(5000)` + dispatches an `input` event (the programmatic path `maxlength` cannot stop — the chip one-tap fill) → counter `5000/4000 — character limit` + `.is-max` → click Send → the error banner shows the "4,000 characters" cap copy; NO brain bubble appended; `#message-input` STILL holds the 5,000 chars (kept for trimming — never stale, PLAN §7.4).
- **`test_short_flow_never_shows_the_counter`**: type a short question → submit → the mock answer lands; `#char-count` never becomes visible during the turn.
2. Regression E2Es — run EACH in isolation (`--no-cov`, DB up), must stay green:
- `tests/e2e/test_suggestion_chips.py` (the chip contract — phase 103's rewrite; the onboarding row this phase restyles),
- `tests/e2e/test_pinned_composer.py` (the sticky cluster now hosts the counter),
- `tests/e2e/test_responsive_polish.py` (the mobile chip row + the chip AA-contrast pairs — the chip colors are unchanged by this phase),
- `tests/e2e/test_chat_history.py` (the send/save flow).
Fix ONLY a regression whose assertion measured the OLD chip wrapping (e.g. a pin that asserted a multi-line chip height) — the asserted BEHAVIOR (a chip exists, is clickable, AA contrast) must survive; note any such fix in the commit message.
3. Full gate: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
4. `git diff --stat` — limited to `frontend/`, `tests/unit/test_chip_sizing_question_cap.py`, `tests/unit/test_schemas.py`, `tests/e2e/test_chip_sizing_question_cap.py`, and the phase files (NO `app/` code diff, no migration, no `shared.js` diff, no `pyproject.toml`/`uv.lock`).
5. Move the phase dir to `.agents/phases/complete/` and make ONE atomic `--no-gpg-sign` Conventional-Commits commit (e.g. `feat(chat): single-line suggestion chips with full-text tooltips + the visible 4,000-char question cap`).
## Testing & Quality
- E2E: the new suite (five tests above) IS the phase's story suite — run in isolation per AGENTS.md rule 9; the four regression suites re-prove the untouched contracts (chip component, sticky composer, contrast, send flow).
- Coverage: **>90%** on `app/` (the `app/` delta is nil — the gate must simply stay green).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_chip_sizing_question_cap.py -v --no-cov` green in isolation (DB up)
- [ ] `test_suggestion_chips.py`, `test_pinned_composer.py`, `test_responsive_polish.py`, `test_chat_history.py` each green in isolation
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean
- [ ] `git diff --stat` scoped per Work step 4 (no `app/` code diff, no `shared.js` diff)
- [ ] Phase dir moved to `.agents/phases/complete/`; one atomic `--no-gpg-sign` commit
@@ -0,0 +1,78 @@
# Phase 105 — Per-source hidden-folders toggle: dot-prefixed paths become indexable per input
**Source:** `TODO.md` L3 — "hidden dot folders aren't being indexed. There should be a toggle per input (next to the ignores button) to allow indexing hidden .folders."
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-14).
**Context:** The single filesystem walk choke point is `iter_importable_files` (`app/rag/importer.py` L161-194): it skips any path with a dot-prefixed component — hidden dirs (vendored caches like `.esphome/.espressif/**`) AND hidden files — plus the well-known `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`); the extension filter (`BOR_IMPORT_EXTENSIONS`, the A9 family) then admits the rest. `import_sources` (L196+) walks every root twice when `progress` is set (the phase-64 pre-walk for the `total` denominator uses the EXACT same rules), collects `(source, rel)` into `seen`, and `_prune` (`prune=True`) deletes every indexed document of the imported sources whose `(source, rel)` is not in `seen` — the same mechanism phase 89 uses so newly-ignored files leave the index. Both live import entry points build a per-root `ignore_by_root: dict[str, list[str]]` map from the `git_sources` rows in one loop — the Sync button (`app/api/sync.py::_run_sync` L233-263) and the CLI (`scripts/import_docs.py::_resolve_sources` L181-226, consumed at L269/L332-334); the map is keyed by `str(root)` — the root path string exactly as passed to `import_sources` — with extend-on-collision for shared roots. `scripts/load_test_kb.py` calls `import_sources` with defaults (untouched). The archive-upload background run no longer scans (phase 90), so no map is needed there. Sources are the `git_sources` rows (kind `git` / `local`, phase 35/38) managed on the Sources page (`/git-sources.html`, view module `frontend/assets/git-sources.js`, skeleton in `frontend/index.html` `#view-git-sources` L514+); the per-row "Ignore paths" button (`makeRow` L357+, `ignoreBtn` ~L393-408) sits in the actions cell LEFT of Remove, stored rows only (env-fallback rows, `id` null, get no control — phase 89 A3); non-empty lists render a `N ignored` text tag in the source cell (`.git-source-ignore-count`, `frontend/assets/styles.css` L2686, text + background, never color alone). The admin-only `PATCH /api/git-sources/{source_id}` (phase 89 A5, `app/api/git_sources.py` L374-393) today takes the REQUIRED `ignore_paths` replace list; the read shapes are `GitSourceRow` (`app/schemas.py` L508) and `GitSourceOut` (L490), the create body `GitSourceIn` (L450, optional `ignore_paths`). Alembic head is `0018`. The KB tree/catalog (phase 97) and the agent tools read the DB, so newly indexed hidden documents appear in them automatically — no change needed.
## Objective
Each stored source carries an **index-hidden-folders flag**, toggled by a per-row checkbox next to the "Ignore paths" button on the Sources page. When ON for a source, the walk no longer skips dot-prefixed components for that source — files inside hidden folders (and hidden files with an importable extension) are indexed, embedded, and summarized exactly like visible files; when OFF (the default for every existing row), behavior is byte-identical to today. `EXCLUDED_DIRS` stays excluded in both states, the extension filter always applies, and the flag — like the ignore list — takes effect on the next sync, with previously indexed hidden files pruned when it is switched off. All entry points (Sync button, CLI) honor the flag; the API stays admin-only.
## Dependencies
- `104_chip_sizing_question_cap` (todo) — pipeline predecessor (execution order) only; no code dependency (this phase touches the importer, the git-sources API, the sync/CLI pipelines, and the Sources view — none of which phase 104's pins reach; its suites must stay green unchanged).
- `89_source_ignore_paths` (complete) — the per-root map, the actions-cell control idiom, the count-tag idiom, and the `PATCH` route this phase extends.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Flag semantics (locked, A1).** `include_hidden=True` lifts ONLY the dot-prefixed-component skip in `iter_importable_files`: the existing check `any(part.startswith(".") or part in excluded for part in rel.parts)` becomes dot-aware only when the flag is False — e.g. `any((not include_hidden and part.startswith(".")) or part in excluded for part in rel.parts)`. Consequences, all deliberate:
- Files INSIDE hidden dirs become importable (`.esphome/esp.md` indexed when ON).
- Hidden files with an importable extension also become importable (`.notes.md` — the dot check covers components, not "the folder of the file", so one rule covers both; the extension filter is the real content gate, and a secret-flavoured file like `.env` has no A9 extension and is never indexed).
- `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`) are skipped in BOTH states — caches/VCS internals are never content.
- The `ignore` tuple (phase 89) composes additively with the flag: an ignored prefix still skips a file when `include_hidden=True`.
- **Storage (task 01).** `git_sources.include_hidden` — BOOLEAN NOT NULL, server default `false`, `Mapped[bool]` (the `documents.is_summary` Boolean precedent, `app/models.py` L136). Alembic `0019_git_source_include_hidden.py` (revises `0018`): `op.add_column("git_sources", sa.Column("include_hidden", sa.Boolean(), server_default=sa.text("false"), nullable=False))`; downgrade drops the column. Existing rows read `False` (A4).
- **Importer signature (task 02).**
- `iter_importable_files(root, extensions, excluded=EXCLUDED_DIRS, ignore=(), include_hidden: bool = False)` — default `False` keeps every existing caller byte-identical; the docstring's skip sentence gains the flag clause.
- `import_sources(sources, llm, *, prune=False, limit=None, session=None, progress=None, ignore_by_root=None, include_hidden_by_root: dict[str, bool] | None = None)` — the map is keyed by **`str(root)`** with the SAME keying convention as `ignore_by_root`; an internal `_include_hidden_for_root(root, include_hidden_by_root) -> bool` (default `False`) is the single read point, used by BOTH the phase-64 progress pre-walk and the processing loop, so `files_total` never disagrees with the walk. `seen` is untouched in shape → `_prune` prunes hidden documents automatically when the flag flips OFF (A2 — the A9/phase-89 precedent). Module docstring "Scope" paragraph updated.
- **API contract (task 03).**
- Schemas (`app/schemas.py`): `GitSourceIn.include_hidden: bool | None = Field(default=None)` (create-time, optional — absent → stored `False`); `GitSourceOut.include_hidden: bool`; `GitSourceRow.include_hidden: bool` (env-fallback rows report `False` — no DB row to store a flag on).
- The PATCH body model is RENAMED `GitSourceIgnoreIn` → `GitSourcePatchIn` (grep-verified: referenced only in `app/schemas.py` and `app/api/git_sources.py` — import L119 + `patch_git_source` L376) and gains:
- `ignore_paths: list[str] | None = Field(default=None)` — **absent/None = the row's list is unchanged; PRESENT = replace semantics exactly as phase 89 A5** (normalization + the A4 fixed-detail 422s run only when present). Every existing client always sends the list, so their behavior is byte-identical; the toggle's PATCH sends only the bool.
- `include_hidden: bool | None = Field(default=None)` — absent/None = unchanged; present = set.
- Both absent → 200 no-op (row untouched).
- `GET /api/git-sources` — DB rows report the stored flag; env rows `False`. `POST /api/git-sources` — both kinds accept `include_hidden`; stored `bool(payload.include_hidden)`.
- `PATCH /api/git-sources/{source_id}` (existing route, still behind `require_admin`) — applies each PRESENT field independently (404 unknown id unchanged); 200 → `GitSourceOut` (id, url, added_at, ignore_paths, include_hidden).
- **Callers (task 04).**
- `app/api/sync.py::_run_sync` — in the existing per-row loop that builds `ignore_by_root` (L233-252), build `include_hidden_by_root: dict[str, bool]` with the SAME `str(root)` keying: `include_hidden_by_root[str(root)] = include_hidden_by_root.get(str(root), False) or bool(row.include_hidden)` (collision → OR — the mirror of the ignore-map union: if either row says "index hidden", the shared root does). Pass `include_hidden_by_root=…` to `import_sources` (L263). Module docstring (L36-40) updated.
- `scripts/import_docs.py` — `_resolve_sources` returns the 3-tuple `(sources, ignore_by_root, include_hidden_by_root)` (manual `--source` → `(sources, {}, {})` — manual dirs have no row; env-fallback rows have no flags); the OR-collision rule is the same; `main` unpacks (L269) and passes the map (L332-334); docstrings updated (module + `_resolve_sources` L181).
- `scripts/load_test_kb.py` — untouched (defaults).
- **UI (task 05).** Sources page = the `git-sources` view. Per **stored** row (`s.id` truthy) in `makeRow`: a **native labeled checkbox** in the actions cell, DOM order **Hidden · Ignore paths · Remove** (the toggle sits next to — left of — the "Ignore paths" button, per the owner's wording; Remove stays last):
- `<label class="git-source-hidden">` wrapping `<input type="checkbox" class="git-source-hidden-box">` + visible text "Hidden"; the checkbox's `aria-label` is `Index hidden folders for ${kindLabel} source: ${value}` (setAttribute — never innerHTML; `value` is the git URL or local path, credential-safety discipline), `checked = s.include_hidden === true`; a `title` on the label explains in plain words ("When checked, files inside hidden (dot) folders are indexed on the next sync. Caches (.git, node_modules, .venv, …) stay excluded.").
- When `s.include_hidden === true`, the source cell also shows a **"hidden on" text tag** (`.git-source-hidden-count` — the `.git-source-ignore-count` idiom: text + background, never color alone, WCAG 1.4.1), next to the `N ignored` tag.
- **§7.4 never-stale lifecycle** — `toggleHidden(s, box)`: on `change`, the box disables immediately (no double-flip); `PATCH /api/git-sources/${s.id}` with `{"include_hidden": box.checked}`; on 200 → clear the error line, `await loadSources()` (the row re-renders from the server), THEN `announce("Hidden folders enabled|disabled for <value>.")` (the phase-89 last-announce order — the confirmation lands after the reload's "N sources listed."); on non-2xx or network failure → the server detail (or the canned "Could not reach the server — the setting was not changed.") into the new page-level `role="alert"` line, and the box **reverts to the server state** (`box.checked = s.include_hidden === true`) and re-enables — the UI never claims a state the server didn't save.
- `frontend/index.html` — one new element after the `#git-sources-table-wrap` region: `<p class="git-source-error" id="git-sources-hidden-error" role="alert" hidden></p>` (reuses the existing `.git-source-error` styling). `frontend/assets/styles.css` — near the phase-89 block (~L2657-2695): `.git-source-hidden` (inline-flex, ~44px hit height matching the action buttons, visible label), `.git-source-hidden input[type="checkbox"]` (sized, `accent-color` on the brand pair — verify + record the AA ratio in the comment, house style), `:disabled` (opacity + `cursor: wait` — the `.git-source-remove:disabled` idiom), focus ring via the GLOBAL `:focus-visible` rule (L146 — no per-control rule needed), and `.git-source-hidden-count` (copy of the `.git-source-ignore-count` rule, provenance comment citing phase 105).
- Env-fallback rows (`id` null) get **no** checkbox — the existing "from .env" tag stays (A3).
- **NOT touched:** the RAG view (`sources.js`), `app/rag/retriever.py`, the chunker, the KB tree/catalog + agent tools (DB-driven — they pick up newly indexed hidden docs for free), `app/rag/git_sources.py` (clone/pull only, no walk), the upload run (phase 90 — no scan), `AGENTS.md`, `.agents/PLAN.md`, any completed phase.
## Tasks
1. `01_include_hidden_column.md` — `git_sources.include_hidden` BOOLEAN column (model + alembic `0019`) + default/round-trip tests.
2. `02_importer_include_hidden.md` — `iter_importable_files`/`import_sources` flag support (walk + progress pre-walk + prune interaction + ignore composition) + unit & integration tests.
3. `03_include_hidden_api.md` — schemas (`In`/`Out`/`Row` + the `GitSourcePatchIn` rename with optional fields) + GET/POST/PATCH wiring + integration tests.
4. `04_include_hidden_pipelines.md` — wire the per-row flag into `_run_sync` and `scripts/import_docs.py` + integration tests.
5. `05_hidden_toggle_sources_ui.md` — the per-row "Hidden" checkbox on the Sources page (tag + §7.4 lifecycle + error line + a11y) + source-level unit pins.
6. `06_e2e_hidden_folders_toggle.md` — dedicated Playwright suite `tests/e2e/test_hidden_folders_toggle.py` (run in isolation), regressions, full gate, atomic commit.
## Testing & Quality
- Unit — `tests/unit/test_importer_include_hidden.py` (new, task 02): `iter_importable_files` on a tmp fixture tree — default OFF pins today's behavior byte-identically (hidden dir + hidden file skipped), ON admits both, `EXCLUDED_DIRS` skipped in BOTH states, `ignore` tuple still bites when ON, extension filter unchanged (`.env` never indexed); the `str(root)` keying + default-False for unlisted roots; `tests/unit/test_hidden_folders_toggle.py` (new, task 05): source-level pins for the JS/HTML/CSS wiring (house pattern — read the assets as text), incl. the single-source cross-file check that the aria-label template names the source value; task 01's pins extend the existing model/migration test surfaces (a fresh row reads `include_hidden is False`; an explicit `True` round-trips).
- Integration — `tests/integration/test_importer_include_hidden.py` (new, task 02): `import_sources` against a fixture dir — hidden file produces NO `Document`/`Chunk` rows by default; WITH the map it is embedded + summarized normally; previously indexed hidden file + flag OFF → pruned on the next run; progress `total` agrees with the walk in both states; sources not in the map behave exactly as before. `tests/integration/test_git_sources_api.py` (extended, task 03): GET reports `False` default / stored `True`; POST create round-trip; PATCH bool-only, list-only, both, neither (no-op 200), the phase-89 422s unchanged for present lists, 404, anonymous 403. `tests/integration/test_sync_api.py` (extended, task 04): a local row with a hidden dir — flag False syncs zero hidden docs, True syncs them; `tests/integration/test_import_docs_git.py` (extended, task 04): the CLI DB-row path with the flag set.
- E2E (mandatory, A16) — `tests/e2e/test_hidden_folders_toggle.py` (task 06), run in isolation with the DB up: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov`.
- Coverage: **>90%** on `app/` (the validate.sh gate — the importer/API/pipeline additions are fully unit+integration covered).
## Completion Criteria
- [ ] A stored source (git, local, or uploaded) has a "Hidden" checkbox next to its "Ignore paths" button: flipping it on persists (`PATCH` 200, the source cell shows the "hidden on" tag, `GET /api/git-sources` round-trips `include_hidden: true`); the failure path reverts the box and announces the error in a `role="alert"` line.
- [ ] A sync (button or CLI) with the flag OFF indexes nothing with a dot-prefixed component (no `documents`/`chunks` rows — the byte-identical default); with the flag ON, `.hidden/note.md` is indexed, embedded, and summarized like any visible file and shows up in the KB catalog; `EXCLUDED_DIRS` content is excluded in both states.
- [ ] A2: a previously indexed hidden file is PRUNED from the KB on the next sync after the flag flips OFF (`detail.pruned` increments; the catalog no longer lists it).
- [ ] API contracts hold: PATCH bool-only / list-only / both / neither; the phase-89 fixed-detail 422s unchanged for present lists; 404 unknown id; anonymous 403 on the route.
- [ ] Env-fallback rows render no checkbox (the "from .env" tag stays); the Sources page stays WCAG-clean (visible label, focus-visible, text never color alone).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` green in isolation (DB up); regression suites `test_source_ignore_paths.py`, `test_git_sources_admin.py`, `test_local_directory_sources.py`, `test_sync_button.py`, `test_smoke.py` green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **A1 — what "on" means (owner-confirmed 2026-09-14).** The flag lifts the dot-prefixed-component skip for that source — files inside hidden folders AND hidden files with an importable extension become indexable; `EXCLUDED_DIRS` (`.venv`, `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`) stay excluded regardless; the extension filter always applies.
- **A2 — toggling off prunes (owner-confirmed).** Previously indexed hidden files leave the KB on the next sync (the `seen`-set prune — the phase-89 A2 / A9 precedent), exactly like newly-ignored files.
- **A3 — per stored row only (owner-confirmed).** Every stored `git_sources` row (git, local, uploaded) gets the toggle; env-fallback rows have no DB row and get no control (the phase-89 A3 precedent).
- **A4 — default off (owner-confirmed).** `include_hidden` defaults to `false` for all existing rows — byte-identical behavior until the owner flips it; the flag takes effect on the NEXT sync (no auto-sync, same as the ignore list).
- **A5 — control idiom (owner-confirmed).** A native labeled checkbox ("Hidden") in the actions cell, left of the "Ignore paths" button (Remove stays last), with a "hidden on" text tag in the source cell when enabled — not a styled switch.
## Commit
```bash
git add app/ alembic/versions/0019_git_source_include_hidden.py scripts/ frontend/ tests/ TODO.md .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): per-source hidden-folders toggle — dot-prefixed paths are indexable per input"
```
@@ -0,0 +1,45 @@
# Task 01 — `git_sources.include_hidden` BOOLEAN column (model + alembic `0019`)
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "…There should be a toggle per input (next to the ignores button) to allow indexing hidden .folders."
## Objective
Persist the per-source hidden-folders flag: one additive, reversible BOOLEAN column on `git_sources`, server-defaulted to `false` so every pre-phase-105 row imports byte-identically (A4).
## Work
1. `app/models.py` — the `GitSource` class (L220-259): add the column directly AFTER `ignore_paths` (L248-251), mirroring its docstring/provenance style (`Boolean` is already imported, L78):
```python
#: Index hidden (dot-prefixed) paths from this source (phase 105,
#: A1): True → the walk (app.rag.importer.iter_importable_files)
#: does not skip dot-prefixed components — files inside hidden
#: folders AND hidden files with an importable extension are
#: indexed; ``EXCLUDED_DIRS`` (``.venv``, ``node_modules``,
#: ``.git``, …) are excluded in BOTH states, and the extension
#: filter always applies. Takes effect on the next sync (no
#: auto-sync — the ignore-list precedent, phase 89). Server
#: default False: every pre-phase-105 row imports exactly as
#: before (A4).
include_hidden: Mapped[bool] = mapped_column(
Boolean, default=False, server_default=text("false"), nullable=False
)
```
(Extend the class docstring's one-line field inventory if it names `ignore_paths` — the module header L19 does: add `include_hidden` (phase 105) to the parenthetical.)
2. `alembic/versions/0019_git_source_include_hidden.py` (NEW — the house format of `0013_git_source_ignore_paths.py`, one additive reversible column):
- `revision = "0019"`, `down_revision = "0018"`.
- `upgrade()`: `op.add_column("git_sources", sa.Column("include_hidden", sa.Boolean(), server_default=sa.text("false"), nullable=False))`.
- `downgrade()`: `op.drop_column("git_sources", "include_hidden")`.
- Module docstring: the phase-89-0013 provenance style (what the flag is, A1/A4, one additive reversible column).
3. Tests — extend the existing model/migration test surfaces (find the current `GitSource` default-pinning tests — the phase-89 column tests live in the `git_sources` unit/integration suites; add alongside them):
- A freshly inserted `GitSource` row (no `include_hidden` passed) reads `include_hidden is False` (the Python `default=False` AND the server default agree).
- An explicit `include_hidden=True` round-trips through the DB (`session.add` → `commit` → fresh session → `True`).
- The migration applies on the test DB from head (`alembic upgrade head` is part of the standard test-db fixture setup — if the suite asserts the column set of `git_sources`, add `include_hidden` to the expected set).
4. Run `uv run pytest tests/unit/ -q && uv run alembic upgrade head` (against the dev/test DB per the house quick reference) — green.
## Testing & Quality
- Unit/integration: the default + round-trip pins above ARE this task's layer (no importer behavior yet — that is task 02).
- Coverage: **>90%** on `app/` (model-only change; the validate.sh gate).
## Completion Criteria
- [ ] `GitSource.include_hidden` exists with `server_default=text("false")`, `nullable=False`, and the A1/A4 provenance comment
- [ ] `alembic/versions/0019_git_source_include_hidden.py` upgrades from `0018` and downgrades cleanly; the dev/test DB is at head
- [ ] Fresh-row-default-False and explicit-True round-trip tests pass; existing `git_sources` suites stay green
- [ ] `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,70 @@
# Task 02 — Importer flag support: `iter_importable_files` / `import_sources` honor `include_hidden` per root
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "hidden dot folders aren't being indexed. …to allow indexing hidden .folders."
## Objective
Make the walk choke point flag-aware: `iter_importable_files` gains `include_hidden` (default `False` — every existing caller byte-identical) and `import_sources` gains `include_hidden_by_root` (same `str(root)` keying as phase 89's `ignore_by_root`), used by BOTH the phase-64 progress pre-walk and the processing loop. Pruning falls out for free through the untouched `seen` set (A2).
## Work
1. `app/rag/importer.py` — `iter_importable_files` (L161-194):
- Signature: `def iter_importable_files(root: Path, extensions: frozenset[str], excluded: frozenset[str] = EXCLUDED_DIRS, ignore: tuple[str, ...] = (), include_hidden: bool = False) -> list[Path]:`
- The skip check (L184) becomes flag-aware — ONE expression, byte-identical when the flag is False:
```python
if any(
(not include_hidden and part.startswith(".")) or part in excluded
for part in rel.parts
):
continue
```
- Docstring: the "Skips:" sentence gains — "…when ``include_hidden`` is False (the default): any path with a dot-prefixed component; when True, dot-prefixed components are ADMITTED (files inside hidden folders, and hidden files) and only *excluded* is consulted (A1 — caches/VCS internals are never content). The *ignore* tuple composes additively in both states."
2. `app/rag/importer.py` — `import_sources` (L196+):
- Keyword-only param after `ignore_by_root`: `include_hidden_by_root: dict[str, bool] | None = None`.
- New private helper next to `_ignore_for_root` (L145-160), same style:
```python
def _include_hidden_for_root(
root: Path, include_hidden_by_root: dict[str, bool] | None
) -> bool:
"""The per-root hidden-folders flag (phase 105, A1).
Keyed by ``str(root)`` — the root string exactly as the caller
passed it in ``sources`` (the ``_ignore_for_root`` convention,
phase 89): ``True`` only for roots the caller lists as True;
unlisted/``None`` roots are ``False`` — every existing caller
behaves byte-identically (A4).
"""
return bool((include_hidden_by_root or {}).get(str(root), False))
```
- The progress pre-walk (the `if progress is not None:` block, ~L244-252) passes `include_hidden=_include_hidden_for_root(root, include_hidden_by_root)` to its `iter_importable_files` call — `files_total` must agree with the walk in both states.
- The processing loop: alongside `ignore = _ignore_for_root(root, ignore_by_root)` (~L272), add `include_hidden = _include_hidden_for_root(root, include_hidden_by_root)` and pass both into the `iter_importable_files` call (~L273-275).
- `import_sources` docstring: after the `ignore_by_root` paragraph, the mirror paragraph: "``include_hidden_by_root`` (phase 105, A1) maps ``str(root)`` to the stored flag: ``True`` admits dot-prefixed components for that root (``EXCLUDED_DIRS`` and the extension filter still apply; the ignore tuple composes additively). Unlisted/``None`` roots are ``False`` — byte-identical to pre-phase-105. A file that was indexed with the flag ON and is walked again with it OFF simply never enters ``seen``, so the next ``prune=True`` run deletes its row automatically (A2 — the A9/phase-89 precedent)."
- Module docstring "Scope" paragraph (L20-22): append the flag clause ("…skipped, plus the well-known exclusion list — UNLESS the source's phase-105 hidden-folders flag admits dot-prefixed paths; the exclusion list always applies").
3. `tests/unit/test_importer_include_hidden.py` (NEW) — `iter_importable_files` on a `tmp_path` tree (extensions `frozenset({".md"})`), fixture layout:
```
visible.md .hidden/note.md .notes.md
.venv/junk.md node_modules/x.md .hidden/.deep.md
keep/ok.md
```
- **Default (flag False) — today's behavior pinned byte-identically:** result == `[keep/ok.md, visible.md]` (sorted); hidden dir, hidden file, `.venv`, `node_modules` all absent.
- **Flag True (A1):** result contains `.hidden/note.md`, `.notes.md`, `.hidden/.deep.md`, `visible.md`, `keep/ok.md`; STILL excludes `.venv/junk.md` and `node_modules/x.md` (EXCLUDED_DIRS in both states).
- **Composition with `ignore`:** flag True + `ignore=(".hidden",)` → `.hidden/*` gone, `.notes.md` present (additive).
- **Extension filter unchanged:** flag True with `.env`-like file `.env` (no A9 extension) → never listed.
- `_include_hidden_for_root` unit pins: `None` map → False; unlisted root → False; listed `True` → True; listed `False` → False; keying is `str(root)` (two `Path` objects, equal strings, same answer).
4. `tests/integration/test_importer_include_hidden.py` (NEW) — `import_sources` against a fixture dir (the `tests/integration/test_importer_ignore.py` harness — fake `Embedder` from `tests/fakes.py` or the module's own fake, real test DB):
- Layout: `visible.md`, `.hidden/note.md` (non-markdown `.hidden/data.yaml` too, to prove the summary path runs for admitted hidden files), `.venv/junk.md`.
- **Run 1 (default, no map):** `Document` rows exist for `visible.md` ONLY — no row for `.hidden/note.md` (A4 byte-identical); `summary` stats unchanged.
- **Run 2 (`include_hidden_by_root={str(root): True}`):** `.hidden/note.md` + `.hidden/data.yaml` get `Document` + `Chunk` rows (embedded via the fake), the yaml gets a `summary` row (the phase-30 path), `.venv/junk.md` still absent (A1).
- **A2 prune:** with the rows from run 2 committed, run 3 with `prune=True` and NO map (flag off) → `.hidden/*` rows deleted (`summary.pruned >= 2`), `visible.md` untouched.
- **Progress agreement:** with `progress=…` set, `total` equals the visible-only count in run 1's rules and the extended count in run 2's rules (the pre-walk uses the same flag).
- **Unlisted roots untouched:** two roots, map lists only one as True — the other root's hidden file stays out (per-root, not global).
5. Run `uv run pytest tests/unit/test_importer_include_hidden.py tests/integration/test_importer_include_hidden.py tests/unit/test_importer_ignore.py tests/integration/test_importer_ignore.py -q` — green (the phase-89 suites are the byte-identical regression proof).
## Testing & Quality
- Unit + integration as above are this task's layer; API/pipeline/UI come later.
- Coverage: **>90%** on `app/` (the new helper + both call sites are fully covered — the validate.sh gate).
## Completion Criteria
- [ ] `iter_importable_files(..., include_hidden=False)` is the default and its default-state result is byte-identical to pre-task (phase-89 importer suites green)
- [ ] `include_hidden=True` admits dot-prefixed components, keeps `EXCLUDED_DIRS` + extension filter + `ignore` tuple in force (A1)
- [ ] `import_sources(include_hidden_by_root=…)` drives BOTH the pre-walk and the loop through `_include_hidden_for_root`; unlisted/`None` → False
- [ ] Flag-off re-run prunes previously indexed hidden docs (A2); progress `total` agrees with the walk in both states
- [ ] `uv run pytest tests/unit/ -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,95 @@
# Task 03 — API: `include_hidden` on GET/POST/PATCH (the `GitSourcePatchIn` rename)
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "There should be a toggle per input (next to the ignores button)…"
## Objective
Expose the flag through the admin API: `GET`/`POST` carry it, and the existing `PATCH /api/git-sources/{id}` becomes the single per-row edit endpoint for BOTH the ignore list and the toggle — each field optional, present-field-wins, so the toggle's PATCH sends only `{"include_hidden": …}` and the phase-89 dialog's PATCH (which always sends the list) keeps byte-identical semantics.
## Work
1. `app/schemas.py`:
- `GitSourceIn` (L450-477): add after `ignore_paths`:
```python
include_hidden: bool | None = Field(default=None)
```
+ docstring clause: "``include_hidden`` (phase 105) is optional at create time (absent → stored ``False`` — A4)."
- `GitSourceOut` (L490-506): add `include_hidden: bool` (after `ignore_paths`) + docstring clause (the stored flag; `False` for a row created without it).
- `GitSourceRow` (L508-528): add `include_hidden: bool` + docstring clause (**env-fallback rows report `False`** — no DB row to store a flag on, the `ignore_paths: []` precedent).
- **RENAME** `GitSourceIgnoreIn` (L530-543) → `GitSourcePatchIn` and rework its body (grep-verified references: only `app/schemas.py` + `app/api/git_sources.py` L119/L376 — no tests import it):
```python
class GitSourcePatchIn(BaseModel):
"""``PATCH /api/git-sources/{source_id}`` body (phase 89 A5;
extended phase 105).
Each field is independent and OPTIONAL: absent/None leaves the
row's value unchanged; PRESENT applies. ``ignore_paths`` when
present keeps the phase-89 A5 REPLACE semantics (the body list,
normalized + A4-validated, becomes the row's whole list — empty
list clears all; every pre-phase-105 client always sends the
list, so their behavior is byte-identical). ``include_hidden``
(phase 105) when present sets the stored flag. Both absent →
200 no-op (the row is untouched).
"""
ignore_paths: list[str] | None = Field(default=None)
include_hidden: bool | None = Field(default=None)
```
2. `app/api/git_sources.py`:
- Import L119: `GitSourceIgnoreIn` → `GitSourcePatchIn`.
- `list_git_sources` (L202-245): DB row construction (~L226-233) gains `include_hidden=row.include_hidden`; the env-fallback row (~L240) gains `include_hidden=False`; the endpoint docstring's field list mentions it.
- `create_git_source` (L249-280): response construction (L278) gains `include_hidden=row.include_hidden`.
- `_create_git_row` (L315, row construction ~L326-335) and `_create_local_row` (L340, ~L358-368): both gain `include_hidden=bool(payload.include_hidden)` (absent → `False`, A4).
- `patch_git_source` (L374-393) — new body:
```python
def patch_git_source(
source_id: uuid.UUID,
payload: GitSourcePatchIn,
db: Session = Depends(get_db), # noqa: B008
) -> GitSourceOut:
"""Edit one source's ignore list and/or hidden-folders flag.
Phase 89 A5 (ignore list) + phase 105 (the flag): 404 unknown
id; each PRESENT body field applies independently —
``ignore_paths`` REPLACES the list (normalized + A4-validated,
fixed 422 details); ``include_hidden`` sets the flag. Both
absent → 200 no-op. Returns the updated row's public shape
(id, url, added_at, ignore_paths, include_hidden).
"""
row = db.get(GitSource, source_id)
if row is None:
raise HTTPException(status_code=404, detail="git source not found")
if payload.ignore_paths is not None:
row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
if payload.include_hidden is not None:
row.include_hidden = payload.include_hidden
db.commit()
db.refresh(row)
return GitSourceOut(
id=row.id,
url=row.url,
added_at=row.added_at,
ignore_paths=row.ignore_paths,
include_hidden=row.include_hidden,
)
```
- Router module docstring (L1-30ish — the contract list): the PATCH line extends to "the ignore list (replace) and/or the hidden-folders flag (phase 105) — each optional, present-wins".
3. `tests/integration/test_git_sources_api.py` — extend the phase-89 PATCH section (the suite already pins 404 + anonymous 403 + the A4 fixed-detail 422s):
- `GET` — a fresh stored row reports `include_hidden: false`; an env-fallback row (table empty + `BOR_GIT_SOURCES` monkeypatched, the suite's existing pattern) reports `include_hidden: false`.
- `POST` — `kind="local"` create with `include_hidden: true` → 201 body `include_hidden: true`; without the field → `false` (A4).
- `PATCH` matrix (one stored row, list `["a/b"]`, flag `false`):
- `{"include_hidden": true}` → 200, flag `true`, list UNCHANGED `["a/b"]` (the toggle's exact payload).
- `{"ignore_paths": ["c/d"]}` → 200, list REPLACED, flag UNCHANGED `false` (the dialog's exact payload — byte-identical to phase 89).
- `{"ignore_paths": [], "include_hidden": true}` → both applied.
- `{}` → 200 no-op (list + flag unchanged).
- the phase-89 422s still fire for a PRESENT bad list (>200 entries / empty-after-normalization / >500-char entry — the fixed details), and a bad list does NOT half-apply the flag (assert the flag is untouched after a 422 with both fields present).
- 404 unknown id; anonymous `PATCH` 403 (the existing pin already covers the route — extend it to include a bool-only body so the pin proves the toggle path is gated too).
4. Run `uv run pytest tests/integration/test_git_sources_api.py -q` — green.
## Testing & Quality
- Integration as above is this task's layer (the unit schema pins ride on the integration 422/shape checks, house pattern for this router).
- Coverage: **>90%** on `app/` (every new/changed branch — both field-present/absent combos — is exercised — the validate.sh gate).
## Completion Criteria
- [ ] `GET` reports the flag for DB rows and `false` for env rows; `POST` create round-trips it (absent → `false`)
- [ ] `GitSourcePatchIn` exists with two optional fields; `GitSourceIgnoreIn` is gone repo-wide (grep-verified)
- [ ] PATCH: bool-only, list-only, both, neither (no-op 200); the phase-89 A4 fixed-detail 422s unchanged for present lists; a 422 never half-applies the other field; 404 + anonymous 403 hold
- [ ] `uv run pytest tests/integration/test_git_sources_api.py -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,48 @@
# Task 04 — Pipelines: the Sync button + the CLI honor the per-row flag
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "hidden dot folders aren't being indexed. …to allow indexing hidden .folders."
## Objective
Build the `include_hidden_by_root` map at the two live import entry points — the in-app Sync (`app/api/sync.py::_run_sync`) and the CLI (`scripts/import_docs.py::_resolve_sources`) — with the SAME `str(root)` keying and collision rule as the phase-89 `ignore_by_root`, and pass it to `import_sources`. The upload run needs nothing (phase 90 — no scan) and `scripts/load_test_kb.py` keeps its defaults.
## Work
1. `app/api/sync.py` — `_run_sync` (the per-row loop L233-252 + the `import_sources` call L262-264):
- After `ignore_by_root: dict[str, list[str]] = {}` (L233): `include_hidden_by_root: dict[str, bool] = {}`
- Inside the loop, next to the phase-89 `ignore_by_root.setdefault(...)` (L251-252) — the comment cites phase 105 + the same sibling/repo-name collision note:
```python
# Phase 105 (A1/A4): the row's hidden-folders flag, keyed by
# the SAME root string the importer sees; a shared-root
# collision ORs — if EITHER row says "index hidden", the
# root does (the ignore-map union's boolean mirror).
include_hidden_by_root[str(root)] = (
include_hidden_by_root.get(str(root), False)
or bool(row.include_hidden)
)
```
- The call (L262-264): add the kwarg — `summary: ImportSummary = await import_sources(sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root, include_hidden_by_root=include_hidden_by_root)`
- Module docstring (L30-45, the pipeline list that names "honoring each row's ``ignore_paths`` (phase 89…)"): extend the clause with "and its ``include_hidden`` flag (phase 105 — the per-root hidden-folders map, same per-row construction)".
2. `scripts/import_docs.py`:
- `_resolve_sources` docstring (L181-185): "Returns ``(sources, ignore_by_root)``" → "Returns ``(sources, ignore_by_root, include_hidden_by_root)`` (phase 89; phase 105 adds the per-root flag map — the flag is stored per row, manual ``--source`` dirs and the legacy fallback have no rows and import with the empty map: hidden paths skipped, A4)."
- All THREE return paths become 3-tuples:
- the `cli_sources` early return (~L190): `return [path.expanduser() for path in cli_sources], {}, {}`
- the rows branch: after the `ignore_by_root` dict init (L206), add `include_hidden_by_root: dict[str, bool] = {}`; inside the loop next to the phase-89 extend (L224-225), the same OR assignment as sync.py (comment: phase 105); the return (L226): `return sources, ignore_by_root, include_hidden_by_root`
- the legacy fallback return (~L227): `return [path.expanduser() for path in DEFAULT_SOURCES], {}, {}`
- `main` (L269): `sources, ignore_by_root, include_hidden_by_root = _resolve_sources(args.source, settings)` (the comment above it, L258, extends to name the flag map).
- The `import_sources` call (L332-335): add `include_hidden_by_root=include_hidden_by_root,`.
- Module docstring: the phase-89 line about the per-root ignore map gains the flag clause.
3. Tests:
- `tests/integration/test_sync_api.py` — extend with the flag (the suite's local-row + fixture-dir pattern; the fixture dir gains `.hidden/note.md` + a visible file):
- Row with `include_hidden=False` (default) + `POST /api/sync` → terminal status `success`, the hidden file has NO `documents` row (A4).
- Same row flipped to `True` (task-03 PATCH or direct model set, the suite's idiom) + sync → the hidden file IS indexed (`documents` row present; `detail.files` counts it).
- `tests/integration/test_import_docs_git.py` (the CLI DB-row suite) — one test: a local row with a hidden file + `include_hidden=True` → running the CLI main path (the suite's existing invocation harness) indexes the hidden file; `False` → not (the CLI's map is built, not lost — the regression this phase most plausibly breaks is here).
4. Run `uv run pytest tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -q` — green.
## Testing & Quality
- Integration as above is this task's layer; the importer itself is task 02's, the UI task 05's.
- Coverage: **>90%** on `app/` (the sync map-building branch is exercised; `scripts/` is outside the `--cov=app` gate — the CLI test is the behavioral pin, house pattern).
## Completion Criteria
- [ ] `_run_sync` builds `include_hidden_by_root` in the existing per-row loop (OR on collision) and passes it to `import_sources`
- [ ] `_resolve_sources` returns the 3-tuple on ALL three return paths (manual, rows, legacy fallback) and `main` passes the map through
- [ ] Sync with the flag off indexes no hidden docs; with it on, does — E2E-provable through `detail` + `documents` rows; the CLI behaves the same
- [ ] `uv run pytest tests/integration/ -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,177 @@
# Task 05 — The per-row "Hidden" toggle on the Sources page (checkbox + tag + §7.4 lifecycle + a11y)
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — "There should be a toggle per input (next to the ignores button) to allow indexing hidden .folders."
## Objective
The owner-visible half of the feature: every stored row on the Sources page gets a labeled **Hidden** checkbox in the actions cell, left of its "Ignore paths" button (DOM order Hidden · Ignore paths · Remove — A5); when on, the source cell shows a "hidden on" text tag; the flip PATCHes the flag with the §7.4 never-stale lifecycle and reverts to the server state on failure.
## Work
1. `frontend/index.html` — inside `#view-git-sources`, directly AFTER the `#git-sources-table-wrap` region closes (~L632), the page-level error line for the toggle (the ignore dialog carries its own error INSIDE the modal; the checkbox lives in the table, so its error lives at page level — reuses the existing `.git-source-error` styling):
```html
<!-- Phase 105: the per-row "Hidden" toggle's error line — the
checkbox is a table-cell control (no dialog of its own), so
its failure announces here (role=alert; git-sources.js
showHiddenError). Hidden until a PATCH fails. -->
<p class="git-source-error" id="git-sources-hidden-error" role="alert" hidden></p>
```
2. `frontend/assets/git-sources.js` — `makeRow` (L357+):
- **State tag** — in the `urlTd` block, right after the phase-89 `N ignored` count-tag block (~L374-382), the same idiom:
```js
/* Phase 105 (A5): the "hidden on" state tag — the
.git-source-ignore-count idiom (TEXT + background, never
color alone — WCAG 1.4.1), so the flag is readable at a
glance without hovering the checkbox. */
if (s.id && s.include_hidden === true) {
const hiddenTag = document.createElement("span");
hiddenTag.className = "git-source-hidden-count";
hiddenTag.textContent = "hidden on";
urlTd.append(hiddenTag);
}
```
- **The checkbox** — in the `if (s.id) {` actions-cell branch (~L390), BEFORE the `ignoreBtn` construction (~L399), so DOM order is Hidden · Ignore paths · Remove:
```js
/* Phase 105 (A5): the per-row hidden-folders toggle — a native
labeled checkbox (the WCAG focus/label idiom) LEFT of the
"Ignore paths" button; Remove stays last. Stored rows only
(A3 — env-fallback rows fall through to the "from .env"
tag). The aria-label is the ONLY place `value` appears
(setAttribute — never innerHTML). Checked state comes from
the SERVER row (s.include_hidden), never from a prior local
flip (§7.4 — makeRow only ever renders server state). */
const hiddenLabel = document.createElement("label");
hiddenLabel.className = "git-source-hidden";
hiddenLabel.title =
"When checked, files inside hidden (dot) folders are indexed on the next sync. Caches (.git, node_modules, .venv, …) stay excluded.";
const hiddenBox = document.createElement("input");
hiddenBox.type = "checkbox";
hiddenBox.className = "git-source-hidden-box";
hiddenBox.checked = s.include_hidden === true;
hiddenBox.setAttribute(
"aria-label",
`Index hidden folders for ${kindLabel} source: ${value}`,
);
hiddenLabel.append(hiddenBox, document.createTextNode("Hidden"));
hiddenBox.addEventListener("change", () => toggleHidden(s, hiddenBox));
actTd.appendChild(hiddenLabel);
```
- **`toggleHidden(s, box)`** — new function next to `saveIgnorePaths` (the phase-89 §7.4 section, ~L593+), same announce/reload discipline:
```js
/* Phase 105: the hidden-folders toggle — PATCH { include_hidden }
only (the row's list is untouched — the PATCH body's optional
fields, task 03). §7.4 never-stale: the box disables at once
(no double-flip while the PATCH is out); on 200 the row
re-renders from the server (loadSources) and the
CONFIRMATION is the LAST announcement (the reload's "N
sources listed." lands first — the phase-89 order); on
failure the box REVERTS to the server state and the detail
lands in #git-sources-hidden-error (role=alert). */
function toggleHidden(s, box) {
const value = s.kind === "local" ? (s.path ?? s.url) : s.url;
const wanted = box.checked;
box.disabled = true; // a PATCH is out — the box must not flip twice
fetch(`/api/git-sources/${s.id}`, {
method: "PATCH",
headers: { "Content-Type": "application/json" },
credentials: "same-origin",
body: JSON.stringify({ include_hidden: wanted }),
})
.then(async (r) => {
if (r.ok) {
hideHiddenError();
await loadSources();
announce(`Hidden folders ${wanted ? "enabled" : "disabled"} for ${value}.`);
return;
}
const detail = await apiDetail(
r, `Could not update the hidden-folders setting (${r.status}).`,
);
showHiddenError(detail);
box.checked = s.include_hidden === true; // revert to server state
box.disabled = false;
})
.catch(() => {
showHiddenError("Could not reach the server — the setting was not changed.");
box.checked = s.include_hidden === true;
box.disabled = false;
});
}
function showHiddenError(message) {
if (hiddenErrorEl) hiddenErrorEl.textContent = message;
if (hiddenErrorEl) hiddenErrorEl.hidden = false;
}
function hideHiddenError() {
if (hiddenErrorEl) hiddenErrorEl.textContent = "";
if (hiddenErrorEl) hiddenErrorEl.hidden = true;
}
```
(Also call `hideHiddenError()` at the top of `loadSources`'s success path (~L315, after `hideLoadError()`) so a healed list clears the stale line — the phase-89 "happy path heals the error state" precedent.)
- Element grabber next to the other page-local grabs (~L245-255): `const hiddenErrorEl = root.querySelector("#git-sources-hidden-error");`
- Module header comment block (L1-177ish, the contract list): add the phase-105 entry — "the per-row Hidden checkbox (makeRow) → PATCH {include_hidden} (task 03's optional field) → loadSources + announce; failure reverts the box + #git-sources-hidden-error (role=alert); env-fallback rows get no checkbox (A3)".
3. `frontend/assets/styles.css` — after the phase-89 block (the `.git-source-ignore-count` rule ends ~L2695):
```css
/* Phase 105 (A5): the per-row "Hidden" checkbox — the actions
cell, LEFT of the "Ignore paths" button (JS builds it,
git-sources.js makeRow). A native labeled checkbox: the
visible "Hidden" text + the checkbox's own aria-label
(full source value). ~44px hit height matches the action
buttons; :focus-visible via the GLOBAL rule (L146) — no
per-control ring needed. */
.git-source-hidden {
display: inline-flex;
align-items: center;
gap: 0.4rem;
height: 44px;
padding: 0 0.5rem;
font-size: 0.8rem;
color: var(--ink);
cursor: pointer;
user-select: none;
white-space: nowrap;
}
/* --brand checkbox fill: the native check renders in --bg on
--brand (the house 5.2:1 brand pairing); [executor: verify the
checked-state contrast of the native widget in both themes +
record the ratio here — house style]. */
.git-source-hidden input[type="checkbox"] {
width: 1.05rem;
height: 1.05rem;
margin: 0;
accent-color: var(--brand);
cursor: pointer;
}
.git-source-hidden:disabled { opacity: 0.5; cursor: wait; }
/* Phase 105 (A5): the "hidden on" state tag — a copy of the
.git-source-ignore-count idiom (TEXT + background, never
color alone — WCAG 1.4.1: --ink on --bg 16.7:1). */
.git-source-hidden-count {
display: inline-block;
margin-left: 0.55rem;
padding: 0.08rem 0.5rem;
border: 1px solid var(--line);
border-radius: 999px;
font-size: 0.72rem;
font-weight: 600;
color: var(--ink);
background: var(--bg);
white-space: nowrap;
}
```
4. `tests/unit/test_hidden_folders_toggle.py` (NEW) — source-level pins (the house pattern: read the asset files as text; mirror `tests/unit/test_source_ignore_paths.py`'s structure if it exists, else the closest JS-pinning suite):
- `git-sources.js`: `makeRow` contains the checkbox construction — class `git-source-hidden-box`, `type` checkbox, the aria-label template `` `Index hidden folders for ${kindLabel} source: ${value}` ``; `checked = s.include_hidden === true`; the label is appended to `actTd` BEFORE the ignore button (pin the slice order: the hidden-label block precedes the `ignoreBtn` construction); `toggleHidden` defined with `PATCH` + `body: JSON.stringify({ include_hidden: wanted })`; the failure branch reverts (`box.checked = s.include_hidden === true`) AND re-enables; the success branch announces AFTER `await loadSources()` (pin the order in the source slice).
- **Cross-file pin (single source of truth for the field name):** the JS body `include_hidden` key and the Python `GitSourcePatchIn.include_hidden` field (regex-parsed from `app/schemas.py`) are the SAME string — a rename on either side breaks the wire contract and this test.
- `index.html`: `#git-sources-hidden-error` exists, carries `role="alert"`, `hidden`, class `git-source-error`, and sits INSIDE `#view-git-sources` after `#git-sources-table-wrap` (source order).
- `styles.css`: the `.git-source-hidden`, `.git-source-hidden input[type="checkbox"]`, `.git-source-hidden:disabled`, `.git-source-hidden-count` rules exist; the checkbox rule sets `accent-color`.
- The `N ignored`-style tag text pin: `hidden on` literal present in the JS (the tag copy is the state — text, never color alone).
5. Run `uv run pytest tests/unit/test_hidden_folders_toggle.py -q` — green.
## Testing & Quality
- Unit (source-level pins) is this task's layer; the behavioral E2E is task 06.
- Coverage: **>90%** on `app/` (no `app/` changes in this task — the gate is unaffected; keep it green).
## Completion Criteria
- [ ] Every stored row renders Hidden · Ignore paths · Remove in the actions cell; env-fallback rows render no checkbox
- [ ] `checked` comes only from server state; a "hidden on" tag appears in the source cell iff the flag is on
- [ ] Flip → box disables → `PATCH {"include_hidden": …}` → 200: error cleared, list reloaded, confirmation announced LAST; failure: box reverts + re-enables, detail in the `role="alert"` line; the line heals on a successful load
- [ ] WCAG: visible label + full-value `aria-label` on the checkbox, global `:focus-visible` ring, tag is text (never color alone), checkbox checked-state contrast verified + recorded in the CSS comment
- [ ] The JS field-name pin matches `app/schemas.py` (cross-file test); `uv run pytest tests/unit/ -q` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,55 @@
# Task 06 — E2E: `tests/e2e/test_hidden_folders_toggle.py` (isolation) + regressions + full gate + commit
**Phase:** `105_hidden_folders_toggle` · **Source:** `TODO.md` L3 — the whole item, proven end to end.
## Objective
One dedicated Playwright suite proving the TODO item through the REAL page + REAL API + REAL sync pipeline (mock LLM, no git, no network — a `kind="local"` row over a fixture dir, the `test_source_ignore_paths.py` module pattern verbatim): default-off byte-identity, toggle-on indexes hidden folders, toggle-off prunes them (A2), env rows have no control, a11y + the error line. Then the phase's full gate and the single atomic commit.
## Work
1. `tests/e2e/test_hidden_folders_toggle.py` (NEW) — copy the module scaffolding of `tests/e2e/test_source_ignore_paths.py` (the module-env `BOR_GIT_SOURCES` URL that is NEVER synced/cloned; `e2e/auth_helpers.login`; `e2e/conftest.py` `ADMIN_PASSWORD`/`SESSION_SECRET`/`USE_REAL_LLM`/`_wait_http`; the sync helper: `POST /api/sync` → poll `GET /api/sync/status` to terminal; the `source_dir` fixture built under `tmp_path_factory`) with the fixture tree:
```
visible.md
.hidden/note.md <- the phase-105 subject
.venv/junk.md <- EXCLUDED_DIRS: never indexed, both states (A1)
```
Contract under test (docstring):
- anonymous: the `#git-sources-gate` sign-in gate, the manager hidden, NO `/api/git-sources` call on load, 403 on `GET`/`POST /api/git-sources` AND `PATCH /api/git-sources/{id}` with a bool-only body (the phase-89 anonymous pin extended to the toggle payload);
- A4: with the default row, a sync indexes `visible.md` ONLY — `detail.files` counts one, `.hidden/note.md` has no `documents` row, the checkbox renders UNCHECKED and no "hidden on" tag;
- A1: flipping the checkbox on (the real click) → the PATCH 200 lands (the "hidden on" tag appears, the announcer `role=status` fires the confirmation AFTER the reload line) → sync → `.hidden/note.md` IS indexed (`documents` row present; the KB catalog lists it — the tree/catalog is DB-driven, no extra surface); `.venv/junk.md` STILL absent (EXCLUDED_DIRS in both states); the checkbox re-renders CHECKED (server state);
- A2: flipping it OFF (real click) → sync → `detail.pruned` includes the hidden doc, the catalog no longer lists it, the tag is gone;
- A3: the env-fallback view (the table-empty state) renders the "from .env" tag with NO Hidden checkbox and NO "Ignore paths" button;
- a11y + error surface: the checkbox has a full accessible name containing the source location (`Index hidden folders for local source: …`), is keyboard-focusable (Tab reaches it, the global `:focus-visible` ring applies), the tag text is "hidden on" (never color alone); `#git-sources-hidden-error` exists with `role="alert"` and stays `hidden` through the happy path.
Test → contract mapping (one test per bullet, the phase-89 suite's shape):
1. `test_anonymous_gate_and_403s`
2. `test_hidden_off_by_default`
3. `test_toggle_on_indexes_hidden_folders`
4. `test_toggle_off_prunes_hidden`
5. `test_env_fallback_rows_have_no_toggle`
6. `test_toggle_a11y_and_error_surface`
2. **Regressions** — each in isolation (DB up), all green:
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` (the phase-89 suite — the PATCH body rename + the actions-cell order must not break it; its dialog still sends the list and gets byte-identical replace semantics)
- `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov`
- `uv run pytest tests/e2e/test_local_directory_sources.py -v --no-cov`
- `uv run pytest tests/e2e/test_sync_button.py -v --no-cov`
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov`
3. **Full gate** (AGENTS.md rule 9 — non-negotiable):
- `uv run pytest` (unit + integration) green
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL **>90%**
- `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` green in isolation
- `uv run ruff check . && uv run pyright` clean
4. **Commit** — one atomic Conventional-Commits commit, `--no-gpg-sign` (AGENTS.md rule 8), per the phase overview's Commit block:
```bash
git add app/ alembic/versions/0019_git_source_include_hidden.py scripts/ frontend/ tests/ TODO.md .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): per-source hidden-folders toggle — dot-prefixed paths are indexable per input"
```
(`TODO.md` is cleared to the bare `# TODO` by the conversion step BEFORE this commit lands — the items now live in this phase; if the pipeline commits per task instead, fold `TODO.md` into this phase's final commit and move the phase dir to `.agents/phases/complete/105_hidden_folders_toggle/` as the last action, per the pipeline gate.)
## Testing & Quality
- This task IS the phase's E2E + gate + commit; no new application code (the E2E may reveal a one-line fix in an earlier task's work — fix it IN that task's file, keep the suite's contract as written).
- Coverage: **>90%** on `app/` (the validate.sh gate — enforced here, not assumed).
## Completion Criteria
- [ ] `tests/e2e/test_hidden_folders_toggle.py` exists, maps 1:1 to the six contract bullets, and passes in isolation (`--no-cov`, DB up)
- [ ] The five regression E2E suites pass in isolation (the phase-89 suite first — it is the rename's canary)
- [ ] The full gate is green: unit+integration, TOTAL coverage >90%, ruff + pyright clean
- [ ] One `--no-gpg-sign` commit contains the whole phase (app + alembic 0019 + scripts + frontend + tests + the cleared `TODO.md` + the phase files)
- [ ] The TODO item is done: the owner can flip "Hidden" per input next to its Ignore paths button, and the next sync indexes (or prunes, when off) the dot-prefixed paths of that source — with `.venv`/`node_modules`/`.git`/… always excluded
File diff suppressed because one or more lines are too long
@@ -0,0 +1,50 @@
# Task 01 — `documents.created_at` + `documents.created_at_manual` (model + alembic `0020`)
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "Date should be stored in the bor database"; "the timestamp can't be null so just set it to today's date during the migration"; "This timestamp should be editable" (the manual flag is D1's phase-97-precedent half).
## Objective
Persist the document creation date: one additive, reversible migration adding `created_at` (NOT NULL, server-defaulted to the migration moment — every existing deployment row reads "today") and `created_at_manual` (default false — the owner-correction lock, D1) to `documents`.
## Work
1. `app/models.py` — the `Document` class (L101-130): add the two columns directly AFTER `indexed_at` (L112), mirroring its docstring/provenance style (`DateTime`/`Boolean`/`func`/`text` are already imported):
```python
#: The document's CREATION date (phase 106, D1/D2/D3) — sourced at
#: sync time (git last-commit date for git sources, file mtime for
#: local dirs / unpacked uploads), normalized by
#: :func:`app.rag.doc_dates.normalize_doc_date` (undetermined or
#: future → today; UTC). NOT NULL: pre-phase-106 rows backfill to
#: the migration moment (≈ today — the owner's instruction) and the
#: next sync refreshes them (the importer's unchanged path,
#: task 04 — a sync may move a date OLDER, D4). Distinct from
#: ``indexed_at`` (the INDEX time, untouched).
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
#: True only while ``created_at`` is the OWNER'S correction (phase
#: 106, D1 — the ``folder_summaries.manually_edited`` phase-97
#: precedent): set ONLY by ``PATCH /api/documents/date``
#: (task 05); the sync-time importer SKIPS the refresh on a manual
#: row (the correction survives syncs, D4) and a content change
#: RESETS both the date and the flag (a new version = a new date).
created_at_manual: Mapped[bool] = mapped_column(
Boolean, default=False, server_default=text("false"), nullable=False
)
```
(If the module header's one-line `documents` field inventory names `indexed_at`, add `created_at`/`created_at_manual` (phase 106) to the parenthetical.)
2. `alembic/versions/0020_documents_created_at.py` (NEW — the house format of `0019_git_source_include_hidden.py`):
- `revision = "0020"`, `down_revision = "0019"`.
- `upgrade()`: `op.add_column("documents", sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False))` then `op.add_column("documents", sa.Column("created_at_manual", sa.Boolean(), server_default=sa.text("false"), nullable=False))`.
- `downgrade()`: `op.drop_column("documents", "created_at_manual")` then `op.drop_column("documents", "created_at")`.
- Module docstring: the phase-106 provenance (what the date is, D1/D2/D3/D4, the NOT-NULL backfill-to-today behavior, one additive reversible migration, A13).
3. Tests — `tests/integration/test_migration_0020.py` (NEW), mirroring `tests/integration/test_migration_0019.py` VERBATIM in shape (the real-Alembic `alembic` fixture that starts/ends at head; `information_schema` column-contract assertions; the explicit 0019 → 0020 step so later migrations cannot break the pins): the 0019 `documents` schema (incl. `indexed_at`, `summary`) survives the upgrade; both new columns exist with the full contract — `timestamp with time zone` NOT NULL default `now()` / `boolean` NOT NULL default `false`; a `documents` row inserted while the DB is at `0019` backfills `created_at ≈ now()` (assert within a few seconds of the upgrade moment) and `created_at_manual is False`; downgrade to `0019` → both columns GONE (A13) while the row + its content survive; upgrade back to `0020` → both columns back (round-trip); the ORM contract agrees — a freshly inserted `Document` (nothing passed) reads `created_at_manual is False` + non-null `created_at`, and an explicit `created_at` + `created_at_manual=True` round-trips through a fresh session.
4. Run `uv run pytest tests/integration/test_migration_0020.py -q` (DB up) + `uv run alembic upgrade head` on the dev/test DB — green.
## Testing & Quality
- Integration: the migration upgrade/downgrade + server-default pins above ARE this task's layer (no importer behavior yet — task 04 writes these columns).
- Coverage: **>90%** on `app/` (model/migration-only change — the validate.sh gate).
## Completion Criteria
- [ ] `Document.created_at` (NOT NULL, `server_default=func.now()`) and `Document.created_at_manual` (NOT NULL, `server_default=text("false")`) exist with the D1/D2/D3/D4 provenance comments
- [ ] `alembic/versions/0020_documents_created_at.py` upgrades from `0019` and downgrades cleanly; the dev/test DB is at head; existing rows read `created_at ≈ now()` (the backfill) and `created_at_manual is False`
- [ ] Fresh-row-defaults + explicit-values round-trip tests pass; existing suites stay green
- [ ] `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,73 @@
# Task 02 — `app/rag/doc_dates.py`: the date normalization choke point (D3)
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "If the date of a document can't be determined or is in the future then assume that document was created today."
## Objective
One pure, stdlib-only module that every document date passes through — the importer (task 04) and the date-edit API (task 05) both call it, so the today/future/naive rules live in exactly one place and can be pinned by unit tests without a database.
## Work
1. `app/rag/doc_dates.py` (NEW):
```python
"""Document-creation-date sourcing + normalization (phase 106, D2/D3).
Every document date the importer writes and every date the owner
edits passes through :func:`normalize_doc_date` — the single choke
point for the owner's rules: an UNDETERMINED date (no source signal)
and a FUTURE date (beyond a small clock-skew tolerance) both assume
the document was created TODAY (UTC). Naive source timestamps (zip
DOS mtimes, tar mtimes, git-free fallbacks) are tz-agnostic epoch-
based values rendered as UTC; aware ones are converted to UTC.
"""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from pathlib import Path
#: Clock-skew tolerance (D3): a source date up to this far in the
#: FUTURE is a drifting clock, not a future document — it keeps its
#: date. Beyond it, the owner's rule applies (→ today).
FUTURE_SKEW_TOLERANCE = timedelta(days=1)
def normalize_doc_date(raw: datetime | None, now: datetime | None = None) -> datetime:
"""*raw* → the stored UTC creation date (the D3 rule, pinned).
``now`` is injectable (tests); it defaults to
``datetime.now(UTC)``. ``raw=None`` (undetermined) → *now*;
naive *raw* → treated as UTC; aware *raw* → converted to UTC;
*raw* beyond *now* + :data:`FUTURE_SKEW_TOLERANCE` → *now*.
The result always carries full precision (no date-truncation —
the display formats, the storage doesn't).
"""
```
Plus:
```python
def file_mtime_datetime(path: Path) -> datetime:
"""The file's mtime as an aware UTC datetime (the D2 fallback).
Epoch mtimes are tz-agnostic — UTC is the correct rendering
(zip DOS timestamps and tar mtimes pass through the same
:func:`normalize_doc_date` after unpacking, task 03).
"""
return datetime.fromtimestamp(path.stat().st_mtime, tz=UTC)
```
Implementation notes: for the naive case, attach UTC (`raw.replace(tzinfo=UTC)`) rather than assuming local time (the homelab host TZ is irrelevant — source mtimes are epoch values); for the aware case, `raw.astimezone(UTC)`; compare the future check in aware space.
2. `tests/unit/test_doc_dates.py` (NEW) — the boundary matrix (pure function, no DB):
- `None` → exactly `now` (inject a fixed `now`);
- naive `2020-05-01T12:00` → `2020-05-01T12:00+00:00` (UTC-attached, not local-converted);
- aware `2020-05-01T08:00-04:00` → `2020-05-01T12:00+00:00` (converted);
- future by 23 h (just INSIDE the tolerance) → keeps its date;
- future by 25 h (beyond) → `now`;
- exactly `now + FUTURE_SKEW_TOLERANCE` → keeps its date (the boundary is strict-greater);
- `file_mtime_datetime` on a tmp file with a `os.utime`'d mtime → the expected UTC datetime (±1 s tolerance for mtime granularity);
- the module imports nothing but stdlib (a source-level pin, the house pattern — grep the file for `import` lines).
3. Run `uv run pytest tests/unit/test_doc_dates.py -q` — green.
## Testing & Quality
- Unit: the matrix above IS this task's layer (the callers land in tasks 04/05).
- Coverage: **>90%** on `app/` (new module fully covered — the validate.sh gate).
## Completion Criteria
- [ ] `app/rag/doc_dates.py` exists with `FUTURE_SKEW_TOLERANCE` (1 day), `normalize_doc_date` (None→today, naive→UTC, aware→convert, future-beyond-tolerance→today, full precision kept) and `file_mtime_datetime`
- [ ] `tests/unit/test_doc_dates.py` pins the full boundary matrix (incl. the strict-greater 1-day boundary and the naive-is-UTC rule) and passes
- [ ] No non-stdlib imports in the module; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,38 @@
# Task 03 — Source date extraction: mtime-preserving unpack + `file_commit_dates` (D2/D10)
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — dates come "via their git timestamp or via file metadata (hopefully) preserved in the tar or zip archive process."
## Objective
Make the source-of-truth date actually EXIST at the filesystem/checkout level: the zip/tar unpacker restores member mtimes (uploads stop losing their dates), and `scripts/git_sync.py` gains a single-call per-file last-commit-date walk (the A11 git site) with the verified shallow-vs-local behavior pinned.
## Work
1. `app/rag/archive_upload.py` — mtime preservation (D2), regular files only, every safety check + the zip-bomb cap UNCHANGED:
- `_unpack_zip` (L214-238): after the `with zf.open(member) as src: _write_capped(...)` block for a regular file, restore the member's DOS mtime:
```python
mtime = datetime(*member.date_time, tzinfo=timezone.utc).timestamp()
os.utime(dest, ns=(mtime, mtime))
```
(`datetime`/`timezone` from the stdlib — add the import; DOS `date_time` is a tz-agnostic epoch value, UTC-rendered exactly like an mtime — the task-02 convention.)
- `_unpack_tar` (L241-270): in the `member.isreg()` branch, after `_write_capped(...)`: `os.utime(dest, ns=(member.mtime, member.mtime))` (tar `mtime` is epoch seconds — `ns=` accepts a float seconds value).
- Directories, symlinks, and hardlinks are untouched (only regular files are ever indexed). A failed unpack still removes the partial tree (the `utime` calls sit inside the existing try/except flow — an `OSError` there is caught by `unpack_archive`'s handler exactly like any other write failure).
- Update the module docstring's guarantees list with the mtime-preservation line (phase 106, D2).
2. `scripts/git_sync.py` — `file_commit_dates(dest: Path) -> dict[str, datetime]` (NEW public function, exported in `__all__`):
- Runs ONE `run_git(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest)` (the A11 single-invocation site — the module docstring's git-inventory sentence gains this command).
- Parse: lines matching `@@` start a commit (ISO-strict `%cI` → `datetime.fromisoformat`, aware); subsequent non-empty lines until the next `@@`/blank-then-`@@` are repo-relative paths (split on whitespace like git's name-only output, normalize `\` → `/`, lstrip a leading `/`). Per path, the FIRST sighting wins (the walk is newest-first) — that is the file's last-commit date.
- **Fail-soft (pinned):** `GitSyncError` (git missing/failed) or ANY parse anomaly → `logger.warning` + return `{}` — the importer (task 04) falls back to file mtimes; a date walk must never break a sync.
- Module docstring: what it is, the one-git-call contract, and the VERIFIED checkout behavior (owner-permission source: this phase's ask, 2026-09-13): a local-path checkout cloned by `clone_or_pull` keeps FULL history (`--depth` is ignored in local clones — git's own warning) → TRUE per-file dates; a URL-transport checkout is shallow and git reports the TIP commit as every existing file's last commit (the shallow boundary is each file's history root) → a uniform per-repo tip date (D10 — no intra-repo distortion, real cross-repo signal).
3. Tests:
- `tests/unit/test_archive_upload_dates.py` (NEW): build in `tmp_path` — a zip with one member whose `ZipInfo.date_time` is an old fixed tuple (e.g. `(2020, 1, 2, 3, 4, 6)` → 2020-01-02 03:04:06 UTC) and a tar with one member `mtime=1577934246` (2020-01-02 03:04:06) — `unpack_archive` → the extracted file's `st_mtime` equals the member's (±1 s, mtime granularity). The existing archive-upload suite (`tests/unit/test_archive_upload*.py` — glob to find it) stays green (no safety behavior moved).
- `tests/integration/test_git_file_dates.py` (NEW — real `git` in the test environment, the `test_import_docs_git.py` precedent for git availability; skip cleanly if `git` is absent, that suite's pattern): in `tmp_path_factory` build a scratch repo with two files committed at controlled `GIT_COMMITTER_DATE`s (file A 2020-01-02, file B touched again 2024-06-15 — the 2026-09-13 verification recipe): (a) `clone_or_pull` from the LOCAL path → `file_commit_dates` returns A's 2020 date and B's 2024 date (true per-file); (b) a `file://` shallow clone (run `git clone --depth 1 file://…` directly in the test — the test harness, not `clone_or_pull`, makes this one) → EVERY file's date is the TIP commit's (2024-06-15) (D10 pinned); (c) a directory without `.git` / a `git` failure → `{}` (fail-soft, no raise).
4. Run `uv run pytest tests/unit/test_archive_upload_dates.py tests/integration/test_git_file_dates.py -q` (DB up for the integration file's `db` fixture only if used — keep it DB-free: `file_commit_dates` takes a path, no session) — green.
## Testing & Quality
- Unit: the zip/tar mtime pins + the safety-suite regression.
- Integration: the git walk against real scratch repos (both checkout kinds + the fail-soft path) — DB-free.
- Coverage: **>90%** on `app/` (the unpacker branches + the new parser fully covered — the validate.sh gate; `scripts/` is outside the `app/` coverage denominator but the integration suite pins its behavior).
## Completion Criteria
- [ ] A zip and a tar with old member timestamps unpack to files carrying those mtimes (regular files only; safety/cap behavior byte-identical — the existing suite green)
- [ ] `scripts/git_sync.py::file_commit_dates` exists, is the ONLY new git invocation (through `run_git`), returns `{path: last_commit_datetime}` with first-sighting-wins parsing, and fails soft to `{}`
- [ ] The verified behavior is pinned: local clone → true per-file dates; `file://` shallow clone → tip date for every file
- [ ] `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,43 @@
# Task 04 — Importer: source the date on every upsert, refresh on unchanged, protect manual (D4)
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "Date should be stored in the bor database and updated when sources are synced"; "It's totally fine for a sync to cause a document or folder's date to get older."
## Objective
The importer writes `documents.created_at` from the source (git map → mtime fallback → `normalize_doc_date`) on every add/update, REFRESHES it on the unchanged path (the backfill-correction case — an existing row stamped "today" by the migration gets its real date on the next sync even when the content didn't change), skips the refresh on manually corrected rows, and counts date-only refreshes in a new additive `dates_updated` counter. Both live entry points (Sync button, CLI) feed the git-date map.
## Work
1. `app/rag/importer.py`:
- `ImportSummary` (L64-111): add `dates_updated: int = 0` (docstring: files whose `created_at` was refreshed on the UNCHANGED path — content untouched, D4) and the `dates_updated=%d` term in `log()`'s `import: summary …` line (PLAN §9 — append it AFTER `summary_errors`, before `formats`, so existing prefix assertions survive).
- `import_sources` (L213-340): new keyword `doc_dates_by_root: dict[str, dict[str, datetime]] | None = None` (AFTER `include_hidden_by_root`), docstring paragraph in the `include_hidden_by_root` style: keyed by `str(root)` — the root string exactly as passed in *sources*; maps a source-relative POSIX path to its RAW source date (git last-commit, task 03); ONLY git roots are listed — unlisted roots (local dirs, uploads) take the mtime fallback; `None` (default) changes nothing for existing callers (the mtime fallback applies — which IS the behavior change: unchanged files now refresh their date, D4). In the processing loop (L281+): `dates_map = (doc_dates_by_root or {}).get(str(root), {})` and pass `raw_date=dates_map.get(rel)` into `_index_file`. The progress pre-walk is UNTOUCHED (dates change no file count).
- `_index_file` (L376-471): new keyword `raw_date: datetime | None = None`:
- Resolve once, up top: `if raw_date is None: raw_date = file_mtime_datetime(full_path)` (import from `app.rag.doc_dates`).
- **added branch** (L409-417): `created_at=normalize_doc_date(raw_date)` on the new `Document(…)`; `created_at_manual` stays the column default (`False`).
- **updated branch** (L418-421): `doc.created_at = normalize_doc_date(raw_date)` and `doc.created_at_manual = False` (a content change resets a previous correction — the correction referred to the old content; D4).
- **unchanged branch** (L401-404, currently the early return): BEFORE returning — if `doc.created_at_manual` → return unchanged (log the existing line, the correction survives — D1/D4); else `target = normalize_doc_date(raw_date)`; if `target != doc.created_at` → `doc.created_at = target`, `session.commit()`, `summary.dates_updated += 1`, `logger.info("import: date-refreshed source=%s path=%s date=%s", source, rel, doc.created_at.isoformat())`; return. (A date-only refresh is still counted `unchanged` — `added/updated/pruned` are untouched → no `sources_meta` bump, no overview/folder-summary regeneration: the gate keys on content, D4.)
- Module docstring: the Scope/workflow paragraph gains the date rule (two sentences — sourced on add/update, refreshed on unchanged unless manual, D2/D4).
2. `app/api/sync.py` — `_run_sync` (the per-row loop L233-263): build `doc_dates_by_root: dict[str, dict[str, datetime]] = {}` alongside the other two maps; for `kind=git` rows, AFTER `clone_or_pull` returns: `doc_dates_by_root[str(root)] = file_commit_dates(root)` (import `file_commit_dates` next to the existing `clone_or_pull` import, L100); local rows add nothing (mtime fallback). Pass `doc_dates_by_root=doc_dates_by_root` to `import_sources` (L300-303). The success `detail` dict (L370-382) gains `"dates_updated": summary.dates_updated` (additive key, after `"summary_errors"`). The module docstring's pipeline step 4 gains the third-map clause.
3. `scripts/import_docs.py` — `_resolve_sources` (L168-226): build the same map for the git rows it clones (after the `clone_or_pull` call, L220) and return it as a 4th tuple element `(sources, ignore_by_root, include_hidden_by_root, doc_dates_by_root)` — manual `--source` dirs and env-fallback rows contribute nothing (no row, no clone → no map entry → mtime fallback); update the return docstring. `main` unpacks the 4-tuple (the L269-ish unpack) and passes the map to `import_sources` (L331-334). Module docstring updated.
4. `scripts/load_test_kb.py` — untouched (the `None` default).
5. Tests:
- `tests/unit/test_importer_dates.py` (NEW — the `tests/unit/test_importer_include_hidden.py` scaffolding: fake LLM from `tests/fakes.py` + a tmp fixture tree; run against the `db` session the house unit pattern uses for importer tests — read `test_importer_include_hidden.py` first and mirror its session handling):
- a file `os.utime`'d to 2020-01-02 imports with `created_at` ≈ that instant (added);
- unchanged re-import with the mtime moved to 2021 → `created_at` refreshed, `summary.unchanged == 1` AND `summary.dates_updated == 1` (content counts preserved);
- unchanged re-import with the same mtime → `dates_updated == 0`;
- a row with `created_at_manual=True` + moved mtime → date UNTOUCHED (the D1 lock) and `dates_updated == 0`;
- a content change on a manual row → date reset from source AND `created_at_manual is False`;
- `doc_dates_by_root` map entry beats the mtime (the git case: map says 2020, mtime says now → 2020 stored);
- a future mtime (2030) → `created_at` folds to today (D3 through the importer).
- `tests/integration/test_importer_dates.py` (NEW — real Postgres, the `tests/integration/test_importer_e2e.py` fake-LLM pattern): the backfill-correction case — a row first imported with a "today" mtime, its file then `os.utime`'d back to 2019 (content identical) → the second `import_sources` run stores the 2019 date (`added/updated/pruned` all 0, `dates_updated == 1`) AND `sources_meta`'s version is UNBUMPED (the date-only-refresh gate, D4 — seed the version row first, read it after); a pruned/manual matrix as needed for coverage.
- Regression sweep (run, and update ONLY exact-string pins that break — the `import: summary` line gained a term and the sync `detail` gained a key): `uv run pytest tests/unit/test_importer*.py tests/integration/test_importer*.py tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -q`.
6. Run the full unit + integration importer slice — green.
## Testing & Quality
- Unit: the semantic matrix above (added/updated/unchanged × manual × map vs mtime × future) against the fake LLM.
- Integration: real Postgres for the backfill-correction + no-version-bump pins.
- Coverage: **>90%** on `app/` (the new branches in `importer.py` + the sync detail all covered — the validate.sh gate).
## Completion Criteria
- [ ] `import_sources` accepts `doc_dates_by_root` (str(root)-keyed, git-only, None = byte-identical for existing callers); `_index_file` sources added/updated dates from the map → mtime fallback → `normalize_doc_date` and resets `created_at_manual` on content change
- [ ] The unchanged path refreshes `created_at` (date may go OLDER — no monotonic guard), counts it in `dates_updated` (new field + log term), skips manual rows, and NEVER counts toward `added/updated/pruned` (no `sources_meta` bump, no overview/summary regeneration)
- [ ] The Sync button and the CLI both feed the map (git rows only, after clone); `scripts/load_test_kb.py` untouched; the success sync `detail` carries `dates_updated`
- [ ] The regression slice above is green (exact-string log/detail pins updated in place where they break); `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,45 @@
# Task 05 — Date API surface: reads, the admin date edit, and the tree's dates (D7/D8/D9)
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "This timestamp should be editable so users can correct for errors"; the catalog needs the file dates + folder last-updated (the UI in task 08 renders exactly what this task serves).
## Objective
Serve the date everywhere the UI (task 08) and the viewer need it — `GET /api/docs`, `GET /api/documents/content`, and `GET /api/docs/tree` (files: `created_at`; folders/sources: the derived subtree-max `updated_at`, D9) — and add the admin-only `PATCH /api/documents/date` (set + revert, the phase-57 gate/idiom, D7).
## Work
1. `app/schemas.py`:
- `DocSummary` (L228-235): add `created_at: str` (ISO-8601 — the `indexed_at` docstring style: "verbatim from the row").
- `DocContent` (L343-358): add `created_at: str` (after `summary`, before `content` — group the metadata).
- NEW `DateUpdate` (the `SummaryUpdate` shape, L361-373): `source: str`, `path: str`, `date: str | None` (docstring: an ISO date `YYYY-MM-DD` or full ISO datetime; **null/absent = the CLEAR** — drop the manual flag, the stored date stands until the next sync refresh; a malformed non-null value 422s through Pydantic… correction: `str` passes any string — the handler parses (step 3); the 422 comes from the handler, not the model, so the error detail can name the field).
- NEW `DateResult`: `source: str`, `path: str`, `created_at: str`, `created_at_manual: bool` (echoes the stored state — the viewer re-renders from it).
- `KbTreeFile` (L241-256): add `created_at: str` (verbatim from the catalogue row — the `indexed_at` field's docstring pattern).
- `KbTreeFolder` (L259-290) and `KbTreeSource` (L293-320): add `updated_at: str | None` (docstring: the subtree's MAX document `created_at` — D9, derived, never stored; `null` for a 0-document source, the `summary: str | None` shape).
2. `app/api/docs.py`:
- `list_indexed_documents` (L80-119): add `Document.created_at` to the select AND the `group_by` (the `indexed_at` twin, L104/L107); `DocSummary(..., created_at=row.created_at.isoformat())`.
- `get_document_content` (L121-162): `created_at=doc.created_at.isoformat()` in the `DocContent` (L161 site).
- NEW `PATCH /api/documents/date` (route order: next to `update_document_summary`, L164-232 — `require_admin` dependency, the phase-57 gate):
```python
@router.patch("/documents/date", response_model=DateResult)
def update_document_date(payload: DateUpdate, db: Session = Depends(get_db),
_admin: None = Depends(require_admin)) -> DateResult:
```
Logic (DB-only — the `/documents/content` row-lookup rule, no filesystem, no LLM/embedding call — a date is never embedded, the phase-57 no-LLM contrast): look up the row by `(source, path)` → none → 404 `{"detail": "document not found"}` (row-lookup semantics, the traversal-string-is-not-a-row note). `payload.date` truthy → `parsed = datetime.fromisoformat(payload.date)` (a bare `YYYY-MM-DD` and full ISO datetimes both parse; `ValueError` → 422 `{"detail": "date must be an ISO date or datetime (e.g. 2024-06-15)"}`) → `doc.created_at = normalize_doc_date(parsed)` (import from `app.rag.doc_dates` — D3: a manually set FUTURE date also folds to today, consistency with the sourced path) → `doc.created_at_manual = True`. `payload.date` falsy (null/absent — the CLEAR) → `doc.created_at_manual = False` only (the stored date stands; the next sync refreshes it — the API cannot re-read the source, D7). `db.commit()`; return `DateResult` with the stored `created_at.isoformat()` + flag.
- The tree (task-05 half of D8/D9): `TreeDocRow` (L296-302) becomes the 6-tuple `(source, path, title, chunks, indexed_at, created_at)` (both ISO strings — the builder stays pure over plain types); `_folder_counts` / `_level_children` / `_source_node` thread a 6th element through their tuple unpacks (the `_`-named slots gain the date) and `_level_children`/`_source_node` compute each folder/source's `updated_at`: the MAX of the direct files' `created_at` and the children's `updated_at` values (ISO-8601 strings compare correctly lexicographically — they're all the same `isoformat()` shape; document that in the builder docstring) — `None` when the node has no documents at all (the 0-document registered source). `KbTreeFolder(…, updated_at=…)` / `KbTreeFile(…, created_at=…)` / `KbTreeSource(…, updated_at=…)` at their construction sites (L342-375, L467-490). `list_kb_tree` (L492-551): add `Document.created_at` to the query's select + group_by (the `indexed_at` twin, L543-546) and the `doc_rows` comprehension. The `build_kb_tree` docstring gains the D9 clause (updated_at = subtree max, derived, None for empty).
3. Tests:
- `tests/unit/test_kb_tree_builder.py` (extended — the pure builder): file nodes carry `created_at` verbatim; a nested fixture asserts each folder's + the source's `updated_at` = the subtree max (a deeper file's date wins over a shallow sibling's); a 0-document registered source → `updated_at is None` and no children; the ls↔`group_folder_listing` cross-check tests (L210-280) still pass with the extended tuples (task 06 changes the agent side — until then the rows stay 6-tuples on BOTH sides only after task 06; for THIS task the cross-check compares file `(path, title[, chunks, indexed_at])` projections — read the current assertions and keep them green: the tree builder's file tuples are internal to the builder, the cross-check uses the builder's OUTPUT nodes, so it should pass unchanged — verify and pin).
- `tests/integration/test_docs_api_dates.py` (NEW — the `tests/integration/test_docs_api.py` scaffolding: real app + `db` fixture, an admin cookie where that suite gets one): seed two documents in a nested folder (distinct `created_at`s via direct row writes):
- `GET /api/docs` (admin) reports `created_at` per row (and `indexed_at` unchanged);
- `GET /api/documents/content` carries `created_at`;
- `GET /api/docs/tree` — the file node's `created_at` verbatim, the parent folder's and the source's `updated_at` = the max, a registered-but-empty source → `updated_at: null`;
- the PATCH matrix — set `2020-01-02` → 200 + response echoes the stored ISO + `created_at_manual: true` + a re-GET confirms; set a full ISO datetime → accepted; malformed `"not-a-date"` → 422 (the detail names the field); `date: null` → 200 + `created_at_manual: false` + the stored date UNCHANGED; a future date `"2999-01-01"` → stored `created_at` folds to today (D3); unknown `(source, path)` → 404 `document not found`; anonymous → 403 (the gate).
4. Run `uv run pytest tests/unit/test_kb_tree_builder.py tests/integration/test_docs_api_dates.py -q` (DB up) — green.
## Testing & Quality
- Unit: the pure builder's date threading (max computation, None-for-empty, verbatim file dates).
- Integration: the full API matrix (reads + PATCH set/malformed/clear/future/404/403) against real Postgres.
- Coverage: **>90%** on `app/` (the new route + the builder branches covered — the validate.sh gate).
## Completion Criteria
- [ ] `GET /api/docs`, `GET /api/documents/content`, and `GET /api/docs/tree` serve `created_at` (files) and `updated_at` (folders/sources — subtree max, `null` when empty, derived in the pure builder, D9)
- [ ] `PATCH /api/documents/date` is admin-only, DB-only, no-LLM: set (ISO date or datetime, future folds to today, `created_at_manual=true`), clear (null → flag drops, date stands), 422 malformed, 404 unknown pair, 403 anonymous — the phase-57 split intact (viewer stays user-gated)
- [ ] `tests/unit/test_kb_tree_builder.py` + `tests/integration/test_docs_api_dates.py` pass; existing docs-API suites stay green
- [ ] `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,56 @@
# Task 06 — LLM surfaces: the date rides every document the model sees (D5)
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "After this phase, all documents must include the date when fed to the LLM."
## Objective
The date appears on all three document surfaces the model reads — the HIGH prompt's `<document>` block (the retrieved top-N), the `read` tool result (agent-fetched), and the `ls` file lines (catalog drill) — with the retriever's raw-SQL detached rows gaining the column, the E2E mock's prompt regex updated in lockstep (house rule), and the existing format pins updated mechanically. No persona/teaching copy changes (phase 03 convention).
## Work
1. `app/rag/retriever.py` — the column plumbing (the vector path returns ORM rows — the column comes for free; only the two raw-SQL detached-row paths need it):
- `_LEXICAL_SQL` (L140-157): add `d.created_at AS created_at,` (after `d.indexed_at`).
- `_NAME_HIT_SQL` (L277-300): same column addition.
- The two detached `Document(…)` reconstructions (L360-372 in `_name_hit_chunks`, L414-424 in `_lexical_candidates`): pass `created_at=row.created_at`.
2. `app/rag/prompts.py` — `build_high_prompt` (L371-376): the block becomes
```python
blocks = [
f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}" '
f'date="{doc.created_at:%Y-%m-%d}">\n'
f"{doc.content}\n"
"</document>"
for doc in documents
]
```
(the UTC date part; the attribute APPENDED after `title` — the only position, always present since `created_at` is NOT NULL). Update the function's docstring line describing the block's identity attributes. The deflection path (`build_deflect_prompt` — titles only) is untouched, and its byte-identity pins hold (no documents involved).
3. `app/rag/agent.py` — two surfaces:
- **`read` result** (L1173-1185): the FIRST line stays `Document {doc.source}/{doc.path}:` BYTE-IDENTICAL (the E2E mock's `_READ_RESULT_PREFIX` header contract — `_read_results` strips exactly that header to recover the path); the date is the SECOND line, both the truncated (L1176-1181) and plain (L1183) results:
```python
f"Document {doc.source}/{doc.path}:\ndate: {doc.created_at:%Y-%m-%d}\n{doc.content[:cap]}\n{TRUNCATION_MARKER}\n…"
```
/ `f"Document {doc.source}/{doc.path}:\ndate: {doc.created_at:%Y-%m-%d}\n{doc.content}"`.
- **`ls` file line** (appended — NEVER inserted before `title`, where the mock's non-greedy `path` capture would swallow it): `_source_document_rows` (L658-669) returns `(path, title, created_iso_date)` triples (add `Document.created_at` to the select, format `%Y-%m-%d` in the comprehension); `group_folder_listing` (L707-780) — `rows: Sequence[tuple[str, str, str]]`, the file output becomes `(source, path, title, date)` 4-tuples (the subfolder tuples + count are untouched); `render_folder_listing` (L856-905) renders `f"source: {source} | path: {path} | title: {title} | date: {date}"`; the `ls_folder` (L782-796) + `NOT_A_FOLDER` branch (L1121-1130) + `ls_top` source lines are UNCHANGED in shape (source/folder lines carry no date — only FILE lines are documents). Docstrings updated (the `LS_MAX_FILE_LINES` comment's line-format phrase, the module header's L73 format line).
4. `app/api/docs.py` — the ls↔tree cross-check (D9/phase-97 invariant): the docstrings at L326/L376/L461 name the compared shapes — update them to the extended file tuples; `build_kb_tree`'s OUTPUT nodes already carry `created_at` (task 05), so the cross-check test's node-side comparisons gain the date field (step 6).
5. `tests/e2e/mock_llm.py` — `_DOCUMENT_BLOCK_RE` (L785-789): make the date attribute an OPTIONAL group so the mock tolerates pre- and post-phase shapes:
```python
_DOCUMENT_BLOCK_RE = re.compile(
r'<document source="(?P<source>[^"]+)" path="(?P<path>[^"]+)" '
r'title="[^"]*"(\sdate="[^"]*")?>\n(?P<content>.*?)\n</document>',
re.S,
)
```
`title=` docstring comment (L780-784) updated. `_CATALOG_LINE_RE` (L873-875) and `_READ_RESULT_PREFIX` (L867) are UNCHANGED by design (verified: the appended ` | date: …` lands in the greedy `title: .+$` tail; the read first line is byte-identical).
6. Tests + pin updates:
- `tests/unit/test_prompts_dates.py` (NEW): the HIGH block renders `<document source="S" path="P" title="T" date="YYYY-MM-DD">` with the date = the row's UTC date part (inject a fixed `created_at`); the deflection prompt is byte-identical to the pre-phase text for the same inputs (the A8 byte-identity contract holds); the `read` result — both shapes — has the identical first line and the `date:` second line (truncated variant: marker + notice still follow); the `ls` line ends with ` | date: YYYY-MM-DD` and the 50-line cap note is unchanged.
- `tests/integration/test_agent_tools_dates.py` (NEW — the `tests/integration/test_agent_tools.py` scaffolding): real rows with distinct `created_at`s — execute a `read` tool call → result second line = the stored date, first line unchanged; an `ls` drill → every file line carries its date in the appended field.
- **Existing-pin sweep (mechanical, test files only)** — run and update exact-string pins that break: `uv run pytest tests/unit/test_retriever.py tests/unit/test_agent.py tests/unit/test_kb_tree_builder.py tests/integration/test_agent_tools.py tests/integration/test_name_hit_lexical.py tests/integration/test_chat_api.py -q` (the detached-`Document` constructors in test fixtures that set fields explicitly may need `created_at` where the SQL now returns it — the model default covers ORM inserts; raw-SQL projections are app-side, so fixture rows created via the ORM already have the column).
7. Run the sweep + new suites — green.
## Testing & Quality
- Unit: prompt block / read line / ls line format pins (the byte-identity contracts).
- Integration: the tool surfaces against real rows.
- Coverage: **>90%** on `app/` (the new SQL columns + render branches covered — the validate.sh gate).
## Completion Criteria
- [ ] The HIGH prompt's `<document>` block carries `date="YYYY-MM-DD"` (after `title`, always present); the deflection prompt stays byte-identical (A8)
- [ ] The `read` result carries `date: YYYY-MM-DD` as its second line (first line byte-identical — the mock header contract); the `ls` FILE line ends with ` | date: YYYY-MM-DD` (source/folder lines unchanged); the ls↔tree cross-check still holds
- [ ] `_DOCUMENT_BLOCK_RE` is date-tolerant (optional group); `_CATALOG_LINE_RE`/`_READ_RESULT_PREFIX` untouched; the mock serves post-phase prompts correctly (a quick smoke: `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov` green in isolation, DB up)
- [ ] The existing-pin sweep is green (test-file-only updates); `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,61 @@
# Task 07 — Recency boost: newer documents rank higher, without breaking retrieval (D6)
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "Newer documents should be ranked higher in retrieval somehow, or at least given a boost, without breaking the existing retrieval process (so make sure to test with documents that have the correct answer but are older against documents that are similar and newer but don't quit correctly answer the question). This will be a fine line to walk, so testing is crucial here."
## Objective
A small, env-tunable, kill-switchable additive recency term on the RRF-fused score — applied once in `retrieve()` after `fuse()` — so a fresh document gets a bounded head start on near-ties while an older document that ACTUALLY answers the question keeps its rank. The owner's scenario is pinned by a permanent real-Postgres battery with deterministic axis vectors. The A7 math, the A8 cosine gate, `query_log.top_score`, and the never-truncated contract are untouched.
## Work
1. `app/config.py` — the hybrid block (L163-174, after `rrf_k`):
```python
#: Recency boost on the RRF-fused retrieval score (phase 106, D6): the
#: MAXIMUM additive score a zero-age document gets —
#: ``fused + recency_boost * exp(-age_days / recency_half_life_days)``
#: (``app.rag.retriever.apply_recency_boost``, applied in
#: ``retrieve()`` after ``fuse()``). ``0`` = off — the pre-phase
#: ranking is byte-identical (the kill switch); negative values fail
#: startup loudly (the ``agent_max_rounds`` validator pattern).
#: 0.001 ≈ a 1-2 rank head start on a 60+ RRF scale — enough to break
#: near-ties toward the newer document, far below the gap between a
#: document that answers and one that merely resembles (the
#: phase-106 fine-line battery pins it).
recency_boost: float = 0.001
#: Age (days) at which the recency boost halves (phase 106, D6).
#: ``<= 0`` fails startup loudly (same validator family).
recency_half_life_days: int = 365
```
Add the startup validator (find `agent_max_rounds`'s field-validator and follow it — fail loudly naming the field): `recency_boost < 0` → error; `recency_half_life_days <= 0` → error. `.env.example` — document `BOR_RECENCY_BOOST` + `BOR_RECENCY_HALF_LIFE_DAYS` (the hybrid section, the existing comment style).
2. `app/rag/retriever.py`:
- NEW pure function (module-level, next to `fuse`):
```python
def apply_recency_boost(
chunks: Sequence[RetrievedChunk],
*,
now: datetime | None = None,
weight: float | None = None,
half_life_days: int | None = None,
) -> list[RetrievedChunk]:
```
Defaults from `get_settings()` when omitted; `now` defaults to `datetime.now(UTC)`. For each chunk: `age_days = max(0.0, (now − doc.created_at).total_seconds() / 86400.0)` (a future `created_at` clamps to 0 — consistent with D3's today-folding), `score = score + weight * math.exp(−age_days / half_life_days)` (import `math`; `replace(rc, score=new_score)` — never mutate inputs, the `fuse` convention). Return the list re-sorted with the EXISTING deterministic key `(−score, −cosine, document.path, position)` — with `weight=0` every score is untouched and the order is byte-identical (pinned). Docstring: the D6 contract, the magnitude rationale (0.001 ≈ 1-2 ranks on the k=60 scale — rank 1 vs 2 in one list differs by ~0.00026, rank 1 vs 10 by ~0.0021), the untouched surfaces (A8 gate = cosine, `query_log.top_score` = cosine, `weak_hit_titles` = titles only, the never-truncated top-N), and the single-apply-site rule (`retrieve()` only — chat API + `eval_retrieval` inherit it).
- `retrieve()` (L398-425): after `return fuse(vector, lexical, settings.rrf_k)` → apply: `fused = fuse(...)`; `if settings.recency_boost > 0: return apply_recency_boost(fused)`; `return fused` (weight-0 callers pay nothing).
3. `scripts/eval_retrieval.py` — the printed top-N table gains two columns: the document's `created_at` (UTC date) and the post-boost effective score (the script calls `retrieve()`, which now applies the boost — print both the raw fused and effective where they differ, or just effective + date; keep the verdict column). Docstring line updated.
4. Tests:
- `tests/unit/test_retriever_recency.py` (NEW — fake rows, no DB): age 0 → `+weight` exact; age = half-life → `+weight*exp(-1)` (±1e-9); age 10× half-life → ~`+weight*exp(-10)` (assert `< weight * 1e-3`); future date → full weight (the clamp); `weight=0` → the returned list's `(score, order)` is byte-identical to the input (the kill-switch pin); a tie on raw score breaks toward the newer document; the sort key's `(path, position)` tie-break still applies when scores AND cosines are equal (two docs, same age).
- `tests/integration/test_recency_boost.py` (NEW — real Postgres, `tests/integration/test_name_hit_lexical.py`'s axis-vector idiom VERBATIM: `D=768` unit vectors, exact cosines, `TRUNCATE chunks, documents` fixture, `retrieve()` + `select_documents()` with settings overrides via the house settings-override pattern — check how that suite's siblings inject settings, e.g. `monkeypatch` on `get_settings` or `Settings(_env_file=None, …)`):
1. **THE OWNER SCENARIO (old-correct beats new-similar).** Question `"How did I configure the backup retention policy?"`. Doc A `backups/retention.md`, `created_at=2020-01-01`: the exact answer — chunk vector = the question vector's axis (cosine 1.0) + its exact tokens in the chunk text (top FTS rank). Doc B `backups/retention-draft.md`, `created_at=yesterday` (the test computes `now − 1d`): topically similar (shares `backup retention policy` tokens — a solid FTS hit at rank 2-3) but a weaker vector (half-parallel axis → cosine ~0.707) and its text says the policy is "under review, no decision yet" (no answer). Assert with DEFAULTS: `select_documents(...)[0].path == "backups/retention.md"` AND the fused (pre-boost, computed via `fuse` directly in the test for the margin) gap A−B ≥ 3× the zero-age boost (record the measured margin in the test docstring — the "comfortable margin" requirement). Assert AGAIN with `recency_boost=0` (settings override): A still first (no-regression pin — relevance alone ordered them).
2. **The boost is real (near-tie flips toward newer).** Docs C (2019) and D (yesterday) with IDENTICAL chunk text + IDENTICAL vectors (a true tie — same fused score, cosine, FTS rank; the deterministic sort key would otherwise order by path, and path is set so the OLDER sorts first lexicographically, e.g. `c-older.md` < `d-newer.md`). With defaults: D (newer) is first. With `weight=0`: C (older) is first (proving the boost — not drift — is the differentiator).
3. **Decay end-to-end:** the same C/D pair with D aged to `half_life + 365` days (≈ `weight*e^{-3}` ≈ 0.00005, below the tie gap 0) → C first again (the boost faded — recency is an age signal, not a binary).
4. **The gate is untouched:** the owner-scenario question's `max cosine` (the A8 input) equals the pre-boost run's (assert on the retrieved chunks' `cosine` values — the boost never touches them).
- If test 1's measured margin under the DEFAULTS is thin (< 3× the boost) or the scenario flips, tune the DEFAULTS (0.001/365 are the starting point — the owner re-tunes live via the env) until old-correct wins comfortably, and record the final margin in the docstring. The test asserts the SEMANTICS (A first, margin ≥ 3× boost), never the exact floats.
5. Run `uv run pytest tests/unit/test_retriever_recency.py tests/integration/test_recency_boost.py -q` (DB up) — green; then `uv run pytest tests/integration/test_name_hit_lexical.py tests/integration/test_chat_api.py -q` (the retriever's existing contract suites stay green — the boost is ON by default in them, so any drift surfaces here).
## Testing & Quality
- Unit: the decay/weight/clamp/tie/kill-switch pins (pure function).
- Integration: the fine-line battery on real Postgres with exact axis cosines — the owner's scenario + the near-tie flip + the decay + the cosine-gate-untouched pin.
- Coverage: **>90%** on `app/` (config validator + retriever branches covered — the validate.sh gate).
## Completion Criteria
- [ ] `Settings.recency_boost` (default 0.001, 0 = byte-identical off, negative fails startup) + `recency_half_life_days` (default 365, `<= 0` fails startup); `.env.example` documents both
- [ ] `apply_recency_boost` is pure (defaults from settings, `now` injectable, inputs unmutated, the existing 4-key sort) and is applied in `retrieve()` after `fuse()` and ONLY there — chat API + `eval_retrieval` inherit it; `eval_retrieval` prints the date + effective score
- [ ] The owner's scenario is pinned: older-correct beats newer-similar under defaults (margin ≥ 3× the zero-age boost, recorded) AND with the boost off; the near-tie flips toward the newer with the boost on and back without; the decay pin holds; the A8 cosine input is untouched
- [ ] `tests/unit/test_retriever_recency.py` + `tests/integration/test_recency_boost.py` + the two existing retriever-contract suites green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,38 @@
# Task 08 — UI: `Created` file column, `Updated` folder column, viewer `Created` badge (D8)
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "The UI must also show a date for every document at the top of that document when the user clicks it"; "I would also like to see a last updated dates/timestamps on folders before the description column but after the documents column in the UI"; "For files, include a date/timestamp before the 'indexed' column in the UI."
## Objective
Render the dates the task-05 APIs serve: the file table's `Created` column (before `Indexed`), the folder/source table's `Updated` column (after `Documents`, before `Description`), and the clicked document's `Created` badge in the shared viewer core's top meta row (modal + full page). The date EDITOR is task 09 — this task ships display only.
## Work
1. `frontend/index.html` — the two header rows (RAG view):
- File table (L485-492): insert `<th scope="col">Created</th>` between `<th scope="col">Chunks</th>` and `<th scope="col">Indexed</th>`.
- Folder table (L470-478): insert `<th scope="col">Updated</th>` between `<th scope="col">Documents</th>` and `<th scope="col">Description</th>`.
(No other shell markup — the rows are built by JS; a brief phase-106 comment above each inserted `<th>` in the house style.)
2. `frontend/assets/sources.js`:
- `makeRow` (L1375-1410): the cell loop (L1401) becomes `for (const value of [d.title, String(d.chunks), fmtDate(d.created_at), fmtDate(d.indexed_at)])` — the `Created` cell lands BEFORE `Indexed` (D8 verbatim). The loop's plain-`td` shape can't carry per-cell titles, so the date cells get one refinement: build the `Created` cell explicitly (a `td` with `textContent = fmtDate(d.created_at)` AND `title = d.created_at` — the ISO hover/precision value, the path-cell `title` idiom) between the `chunks` and `Indexed` cells (the E2E asserts on the locale-stable `title`, not on `toLocaleString` output). The row object fed from tree file nodes (L1341-1350) gains `created_at: f.created_at` (task 05's tree shape — the flat `GET /api/docs` path, if `makeRow` is still fed from it anywhere, carries `created_at` too — grep `makeRow(` call sites and extend every one).
- `makeSourceRow` (L1217-1235) + `makeFolderRow` (L1237-1261): between the count `td` and the description cell, one new `td` — `const updatedTd = document.createElement("td"); updatedTd.textContent = s.updated_at ? fmtDate(s.updated_at) : "–";` (the `statLast` null idiom, L1294 — `None` for a 0-document source, D9). `title` attribute = the ISO value (hover precision on the ellipsized cell, the `makeRow` path-cell idiom).
- `renderLevel`/`treeStats` — UNCHANGED (the stat cards keep their `indexed_at` "last indexed" semantics — the owner asked for the column, not the cards).
3. `frontend/assets/document.js` — `renderDocument` (L118-176, the ONE shared core — the modal AND `/document.html` render through it): the `.doc-meta` badge row (L123-129) gains the badge BEFORE the `Indexed` one:
```js
metaBadge("doc-created", `Created ${fmtDate(doc.created_at)}`),
metaBadge("doc-indexed", `Indexed ${fmtDate(doc.indexed_at)}`),
```
(the date at the top of a clicked document, D8). `doc-created` is the NEW class — the badge's `title` attribute carries the full ISO timestamp (the `titleEl` ellipsis-precision idiom, L122-124). `document-modal.js` needs no change (it calls the shared core with its own `metaEl` — the module docstring's contract is unchanged; verify the modal's meta element exists — it does: `metaEl` L48).
4. `frontend/assets/styles.css` — next to the existing `.doc-indexed` rule (grep for it): `.doc-created` — same badge family (the `doc-indexed` rule copied, provenance comment citing phase 106 D8); the new table cells need no new CSS beyond what `.docs-table` already styles (verify the column count change doesn't break the table's responsive rules — the `#docs-table`/`.kb-folders-table` grid/width rules: if a rule hard-codes the column count, extend it). WCAG: the date text reuses the table ink (≥4.5:1 by construction — record the verified pair in the comment, house style); the badge contrast mirrors `doc-indexed`'s recorded ratio.
5. `tests/unit/test_sources_dates.py` (NEW — the house read-the-assets-as-text pattern, `tests/unit/test_source_ignore_paths.py`'s sibling style):
- `frontend/index.html` — both header rows' cell ORDER pinned (the `<th>` sequence strings: `Source | Path | Title | Chunks | Created | Indexed` and `Folder | Documents | Updated | Description`);
- `frontend/assets/sources.js` — the `makeRow` value-list order (`created_at` before `indexed_at`), the `updatedTd` null→`"–"` branch present in BOTH row builders, the file-row object carries `created_at`;
- `frontend/assets/document.js` — the badge order in the meta row (`doc-created` before `doc-indexed`), the `Created ` label + `fmtDate(doc.created_at)` template, the single-source cross-file check that the `doc-created` class exists in `styles.css`;
- `frontend/assets/styles.css` — the `.doc-created` rule present with a provenance comment.
6. Run `uv run pytest tests/unit/test_sources_dates.py -q` + the existing sources/JS unit suites — green.
## Testing & Quality
- Unit: the source-level wiring pins above (order, null handling, cross-file class check).
- Coverage: **>90%** on `app/` (no `app/` code this task — the gate is the full-suite one, held by the other tasks; the JS pins are the house frontend-test pattern).
## Completion Criteria
- [ ] The file table shows `Created` between `Chunks` and `Indexed` (formatted like the `Indexed` cell — `fmtDate`); the folder/source table shows `Updated` between `Documents` and `Description` (subtree max from the tree API, `–` when null)
- [ ] The clicked document's top meta row carries `Created <date>` BEFORE `Indexed` in BOTH the modal and `/document.html` (one shared core — no per-surface copy)
- [ ] `tests/unit/test_sources_dates.py` pins the orders + null branch + cross-file class and passes; existing JS unit suites stay green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,31 @@
# Task 09 — The admin date editor in the viewer (D7, the phase-57 idiom)
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "This timestamp should be editable so users can correct for errors."
## Objective
An admin-only inline date editor in the shared viewer core (modal + page, where task 08 put the badge): set a corrected date (→ `PATCH /api/documents/date`, `created_at_manual` locks it against syncs, D1) or revert to sync-managed (the CLEAR — flag drops, the date stands until the next sync refreshes). Non-admins see the byte-identical task-08 badge row — no button, no wiring, no network call (the phase-57 split, owner-locked).
## Work
1. `frontend/assets/document.js` (extend the task-08 core — the phase-57 `wireSummaryEdit` idiom verbatim in structure; read it first, L180+):
- After the `Created` badge (task 08's insertion point), the admin gate: `void docAdminReady().then((admin) => { if (admin) wireDateEdit(metaEl, doc); })` — the module-cached `docAdminReady()` promise (the phase-57/79 single-request-per-page convention — no extra fetch). Anonymous / token holders / a failed whoami: the badge row stays exactly what task 08 built (byte-for-byte).
- `wireDateEdit(metaEl, doc)`:
- **The button** — a text button `Edit date` (`.doc-date-edit`, the `.kb-summary-edit`/`.doc-summary-edit` button family — reuse the existing edit-button class if its styling fits, else a sibling class in `styles.css` with the provenance comment), inserted after the Created badge, `aria-label` = `Edit creation date: ${doc.source}/${doc.path}` (setAttribute — never innerHTML).
- **The editor** (opened on click — the badge row swaps in-place, the summary editor's swap pattern): the `Edit date` button is replaced by a container holding a native `<input type="date">` (value = `doc.created_at`'s UTC date part — `new Date(doc.created_at).toISOString().slice(0, 10)`; `aria-label="Document creation date"`) + `Save` / `Cancel` text buttons + a `role="status"` live line (the phase-57 status-line shape). `Save` with an empty input → the clear path (see below) is NOT implicit — an empty `type=date` input is disabled-look only: disable Save when empty (an explicit `Revert` link below handles the clear — no accidental wipes).
- **Revert affordance** (the D7 CLEAR, the phase-57 "clear = explicit" contrast): a `Revert to sync` text link/button in the editor container (the muted marker style) → sends `{source, path, date: null}`.
- **§7.4 never-stale lifecycle** (the phase-57/89 last-announce order): on Save/Revert — the editor controls disable IMMEDIATELY (no double-submit); `PATCH /api/documents/date` with `{source: doc.source, path: doc.path, date: <input.value>}` (or `date: null` for the revert); on 200 → the badge's text re-renders from the RESPONSE's `created_at` (`Created ${fmtDate(res.created_at)}` — the UI shows exactly what the server stored, never the input's optimistic value), the status line announces `Date saved for <source>/<path>.` / `Reverted to sync-managed date.` (the `role=status` live line + the shared announcer where the page has one — follow whatever `wireSummaryEdit` uses), the editor collapses back to the badge + `Edit date` button; on non-2xx or network failure → the server `detail` (or the canned `Couldn't save the date — try again.` on a plain network error) into a `role="alert"` line (the phase-89 error-line idiom — the nearest existing error surface in this file), the input reverts to the stored date, the controls re-enable — the UI never claims a state the server didn't save.
- **No other surface:** the editor lives in `renderDocument`'s shared core only — the modal and the page both get it (both already call the core with `docAdminReady` available — verify `docAdminReady` is reachable in the modal's bundle context; `document-modal.js` imports `renderDocument` from this module, so the wiring rides along with the module — no second copy).
2. `frontend/assets/styles.css` — the editor's controls (the `.kb-summary-edit` / summary-editor rule family as the model, near it): `.doc-date-edit` (the button), the date input (sized, the global `:focus-visible` ring applies — no per-control rule, the phase-105 checkbox idiom), `:disabled` (opacity + `cursor: wait` — the `.git-source-remove:disabled` idiom), `role="alert"` line (the `.git-source-error` styling reuse or a local sibling), provenance comments citing phase 106 D7; contrast ≥4.5:1 verified + recorded in comments (house style).
3. `tests/unit/test_date_editor.py` (NEW — the read-the-assets-as-text pattern, task 08's suite extended or a sibling):
- `frontend/assets/document.js`: `wireDateEdit` exists and is called ONLY behind `docAdminReady()`'s `if (admin)` (a source-level pin — the string sequence `docAdminReady().then` … `wireDateEdit`); the PATCH URL is `/api/documents/date` (the single-source cross-file check — the endpoint string appears exactly once in the JS, matching `app/api/docs.py`'s route); the response-driven badge re-render (the `res.created_at` reference, NOT `input.value`); the revert link sends `date: null`; the disable-on-submit + revert-on-failure branches exist (the error-line `role="alert"` + the re-enable); the aria labels (`Edit creation date: `, `Document creation date`);
- `frontend/assets/styles.css`: the editor classes present with provenance comments.
4. Run `uv run pytest tests/unit/test_date_editor.py -q` + task 08's suite + the phase-57 suite's unit pins — green.
## Testing & Quality
- Unit: the source-level wiring pins above (gate, endpoint, response-driven render, §7.4 branches, a11y strings).
- Coverage: **>90%** on `app/` (no `app/` code this task — the endpoint's coverage landed in task 05; the gate is the full-suite one).
## Completion Criteria
- [ ] An admin sees an `Edit date` affordance next to the Created badge in BOTH the modal and the page (shared core — one implementation); the editor sets the date (input → `PATCH /api/documents/date` → the badge re-renders from the RESPONSE) and offers `Revert to sync` (→ `date: null`, the manual flag drops)
- [ ] The §7.4 lifecycle holds: controls disable on submit, a failure reverts the input to the stored value + announces in a `role="alert"` line + re-enables; the happy path announces through the live line after the badge update
- [ ] A non-admin / token holder / failed-whoami viewer is byte-for-byte the task-08 badge row (no button, no wiring, no extra request — the phase-57 split)
- [ ] `tests/unit/test_date_editor.py` passes; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,61 @@
# Task 10 — E2E: `tests/e2e/test_document_dates.py` (isolation) + regressions + full gate + commit
**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — the whole item, proven end to end (dates sourced → stored → shown → editable → retrieval-weighted).
## Objective
One dedicated Playwright suite proving the owner's item through the REAL page + REAL API + REAL importer (mock LLM — deterministic token-overlap embeddings, so the cosine/retrieval behavior is production-shaped; no git, no network — a local fixture dir with `os.utime`'d mtimes, built under `tmp_path_factory`, NEVER the shared `tests/fixtures/docs` whose 13-file counts are pinned by other suites). Then the phase's full gate and the single atomic commit.
## Work
1. `tests/e2e/test_document_dates.py` (NEW) — module scaffolding from `tests/e2e/test_retrieval_quality.py` (`_import_fixtures`'s Settings-with-mock-port pattern, `_run_in_thread`, `_reset_db`, `e2e.auth_helpers.login`, the `app_url`/`mock_llm`/`db_ready` fixtures, the `source-chip` assertions) with the dedicated fixture tree (the module builds it ONCE per module under `tmp_path_factory`, `os.utime`'d — a `mkdocs + utime` helper at the top):
```
backups/retention.md utime 2020-01-01 03:04:06Z — THE CORRECT answer:
"The backup retention policy is 30 days; snapshots
are pruned nightly…" (rich in the question's tokens)
backups/retention-draft.md utime = now (default mtime) — the SIMILAR-but-wrong
doc: shares "backup retention policy" wording,
concludes "under review, no decision yet"
legacy/old-doc.md utime 2019-06-15 — single-doc folder (a clean
folder-`Updated` max: the 2019 date alone)
future/forward.md utime 2999-01-01 — the future-date case (→ today, D3)
```
Contract under test (docstring) — six tests, one per bullet:
1. **`test_dates_landed_on_import`** — import the tree (real importer, mock LLM, in a thread): admin-cookie `GET /api/docs` — `retention.md`'s `created_at` ISO date-part = `2020-01-01`, `old-doc.md`'s = `2019-06-15`, `forward.md`'s = TODAY (the D3 future-fold, the test computes today in UTC); `GET /api/docs/tree` — file nodes carry the same dates; the `legacy` folder node's `updated_at` = the 2019 date (single-doc max), the source node's `updated_at` = the max of all (the `now`/today side); `indexed_at` on every row is UNCHANGED in meaning (still ≈ import time, after the created dates).
2. **`test_file_and_folder_columns`** — real form login → the RAG view → the file table header order `… Chunks · Created · Indexed` (the `<th>` sequence) and the drilled-in rows: `retention.md`'s Created cell `title` attribute = the ISO string (locale-stable — task 08's idiom) and its text contains `2020`; the folder table header order `Folder · Documents · Updated · Description`; at the top level the source row's `Updated` cell is non-empty; drilled into `legacy`'s parent, the `legacy` folder row's `Updated` cell `title` carries `2019-06-15`.
3. **`test_viewer_shows_date_at_top`** — click `retention.md`'s row link (the real click — the same-page modal, phase 26): the modal's top meta row contains a badge with text starting `Created` whose `title` attribute = the 2020 ISO, and it DOM-precedes the `Indexed` badge (the date at the top of the clicked document, D8); the badge row also still shows `Indexed` + the source/format badges (no regression).
4. **`test_old_correct_beats_new_similar`** — THE OWNER SCENARIO end to end: ask `How did I configure the backup retention policy?` → the grounded answer arrives (mock marker, no deflection), the FIRST `.source-chip` = `backups/retention.md` (the OLDER correct doc beats the newer similar one — the real retriever + the default recency boost over the mock's token-overlap embeddings); `query_log` — one row, `deflected is False`, `sources` contains `backups/retention.md`. (If the fixture wording doesn't produce the order under the DEFAULTS — the token-overlap geometry differs from task 07's axis vectors — adjust the FIXTURE TEXT until the old-correct doc is the clear top-1 (more exact question-phrase overlap in `retention.md`, the draft sharing only loose keywords), and record the working wording + the reason in the test docstring. Do NOT change the boost defaults here — task 07 owns them.)
5. **`test_date_edit_and_sync_preserves`** — the admin-only edit through the REAL UI: open `legacy/old-doc.md` in the modal → the `Edit date` button is present (admin session) → click → set the date input to `2021-05-05` → Save → the badge re-renders from the response (title = a 2021 ISO) → admin-cookie `GET /api/docs` confirms `2021-05-05`. Re-run the import (in a thread, same tree — the mtimes are untouched): `old-doc.md` keeps `2021-05-05` (the manual flag, D1) while `retention.md` still reads 2020 (refreshed, not stale) and `forward.md` still reads today. Then the REVERT: open the editor again → `Revert to sync` → re-run the import → `old-doc.md`'s date is refreshed back to `2019-06-15` (the flag dropped — sync manages it again).
6. **`test_anonymous_gate_and_editor_a11y`** — anonymous (no login): the RAG view shows the sign-in gate (no tables), a raw `PATCH /api/documents/date` with a date payload → 403; signed in (admin): the `Edit date` button's accessible name contains `legacy/old-doc.md` (the aria-label), the editor's date input has the `Document creation date` accessible name, is keyboard-reachable (Tab from the button), the status line is `role="status"` (and the error path's line `role="alert"` exists in the DOM — the phase-57/89 surfaces); the badge text pairs (text + formatting, never color alone — the monochrome-theme contract, B5).
2. **Regressions** — each in isolation (DB up), all green (task 06 changed pinned formats — the `ls` line, the `read` result, the `<document>` block; task 04's unchanged-path date refresh must not move any content count):
- `uv run pytest tests/e2e/test_retrieval_quality.py -v --no-cov` (the fixture-import + ranking E2E — the mock-regex canary)
- `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov` (the `<document>` block)
- `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov`
- `uv run pytest tests/e2e/test_ls_tree_drilldown.py -v --no-cov` (the `ls` line format)
- `uv run pytest tests/e2e/test_read_truncation_cap.py -v --no-cov` (the `read` result shape)
- `uv run pytest tests/e2e/test_kb_tree.py -v --no-cov` + `uv run pytest tests/e2e/test_kb_tree_nav.py -v --no-cov` (the tree shape + the tables)
- `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` + `uv run pytest tests/e2e/test_edit_summaries.py -v --no-cov` (the viewer core + the sibling admin-edit idiom)
- `uv run pytest tests/e2e/test_import_documents.py -v --no-cov` + `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` (importer counts + the sync detail)
- `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` (phase 105 — the importer map idiom)
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov`
(Where a suite pins a pre-phase format EXACTLY — an `ls` line without the date field, a `read` result without the `date:` line — update the pin in that test file to the phase-106 shape (mechanical, the new field is deterministic). A suite that breaks for any OTHER reason is a regression — fix the product code in its owning task's files, keep this phase's contract as written.)
3. **Full gate** (AGENTS.md rule 9 — non-negotiable):
- `uv run pytest` (unit + integration) green
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL **>90%**
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` green in isolation
- `uv run ruff check . && uv run pyright` clean
4. **Commit** — one atomic Conventional-Commits commit, `--no-gpg-sign` (AGENTS.md rule 8), per the phase overview's Commit block:
```bash
git add app/ alembic/versions/0020_documents_created_at.py scripts/ frontend/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "feat(dates): document dates end to end — sourced at sync, shown in UI, editable, recency-weighted in retrieval"
```
(If the pipeline commits per task instead, fold everything into this phase's final commit and move the phase dir to `.agents/phases/complete/106_document_dates/` as the last action, per the pipeline gate.)
## Testing & Quality
- This task IS the phase's E2E + gate + commit; no new application code (the E2E may reveal a one-line fix in an earlier task's work — fix it IN that task's file, keep the suite's contract as written).
- Coverage: **>90%** on `app/` (the validate.sh gate — enforced here, not assumed).
## Completion Criteria
- [ ] `tests/e2e/test_document_dates.py` exists, maps 1:1 to the six contract bullets, and passes in isolation (`--no-cov`, DB up)
- [ ] The owner's scenario holds end to end: the older document that answers the question is the FIRST cited source over the newer similar one (defaults, real retriever, mock embeddings)
- [ ] The date edit round-trips through the real UI + API and SURVIVES a re-import (manual flag), and `Revert to sync` hands management back to the next import
- [ ] All ten regression suites pass in isolation (the format-pin updates are mechanical and live in the test files)
- [ ] The full gate is green: unit + integration, TOTAL coverage >90%, ruff + pyright clean
- [ ] One `--no-gpg-sign` commit contains the whole phase (app + alembic 0020 + scripts + frontend + tests + `.env.example` + the phase files)
@@ -0,0 +1,58 @@
# Phase 107 — True per-file document dates for URL git sources: full-history checkouts (revisits phase 106 D10)
**Source:** Owner bug report 2026-09-16 (chat): after the phase-106 re-sync of brain.reeseapps.com, the git source `https://gitea.reeseapps.com/services/homelab.git` shows `active/container_bifrost` created **8/16/2026** — "completely wrong, container_bifrost is > 6 months old at this point"; "This is a git source, so it should be easy to tell when a document was last edit[ed]".
**Story:** n/a (owner bug report, phase-106 follow-up — the phase's E2E suite proves the fix end to end).
**Context / verified root cause (2026-09-16, scratch + dev checkouts of the live homelab repo):** `clone_or_pull` (`scripts/git_sync.py` L63) clones URL-transport sources with `--depth 1` — phase 28's strategy, which phase 106's **D10 explicitly locked** ("No clone-strategy change… `--depth 1` stays"), predicting the consequence: "URL git sources are shallow → every file carries the repo's TIP-commit date (uniform within the repo…); revisit only if the owner later wants intra-repo recency on URL sources." In a shallow checkout git cannot see history past the shallow boundary (= the tip commit), so `file_commit_dates` (phase 106, `git log --name-only --format=@@%cI` first-sighting-wins) returns the **tip commit's date for EVERY file in the repo** — that uniform tip date is what the live site now displays for `container_bifrost` (the live clone's tip; the dev clone of the same repo shows 2026-09-07 for every file, `git rev-parse --is-shallow-repository` → `true`, exactly ONE `@@` line in the date walk). After `git fetch --unshallow` (438 commits visible) the TRUE last-commit date of `active/container_bifrost/bifrost.md` is **2026-05-05T06:26:40-04:00** — months older than the displayed date. Local-PATH git sources were never affected (git ignores `--depth` for local clones → full history → true dates — which is why the bug only surfaced on the URL source). Everything DOWNSTREAM of the checkout (the importer's `doc_dates_by_root` map + first-sighting-wins walk, D3 normalization, D4 sync semantics, the API/LLM/UI date surfaces, the recency boost) is CORRECT as built — the wrong value is produced at the checkout, so the fix is confined to `clone_or_pull` plus the docstrings/tests that enshrine the shallow assumption.
**⚠ LOCKED-DECISION REVISIT (AGENTS.md rule 3 — flagged, not silent):** this phase REVISITS phase 106's locked decision **D10** (shallow clone strategy). The revisit is owner-authorized: D10 itself names this exact trigger ("revisit only if the owner later wants intra-repo recency on URL sources") and the owner's 2026-09-16 report is precisely that request. D10's tip-date expectation is hereby SUPERSEDED; every other phase-106 decision (D1–D9 — storage/flag, provenance walk, normalization, sync semantics, LLM surfaces, recency boost, UI columns, derived folder dates) stands unchanged and simply receives true dates. No `PLAN.md` change is involved (D10 is a phase-level decision, not a PLAN §2 anchor).
## Objective
Git-source checkouts keep FULL history — a fresh `clone_or_pull` clones without `--depth 1`, and any EXISTING shallow checkout (including the live + dev homelab checkouts) self-heals via `git fetch --unshallow` on its next sync — so `file_commit_dates` yields the TRUE per-file last-commit date for every git source (local and URL). A `file://` E2E proves it end to end: an old file shows its old commit date and a tip-touched file shows the tip date, in the API, the Sources tables, and the document viewer — instead of the uniform tip date the bug produced.
## Dependencies
- `106_document_dates` (complete) — the entire date pipeline this phase corrects AT THE SOURCE: `file_commit_dates` (task 03), the importer's `doc_dates_by_root` map + D4 refresh semantics (task 04), D3 normalization, and the API/LLM/UI surfaces. Only D10's shallow-clone assumption is revised; its suites are the regression gate.
- `28_git_based_sources` (complete) — `scripts/git_sync.py` (the ONLY git-invocation site, A11 — `run_git` contract, `GitSyncError` semantics) and `clone_or_pull` itself. `app/core/docs_push.py`'s own `--depth 1/100` fetches operate on the DOCS repo (phase 18/59) and are NOT touched.
- `32_admin_sync_button` (complete) — `tests/e2e/test_sync_button.py`'s app-server idiom: per-module env with `BOR_GIT_SOURCES=file://<fixture repo>` + its own `BOR_SOURCES_DIR`, a real fixture repo built via `git` subprocesses with controlled commit dates, the real in-app sync. The new E2E copies it.
- `99_kb_tree_table_and_back_nav` (complete) — the Sources RAG-view tables + `GET /api/docs/tree` the E2E asserts on (the `Created`/`Updated` columns phase 106 D8/D9 added).
## Design (shared by all tasks — the executor reads this, not the chat)
- **The fix — `scripts/git_sync.py::clone_or_pull` (task 01; the single clone/pull entry point BOTH sync entry points call — `app/api/sync.py` L296 and `scripts/import_docs.py` L237 — so one change fixes the UI Sync button and the CLI at once):**
1. **Fresh checkout** (dest absent or without `.git`): `run_git(["git", "clone", url, str(dest)], cwd=dest.parent)` — the `--depth 1` flags are REMOVED (full history on the first clone for every transport: https/ssh/`file://`/local-path). One-time cost only — subsequent syncs are incremental; the KB repos are small homelab-docs repos. Deliberately NO env knob to restore shallow: shallow == the bug (D12).
2. **Existing checkout** (dest has `.git`): probe first — `run_git(["git", "rev-parse", "--is-shallow-repository"], cwd=dest)`; stdout.strip() == `"true"` → `run_git(["git", "fetch", "--unshallow"], cwd=dest)` (the ONE-TIME self-heal for checkouts that are already shallow — live + dev homelab included — so the fix reaches deployed sites WITHOUT a re-clone: the next sync after deploy unshallows, pulls, and re-dates), then `run_git(["git", "pull", "--ff-only"], cwd=dest)` as today. Probe `"false"` → straight to `git pull --ff-only` (the common steady-state path — one extra cheap probe per sync, no network fetch).
3. **Fail loud (D12):** ANY of the probe/unshallow/pull steps raising `GitSyncError` propagates exactly like today's clone/pull failures — the sync aborts with the named repo + reason (the `app/api/sync.py` 502 surface, the CLI traceback). NEVER a silent fallback: continuing a failed unshallow would silently re-serve tip dates (the bug), and falling back to mtimes would be worse. A broken checkout failing loudly is the phase-28 contract.
- Every git invocation still goes through `run_git` (A11 — the module docstring's git-inventory sentence lists the three new/changed commands).
- `file_commit_dates` is UNCHANGED in code — on a full-history checkout its existing newest-first, first-sighting-wins walk already returns the true per-file last commit (verified 2026-09-16: 438 commits, `bifrost.md` → 2026-05-05). ONLY its docstring + the module docstring lose the "shallow URL → uniform tip date" narrative and state the new guarantee: every `clone_or_pull` checkout is full-history → TRUE per-file dates for ALL git sources.
- **Downstream — deliberately untouched:** the importer's `doc_dates_by_root` plumbing, D1/D3/D4 (storage, normalization, sync-refresh semantics — including "a date may go OLDER", which is exactly how the wrong tip dates self-correct on the first post-fix sync: every git document's stored date is refreshed to its true commit date, `dates_updated` counts them, `sources_meta` does NOT bump — a date-only refresh is `unchanged` per D4), the admin date API, the LLM surfaces, the UI columns, the recency boost. No migration, no data fix — the sync IS the fix. `created_at_manual = true` rows keep the owner's corrections (D1) — correct as designed.
- **Stale narratives to correct (same task as the code — a comment that lies is a bug):** `scripts/git_sync.py` module docstring (the L18-28 "Per-file last-commit dates" block: the URL-shallow bullet is false after this phase), `clone_or_pull`'s docstring (L48-51 "shallow, depth 1" + the behavior list), `file_commit_dates`'s docstring (L104-111), `scripts/import_docs.py` docstring L19 ("first run, shallow ``--depth 1``" → "first run, full history"), `app/api/sync.py` per-row comment (L298-303, "shallow URL checkouts → the tip date, D10" → true per-file dates for all git sources).
## Tasks
1. `01_full_history_checkouts.md` — `clone_or_pull`: no-`--depth` clone + the shallow-probe/`fetch --unshallow` self-heal + fail-loud; the unit argv pins, the real-git integration pins (true per-file dates over `file://` + the existing-shallow self-heal), and the five stale docstring/comment sites.
2. `02_e2e_git_source_dates.md` — dedicated Playwright suite `tests/e2e/test_git_source_dates.py` (isolation): a two-commit `file://` fixture repo, a real in-app sync, true per-file dates asserted in the API + the Sources tables + the viewer badge.
3. `03_gates_and_commit.md` — full gate (unit + integration, coverage >90%, the new E2E + the three regression suites in isolation, ruff + pyright), one atomic `--no-gpg-sign` commit, phase dir → `complete/`.
## Testing & Quality
- Unit — `tests/unit/test_git_sync.py` (task 01): fresh-clone argv re-pinned to `["git", "clone", url, str(dest)]` (NO `--depth`); the existing-checkout path asserts probe-then-pull; NEW: probe `"true"` → `fetch --unshallow` THEN `pull --ff-only` (argv + order); NEW: `fetch --unshallow` failure → `GitSyncError` propagates (D12 fail-loud); the `git clone --depth 1 … failed (exit 128)` match-string updated to the new argv.
- Integration — `tests/integration/test_git_file_dates.py` (task 01, real `git`, DB-free, the git-availability skip pattern): the module docstring's D10 tip-date expectation is REPLACED with the full-history guarantee; `test_shallow_file_clone_yields_tip_date_for_every_file` (which pinned THE BUG) is REPLACED by `test_url_clone_yields_true_per_file_dates` — the same two-commit recipe (`a.md`/`docs/deep.md` committed 2020-01-02, `b.md` touched again at the 2024-06-15 tip), but `clone_or_pull(f"file://{scratch_repo}", dest)` (a URL transport, through the real function) must yield `{"a.md": DATE_A, "b.md": DATE_B, "docs/deep.md": DATE_A}` — the regression pin: pre-fix this returned `DATE_B` for all three; NEW `test_existing_shallow_checkout_self_heals` — the harness builds a `--depth 1` `file://` clone directly (simulating the deployed checkouts: dates uniform tip, `is-shallow` true), then `clone_or_pull(url, dest)` → no longer shallow + true per-file dates; the local-clone true-date test and every fail-soft test stay green unchanged.
- E2E (mandatory, A16) — `tests/e2e/test_git_source_dates.py` (task 02): `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` with the DB up.
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing` — the validate.sh gate; `scripts/` is outside the `app/` denominator but fully pinned by the suites above).
## Completion Criteria
- [ ] A fresh `clone_or_pull` over a URL transport yields a NON-shallow checkout (unit argv pin: no `--depth`; integration: `git rev-parse --is-shallow-repository` on the `file://` clone → `false`) with TRUE per-file last-commit dates (2020 file stays 2020, tip-touched file gets the tip date — NOT uniform).
- [ ] An EXISTING shallow checkout (made with `--depth 1`, like every deployed one) unshallows on its next `clone_or_pull` (integration pin: probe → `fetch --unshallow` → pull; dates true afterwards) and a non-shallow checkout takes the plain pull path (unit argv pins); a failed unshallow aborts the sync with `GitSyncError` (fail loud, D12).
- [ ] `tests/integration/test_git_file_dates.py` green with the D10 tip-date test replaced by the true-date regression pin; `tests/unit/test_git_sync.py` green with the updated argv pins.
- [ ] E2E green in isolation: after a real in-app sync of a two-commit `file://` fixture, `GET /api/docs` carries `created_at[:10]` = the OLD commit date for the old file and the TIP date for the new file (the two DIFFER — the bug made them identical); the Sources file table's `Created` column renders the two different years; the old document's viewer `Created` badge carries the old date (ISO `title`); the folder `Updated` columns are the subtree maxes (old folder 2020, new folder 2024).
- [ ] The phase-106 regression suites green in isolation: `uv run pytest tests/e2e/test_document_dates.py -v --no-cov`, `tests/e2e/test_sync_button.py -v --no-cov`, `tests/e2e/test_git_sources_admin.py -v --no-cov`; `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] The five stale shallow/tip-date narratives (git_sync module + two function docstrings, import_docs docstring, sync.py comment) now describe full-history checkouts — no remaining claim that URL sources carry tip dates.
- [ ] One `--no-gpg-sign` Conventional Commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **D11 — Full-history git checkouts (SUPersedes phase 106 D10; owner-authorized 2026-09-16 — D10's own revisit clause).** Fresh `clone_or_pull` checkouts clone WITHOUT `--depth 1`; existing checkouts are probed with `git rev-parse --is-shallow-repository` and, while shallow, `git fetch --unshallow` before the usual `git pull --ff-only` (one-time self-heal of deployed checkouts, no re-clone). Consequence: `file_commit_dates` returns the TRUE per-file last-commit date for EVERY git source — local and URL — and the next sync after deploy refreshes every git document's stored `created_at` to its true commit date (D4's "may go older" makes this a plain date-only refresh: no `sources_meta` bump, manual corrections survive, `dates_updated` counts them).
- **D12 — Fail loud, never silently re-shallow (the house fail-loud rule applied to the self-heal).** A probe/unshallow/pull failure raises `GitSyncError` and aborts the sync exactly like any clone/pull failure; there is no env knob to restore shallow clones or to skip the self-heal — a shallow checkout would silently re-serve the uniform tip date, i.e. the bug this phase fixes.
## Commit
```bash
git add scripts/git_sync.py scripts/import_docs.py app/api/sync.py tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(git): full-history checkouts so URL sources get true per-file document dates"
```
@@ -0,0 +1,43 @@
# Task 01 — `clone_or_pull` full-history checkouts: no `--depth 1` + the shallow self-heal (D11/D12)
**Phase:** `107_git_full_history_dates` · **Source:** owner bug report 2026-09-16 — URL git sources show the repo TIP date for every document (`container_bifrost` "created 8/16/2026", months off) because phase 106 D10 locked `--depth 1` shallow URL clones.
## Objective
Make every `clone_or_pull` checkout carry FULL git history — fresh clones without `--depth 1`, existing shallow checkouts self-healing via `git fetch --unshallow` before the usual fast-forward — so `file_commit_dates` (unchanged) yields true per-file last-commit dates for all git sources; and correct every docstring/comment that enshrines the old shallow assumption.
## Work
1. `scripts/git_sync.py::clone_or_pull` — the behavior change (keep the signature, the `dest`/`.git` dispatch, and the `GitSyncError` semantics; EVERY invocation through `run_git`, A11):
- Fresh checkout (dest absent or without `.git`): `dest.parent.mkdir(parents=True, exist_ok=True)` then `run_git(["git", "clone", url, str(dest)], cwd=dest.parent)` — the `"--depth", "1"` arguments are REMOVED (full history for every transport).
- Existing checkout (`.git` present): probe `shallow = run_git(["git", "rev-parse", "--is-shallow-repository"], cwd=dest).strip() == "true"`; if `shallow` → `run_git(["git", "fetch", "--unshallow"], cwd=dest)` (the one-time self-heal — deployed checkouts, live + dev included, become full-history on their NEXT sync with no re-clone); then `run_git(["git", "pull", "--ff-only"], cwd=dest)` as today. Probe `"false"` → straight to the pull (steady state: one cheap local probe, no network).
- **D12 fail-loud:** do NOT catch `GitSyncError` — probe, unshallow, and pull failures all propagate exactly like today's clone/pull failures (the sync aborts with the named repo + git's stderr; never a silent fallback to tip dates or mtimes).
- Docstrings (the lies are the rest of the bug):
- module docstring: L3 "(shallow, depth 1)" → full-history clone; the L18-28 "Per-file last-commit dates (phase 106, D2/D10)" block → rewrite for phase 107: a `clone_or_pull` checkout is FULL-history for every transport (fresh: no `--depth`; existing: `--is-shallow-repository` probe + one-time `git fetch --unshallow` self-heal before the `--ff-only` pull) → `file_commit_dates` yields TRUE per-file last-commit dates for ALL git sources (local AND URL); cite the 2026-09-16 verification (shallow walk = one tip commit; after `--unshallow` the true per-file dates, e.g. `bifrost.md` 2026-05-05 not the 2026-09-07 tip).
- `clone_or_pull` docstring (L48-51 + the behavior bullets): "shallow, first run" → "full history, first run"; the bullet list gains the probe/unshallow step; drop "(shallow: the KB is re-imported incrementally anyway)".
- `file_commit_dates` docstring (L104-111): the "local FULL history / URL shallow → uniform TIP date (D10)" paragraph → "every `clone_or_pull` checkout is full history → true per-file last-commit dates for all git sources (phase 107 D11 — supersedes phase 106 D10)"; the fail-soft paragraph stays.
2. `scripts/import_docs.py` — docstring L19: "(first run, shallow ``--depth 1``) or fast-forwarded" → "(first run, full history — no ``--depth``; an existing shallow checkout is unshallowed first, phase 107) or fast-forwarded". No code change here (it already calls `clone_or_pull` and then `file_commit_dates` — the fix flows through).
3. `app/api/sync.py` — per-row comment (L298-303): replace "(local-path checkouts → true per-file dates, shallow URL checkouts → the tip date, D10)" with "(full-history checkouts → true per-file last-commit dates for every git source — phase 107 D11 supersedes phase 106 D10's shallow tip-date behavior)". No code change.
4. Tests (run `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py -v` — DB-free):
- `tests/unit/test_git_sync.py` (subprocess fully faked, argv pins — update the module docstring's first lines too: "``git clone`` (full history, fresh dest) or a shallow-probe + optional ``git fetch --unshallow`` + ``git pull --ff-only`` (existing checkout)"):
- `test_clone_or_pull_clones_when_dest_has_no_git_dir` → `call["argv"] == ["git", "clone", url, str(dest)]` (NO `--depth`) — same cwd-parent assertion.
- `test_clone_or_pull_creates_missing_parent_before_clone` → same argv re-pin.
- `test_clone_or_pull_pulls_when_git_dir_exists` → the fake `run_git` must now answer the probe (stdout `"false\n"`) before the pull: `calls[0]["argv"] == ["git", "rev-parse", "--is-shallow-repository"]`, `calls[1]["argv"] == ["git", "pull", "--ff-only"]`, both cwd=dest.
- NEW `test_clone_or_pull_unshallows_existing_shallow_checkout` → probe stdout `"true\n"` → `calls[0]` probe, `calls[1]["argv"] == ["git", "fetch", "--unshallow"]`, `calls[2]["argv"] == ["git", "pull", "--ff-only"]` (order pinned: unshallow BEFORE pull).
- NEW `test_clone_or_pull_unshallow_failure_propagates` → probe `"true"`, unshallow raises `GitSyncError("git fetch --unshallow failed (exit 128): fatal: …")` → `pytest.raises(GitSyncError, match="--unshallow")` (D12 fail-loud) and the pull is NEVER called.
- the error-path test's match string `r"git clone --depth 1 .* failed \(exit 128\): fatal: repository not found"` → `r"git clone .* failed \(exit 128\): fatal: repository not found"`.
- The fake-run helper must dispatch by argv (clone/probe/unshallow/pull) — extend it, keep the existing `_FakeProc` shape.
- `tests/integration/test_git_file_dates.py` (real `git`, DB-free, keep the `GIT` skip mark + the `scratch_repo` recipe + every fail-soft/parser test):
- module docstring: replace the "shallow URL-transport clone → TIP date for EVERY file (D10)" bullet with the phase-107 guarantee (a `clone_or_pull` `file://` checkout is full history → true per-file dates; an existing shallow checkout self-heals) and note the D10 supersession.
- REPLACE `test_shallow_file_clone_yields_tip_date_for_every_file` with `test_url_clone_yields_true_per_file_dates(scratch_repo, tmp_path)`: `clone_or_pull(f"file://{scratch_repo}", tmp_path / "url-clone")` → `file_commit_dates(dest) == {"a.md": DATE_A, "b.md": DATE_B, "docs/deep.md": DATE_A}` (the REGRESSION PIN — pre-fix this returned `DATE_B` for all three files) AND `run_git(["git", "rev-parse", "--is-shallow-repository"], cwd=dest).strip() == "false"` (the checkout is not shallow).
- NEW `test_existing_shallow_checkout_self_heals(scratch_repo, tmp_path)`: the harness builds a shallow checkout directly (`_git(tmp_path, "clone", "-q", "--depth", "1", f"file://{scratch_repo}", str(dest))` — simulating the pre-phase deployed checkouts) → pre-heal: `is-shallow` true and `file_commit_dates` uniform `DATE_B` for all three files; then `clone_or_pull(f"file://{scratch_repo}", dest)` → `is-shallow` false and true per-file dates (same dict as the previous test).
- `test_local_clone_yields_true_per_file_dates` stays green unchanged (local path was always full history).
## Testing & Quality
- Unit: argv + order pins for the fresh-clone (no `--depth`), probe-then-pull, probe→unshallow→pull, and fail-loud paths (the fake subprocess never sees the real git — deterministic).
- Integration: real `git` scratch repos — URL-transport true dates, the self-heal lifecycle, the pre-existing fail-soft battery.
- Coverage: **>90%** on `app/` (no `app/` code changes this task — the gate still passes; `scripts/` is pinned by the suites above).
## Completion Criteria
- [ ] `clone_or_pull` fresh-clone argv is `["git", "clone", url, str(dest)]` (unit-pinned); existing checkouts take probe → (unshallow iff shallow) → `pull --ff-only` (unit-pinned, order asserted); a failed unshallow raises `GitSyncError` before the pull runs (D12)
- [ ] `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py -v` green, including `test_url_clone_yields_true_per_file_dates` (per-file true dates over `file://` — the bug's regression pin) and `test_existing_shallow_checkout_self_heals`
- [ ] No remaining "shallow"/"tip date" narrative in `scripts/git_sync.py`, `scripts/import_docs.py`, or `app/api/sync.py` describing CURRENT behavior (phase-106 phase records in `complete/` are history — untouched)
- [ ] `uv run ruff check . && uv run pyright` clean; full `uv run pytest` green
@@ -0,0 +1,30 @@
# Task 02 — E2E: true per-file git dates end to end (`tests/e2e/test_git_source_dates.py`)
**Phase:** `107_git_full_history_dates` · **Source:** owner bug report 2026-09-16 — the UI must show the document's true last-commit date (a `file://` URL git source is the transport-true stand-in for the live https source; pre-fix it rendered the uniform tip date).
## Objective
Add the phase's dedicated Playwright suite (A16, run in isolation) that proves the fix end to end: a real `file://` git fixture with two commits of controlled dates, a real in-app admin sync, and assertions that the OLD file shows its OLD commit date and the NEW file shows the TIP date — in `GET /api/docs` / `GET /api/docs/tree` (deterministic ISO) and in the Sources tables + document viewer (locale-tolerant).
## Work
1. `tests/e2e/test_git_source_dates.py` (NEW — copy the app-server + fixture idiom from `tests/e2e/test_sync_button.py` verbatim in shape: module-scoped `app_server` with per-module env, `tmp_path_factory`-scoped real git fixture, per-test fresh-KB fixture, `e2e.auth_helpers.login`, the conftest `_wait_http`/`USE_REAL_LLM` imports; module docstring: story n/a — owner bug report 2026-09-16; the isolation command; what each test pins):
- **Fixture repo** (real `git` subprocesses, controlled `GIT_COMMITTER_DATE`/`GIT_AUTHOR_DATE` + fixed identity, the `sync_git_repo` helper shape — two commits, MID-YEAR dates so the browser's locale/TZ rendering of the year is stable in any timezone, the `test_document_dates.py` L455 lesson):
- commit one @ `2020-06-15T12:00:00Z` adds `old/old-note.md` (body: a sentence about an old, stable note).
- commit two (the tip) @ `2024-06-15T12:00:00Z` adds `recent/recent-note.md` (body: a sentence about a recent note).
- App env: `BOR_GIT_SOURCES=file://<repo>` + its own `BOR_SOURCES_DIR` (fresh dir, the `test_sync_button.py` L141-143 pattern); mock LLM; the session app (no git sources) is never started in this isolated run (no port clash).
- **Per-test sync helper:** admin login → the admin Sources page → click "Sync sources" → wait for the success detail (the `test_sync_button.py` lifecycle wait, generous timeout — real clone + mock-LLM embed).
- **Tests:**
1. `test_api_created_dates_are_true_per_file` — after a sync: `GET /api/docs` (admin, the page context's request client) — the row with `path == "old/old-note.md"` has `created_at[:10] == "2020-06-15"`, the row with `path == "recent/recent-note.md"` has `created_at[:10] == "2024-06-15"`, and the two values DIFFER (the regression assertion — the phase-106 bug made every URL-source file carry the tip date, i.e. both `2024-06-15`). `GET /api/docs/tree`: the file nodes carry those `created_at` verbatim; the `old` folder's `updated_at[:10] == "2020-06-15"`, the `recent` folder's `updated_at[:10] == "2024-06-15"`, the source node's `updated_at[:10] == "2024-06-15"` (subtree max, phase 106 D9 — now over TRUE dates).
2. `test_sources_tables_render_distinct_created_dates` — the RAG view's FILE table: the row whose path cell contains `old/old-note.md` has its `Created` cell (the column phase 106 D8 placed between `Chunks` and `Indexed`) matching a regex for the year `2020`; the `recent/recent-note.md` row's `Created` cell matches `2024`; the two cells' text differs. The FOLDER table: the `old` row's `Updated` cell (between `Documents` and `Description`) matches `2020`, the `recent` row's matches `2024`. (Year-regex assertions — `toLocaleString` rendering is locale/TZ-dependent; the year is stable for the mid-year fixture dates. `textContent` reads only — never set innerHTML.)
3. `test_viewer_created_badge_is_the_true_git_date` — open the old document (click its row/title, the phase-26 same-page modal): the viewer's `.doc-created` badge (phase 106, before the `Indexed` badge) has `title == "2020-06-15T12:00:00+00:00"` (the raw ISO — `metaBadge`'s title, deterministic under any locale/TZ, the house solution to the L455 rendering trap) and its visible text starts with `Created `; close the modal (Escape) — the viewer reverts cleanly.
4. `test_page_a11y_and_no_cdn_basics` — the standard light pass (AGENTS.md rules 5/6, the `test_git_sources_admin.py` a11y test shape): landmarks on the Sources view, the `Created`/`Updated` `<th>` cells present in both tables, ≥4.5:1-free text (no new color), 3px `:focus-visible` on a table row link, same-origin assets only (no external `src`/`href`).
- Fresh-KB fixture: truncate `documents`, `chunks`, `git_sources`, `sources_meta`, `kb_overview` (the `test_sync_button.py` `clean_kb` shape) so each test's sync counts are its own.
2. Run `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` (DB up: `podman compose up -d db`) — all four tests green in isolation.
## Testing & Quality
- E2E (the task IS the test): real `file://` URL-transport clone through `clone_or_pull` (task 01's full-history path) + the real importer + the real API + the real UI — the owner's scenario, transport-true.
- Coverage: the suite is `--no-cov`; it exercises `app/` (sync, docs API, tree builder) and `scripts/git_sync.py` for real — the `app/` >90% gate is unaffected.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` green in isolation (DB up) — all four tests
- [ ] The regression assertion holds: the two fixture documents' `created_at` values DIFFER (2020 vs 2024) in the API, the tables, and the viewer — the uniform-tip-date bug is provably gone for URL transports
- [ ] The suite leaves no other suite's fixtures touched (isolation: the session app never starts, own `BOR_SOURCES_DIR`, per-test DB reset)
@@ -0,0 +1,36 @@
# Task 03 — Full gate: suites, coverage, lint/types, regression E2Es, atomic commit
**Phase:** `107_git_full_history_dates` · **Source:** AGENTS.md rules 8/9 — the test gates are non-negotiable; one atomic, professional commit per completed phase.
## Objective
Run the complete quality gate for the phase — unit + integration green, `app/` coverage >90%, the new E2E + the three phase-106/28/35 regression E2E suites green in isolation, ruff + pyright clean — then land the single `--no-gpg-sign` Conventional Commit and move the phase directory to `complete/`.
## Work
1. DB up: `podman compose up -d db` (and `uv run alembic upgrade head` if the dev DB is behind — no NEW migration this phase, so head is unchanged).
2. Unit + integration: `uv run pytest` — green.
3. Coverage: `uv run pytest --cov=app --cov-report=term-missing` — TOTAL >90% (no `app/` code changed this phase; this is the regression check on the phase-106 suites + the new E2E's `app/` exercise).
4. E2E, each in isolation (`--no-cov`):
- the NEW suite: `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov`
- regressions (the three suites whose behavior this phase touches or that pin phase-106 dates end to end):
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov`
- `uv run pytest tests/e2e/test_sync_button.py -v --no-cov`
- `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov`
5. Lint + types: `uv run ruff check . && uv run pyright` — clean.
6. UI Structure Check (AGENTS.md rule 5): the phase adds NO UI of its own — the date columns/badge asserted by task 02 are phase-106 UI, re-verified only; confirm nothing in `frontend/` changed this phase (`git status` shows none) so the byte-identical contracts are untouched.
7. Commit (exactly one atomic commit, `--no-gpg-sign`, the 00_phase.md message):
```bash
git add scripts/git_sync.py scripts/import_docs.py app/api/sync.py tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(git): full-history checkouts so URL sources get true per-file document dates"
```
8. Move the phase directory: `mv .agents/phases/todo/107_git_full_history_dates .agents/phases/complete/` (the pipeline gate does this on success — do it only after step 7 succeeds and include the move in the SAME commit's tree if the gate script does not, per the house protocol; check `.agents/pipeline.log` / the phased-execution gate for how prior phases recorded the move and match it).
## Testing & Quality
- This task runs, not writes, the gate: every command above must pass before the commit exists.
- Coverage: **>90%** on `app/` (TOTAL line of the `term-missing` report).
## Completion Criteria
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
- [ ] `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` green in isolation (DB up)
- [ ] `tests/e2e/test_document_dates.py`, `tests/e2e/test_sync_button.py`, `tests/e2e/test_git_sources_admin.py` each green in isolation
- [ ] `uv run ruff check . && uv run pyright` clean
- [ ] Exactly one new commit, message `fix(git): full-history checkouts so URL sources get true per-file document dates`, signed with `--no-gpg-sign`; `git status` clean afterwards (only gitignored runtime artifacts aside)
- [ ] Phase dir at `.agents/phases/complete/107_git_full_history_dates/`
@@ -0,0 +1,56 @@
# Phase 108 — History wire check: verify (or fix) the "missing first turn" follow-up bug
**Source:** `TODO.md` L4 (owner 2026-09-16): "I've noticed at least one instance where a follow-up chat is missing the first message and response as context. So if I ask 'What is my name' and then 'What did I just ask you?' the model responds 'This is the first question you've asked'. But if I send a third message 'What was the previous question' the model responds correctly 'What did I just ask you?' Just check if there's a bug, there may not be and this was user error"
**Story:** n/a (owner bug report, phase-74 follow-up — the phase's E2E proves the wire end to end).
**Context (traced 2026-09-16):** the chat-history wire landed in phase 74 and is three layers deep: (1) the CLIENT maps the `bor.chat.v1` conversation record minus the current question into the request body — `conversation.slice(0, -1)` → `{who, text, thinking?}` per turn (`frontend/assets/app.js` L2247, invariant comment L2229-2246 — the phase-49 retry and phase-53 stale-regen paths pop the old answer before re-sending, so `slice(0,-1)` is exactly the prior turns); (2) the SERVER trims + maps — `history_to_messages` (`app/rag/prompts.py` L199-252): walks NEWEST-FIRST, keeps turns while BOTH budgets hold (`history_max_turns` default **40**, `history_max_chars` default **24_000** — `app/config.py` L94/L101), drops a whole turn on overflow, returns the kept window chronological; `user`→user, `brain`→assistant with `reasoning_content` ONLY when thinking is non-empty (A4); (3) the ENDPOINT splices the block between the system prompt and the current user message on BOTH turn branches (deflected + grounded — `app/api/chat.py` L353-358, "BOTH branches below … reuse the same block"). A short 2-turn conversation is orders of magnitude under both budgets, and both the client mapping and the trimmer READ correctly — so this phase is a deterministic three-layer VERIFICATION with a built-in fix branch, not a rewrite. The wire oracle already exists: the mock LLM's `HISTORY_TRIGGER = "echo my history"` (`tests/e2e/mock_llm.py` L597; checked at L1671 BEFORE the DEFLECT_MODE branch — the echo fires on both branches) answers with `_history_echo(body)` (L1492): a byte-stable `history: N prior messages; last answer tail: <last 24 chars of the most recent prior assistant message, or "none">; thinking: yes|no` — exactly what the owner's scenario needs. The phase-74 E2E (`tests/e2e/test_llm_history.py`) already asserts on this echo, deriving the expected tail from the localStorage record; the existing suites to extend live at `tests/unit/test_history.py` (the pure trimmer) and `tests/integration/test_chat_api.py` (the `HISTORY`/`HISTORY_MESSAGES` idiom L1620-1656 + the `_stream_chat_with_history` helper L1659).
## Objective
Prove — at the trimmer, the endpoint, and the full browser wire — that a follow-up question carries the COMPLETE prior conversation (the owner's exact 2- and 3-turn scenarios, byte-exact via the history echo), and either ship the minimal fix at the layer that reproduces the missing-first-turn symptom or record the verdict "no bug — model behavior/user error" with the pins as the permanent guard.
## Dependencies
- `74_llm_chat_history` (complete) — the feature under verification: the `history` request field, the trimmer, both-branch splicing, the echo marker, and the suites this phase extends. All its pins are regression gates.
- `17_thinking_display` (complete) — the record's `thinking` key and the A4 `reasoning_content` wire convention the echo's `thinking: yes|no` term covers.
- `49_retry_answer` / `53_stale_saved_chats` (complete) — the client paths (retry, stale-regen) that POP the old answer before re-sending; the client-invariant comment names them — if the client layer ever reproduces, their pop logic is the first suspect.
## Design (shared by all tasks — the executor reads this, not the chat)
- **The three layers, each isolating a suspect (tasks 01-02):**
1. **Unit — the trimmer** (task 01): the owner's exact shape — a 2-turn history (user Q1, brain R1) under the default budgets → ALL turns kept, chronological, roles mapped, thinking mapped (non-empty → `reasoning_content`, empty/absent → key absent). If this fails, the bug is in `history_to_messages` and nothing else needs running.
2. **Integration — the endpoint** (task 01): the SAME 2-turn history through the real `POST /api/chat` (the `test_chat_api.py::_stream_chat_with_history` idiom): the SSE turn completes AND the LLM request the turn made carries exactly `[system, user Q1, assistant R1, user Q2]` (captured per the house fake-LLM pattern). If layer 1 passes and this fails, the bug is in the endpoint plumbing (the `request.history` → `hist` → prompt splice, one of the two branches).
3. **E2E — the full client wire** (task 02): the owner's exact 3-message scenario in the browser, echo markers on turns 2 and 3 (see task 02 for the messages + expected echoes). If layers 1-2 pass and this fails, the bug is in the CLIENT record→history mapping (push/pop timing, the phase-49/53 paths, localStorage restore).
- **The verdict (task 03, D13):** all three green → NO BUG: the wire is proven complete at every layer; the reported instance is model behavior/user error (the owner's own hypothesis). The pins stay as the permanent guard (a future regression that drops the first turn fails layer 1, 2, or 3). A failure at layer N → the bug reproduces at layer N; the executor makes the MINIMAL fix in that layer, re-runs the failing layer green, and the verdict records the fix + evidence. `VERDICT.md` (NEW file inside this phase dir) is written BEFORE the commit and states: the layer outcomes, the verdict, and (if fixed) the one-line root cause.
- **NOT touched (D13/D14):** the A10 stateless contract, the budget defaults (40 turns / 24k chars), the `bor.chat.v1` record schema, the echo's format (the existing marker IS the oracle — D14: NO new mock marker this phase), any phase-74 pin (regression), `PLAN.md`, completed phases.
## Tasks
1. `01_server_wire_verification.md` — layer 1 (unit pins on the trimmer for the owner's 2-turn shape) + layer 2 (integration: real endpoint, captured LLM request = full prior history).
2. `02_client_e2e_owner_scenario.md` — layer 3: new E2E `tests/e2e/test_history_wire_check.py` (isolation) — the owner's exact 3-message scenario, byte-exact echo assertions on turns 2 and 3.
3. `03_verdict_fix_or_pin.md` — read the layer outcomes; fix the reproducing layer (or record "no bug"); `VERDICT.md`; full gate; atomic commit.
## Testing & Quality
- Unit — `tests/unit/test_history.py` (extended): the 2-turn-under-budget keep-all pin + the role/thinking mapping for that shape (the existing budget/trim pins stay green — regression).
- Integration — `tests/integration/test_chat_api.py` (extended): the 2-turn request through the real endpoint with the captured-LLM-request assertion (the house fake-LLM capture pattern; the existing phase-74 history tests stay green).
- E2E (mandatory, A16) — `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` with the DB up.
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing` — the validate.sh gate).
## Completion Criteria
- [ ] Layer 1 green: a 2-turn history under the default budgets survives `history_to_messages` whole, chronological, correctly mapped (unit pin).
- [ ] Layer 2 green: a real `POST /api/chat` with a 2-turn history makes the LLM request `[system, user Q1, assistant R1, user Q2]` — the server wire is proven complete (or the bug is fixed here).
- [ ] Layer 3 green: the owner's scenario in the browser — turn 2's echo shows `history: 2 prior messages` + R1's exact 24-char tail; turn 3's echo shows `history: 4 prior messages` + R2's tail (or the bug is fixed at the client).
- [ ] `VERDICT.md` exists in the phase dir: layer outcomes + the verdict (fixed-at-layer-N with root cause, or "no bug — model behavior") — written before the commit.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` green in isolation; `uv run pytest tests/e2e/test_llm_history.py -v --no-cov` (phase 74) green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit (message per the verdict — see Commit); phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **D13 — Verify-or-fix protocol (owner-instructed: "Just check if there's a bug, there may not be").** The phase's deliverable is the three-layer pins + a recorded verdict. Code changes happen ONLY when a layer reproduces the missing-turn symptom, are MINIMAL, and are confined to the reproducing layer — no A10 contract change, no budget-default change, no record-schema change, no new endpoint. If no layer reproduces, the phase ships tests-only.
- **D14 — The existing echo IS the oracle.** The phase-74 `echo my history` marker (`_history_echo`) is reused unmodified — its `N prior messages` count + `last answer tail` are exactly the owner-scenario assertions; NO new mock marker is added this phase (new markers land only in phases that change prompt/tool shapes).
## Commit
```bash
# verdict = no bug (tests-only):
git add tests/ .agents/phases/ && git commit --no-gpg-sign -m "test(chat): history-wire verification pins — TODO L4 verdict: no bug (model behavior)"
# verdict = bug found (adjust <layer> to the fix site):
git add <fixed files> tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(chat): <layer> — follow-up turns carry the full prior history (TODO L4)"
```
@@ -0,0 +1,29 @@
# Task 01 — Server wire verification: the trimmer (unit) + the endpoint (integration)
**Phase:** `108_history_wire_check` · **Source:** `TODO.md` L4 — "a follow-up chat is missing the first message and response as context … Just check if there's a bug."
## Objective
Prove (or disprove) the two SERVER layers of the history wire for the owner's exact 2-turn shape: a short history must survive `history_to_messages` whole and reach the LLM as the complete prior conversation on the real `POST /api/chat`.
## Work
1. `tests/unit/test_history.py` (EXISTING — extend, keep every pin green) — add the owner-shape pins for the DEFAULT budgets (no env overrides; construct `Settings` the file's existing `_settings()` way):
- `test_short_two_turn_history_kept_whole_and_chronological` — `history = [user "What is my name?", brain "Your name is Reese."]` (the owner's own Q1/R1) → `history_to_messages` returns exactly `[{"role": "user", "content": "What is my name?"}, {"role": "assistant", "content": "Your name is Reese."}]` — both turns, chronological, no trim, no reordering.
- `test_two_turn_history_thinking_mapping` — the same 2-turn history with the brain turn carrying a non-empty `thinking` → the assistant message gains `reasoning_content` (A4); with `thinking` empty/absent → the key is ABSENT (not an empty string).
- (If either pin fails: STOP — layer 1 reproduces the bug. Fix `app/rag/prompts.py::history_to_messages` minimally (D13), keep this task's pins + the existing suite green, and note the root cause for task 03's `VERDICT.md`. Do not touch the budget defaults.)
2. `tests/integration/test_chat_api.py` (EXISTING — extend next to the phase-74 history block, L1620-1670) — add the endpoint-layer pin:
- Reuse the file's `_stream_chat`/`_stream_chat_with_history` helpers + fake-LLM capture pattern (read the file's existing setup first — match its house idiom for capturing what the LLM was called with).
- `test_endpoint_two_turn_history_reaches_the_llm` — `POST /api/chat {message: "What did I just ask you?", history: [{who: user, text: "What is my name?"}, {who: brain, text: "Your name is Reese."}]}` → the SSE stream completes (`done`), and the chat request the turn made to the LLM carries, IN ORDER, the system prompt, `user "What is my name?"`, `assistant "Your name is Reese."`, then the current `user` question — i.e. the 2 prior turns are NOT dropped (the owner's symptom would be their absence). Assert on the captured `messages` list (roles + contents, exact).
- The turn may be LOW/deflected with an empty KB (the history block is branch-independent — pinned in phase 74) or HIGH with one seeded fixture doc (the file's existing seeding idiom) — either is fine; pick what the file's helpers make easiest and say so in a comment.
- (If this fails while layer 1 passed: the bug is in the endpoint plumbing — `app/api/chat.py`'s `request.history` → `hist` → prompt splice. Fix minimally (D13), keep this pin + the phase-74 pins green, note the root cause for task 03.)
3. Run `uv run pytest tests/unit/test_history.py tests/integration/test_chat_api.py -v` (DB up: `podman compose up -d db`) — green.
## Testing & Quality
- Unit: the owner-shape trimmer pins (keep-all + mapping) alongside the existing budget pins.
- Integration: the real endpoint with a captured LLM request — the server wire proven (or fixed) at the exact layer.
- Coverage: **>90%** on `app/` (no `app/` change unless a fix is needed; the gate still passes).
## Completion Criteria
- [ ] `tests/unit/test_history.py` green with the two new owner-shape pins (or the trimmer fixed + pinned)
- [ ] `tests/integration/test_chat_api.py` green with `test_endpoint_two_turn_history_reaches_the_llm` (or the endpoint fixed + pinned)
- [ ] Every pre-existing pin in both files still green (no regression)
- [ ] `uv run ruff check . && uv run pyright` clean; the layer-1/layer-2 outcome is noteable for task 03's `VERDICT.md` (pass, or pass-after-fix with root cause)
@@ -0,0 +1,29 @@
# Task 02 — Client E2E: the owner's exact 3-message scenario, byte-exact via the history echo
**Phase:** `108_history_wire_check` · **Source:** `TODO.md` L4 — the owner's repro: Q1 "What is my name?" → Q2 "What did I just ask you?" (model claims it's the first question) → Q3 "What was the previous question?" (model answers correctly).
## Objective
Prove (or disprove) the THIRD layer — the full browser wire: the localStorage conversation record → `conversation.slice(0,-1)` mapping → request body → the LLM — using the owner's exact scenario and the phase-74 `echo my history` oracle (D14: no new marker).
## Work
1. `tests/e2e/test_history_wire_check.py` (NEW — copy the app-server + fixture idiom from `tests/e2e/test_llm_history.py`: module-scoped mock-LLM app, the fixture-docs import for a non-empty KB, `e2e.auth_helpers.login`, the localStorage `bor.chat.v1` record reads, per-test conversation reset; module docstring: story n/a — owner bug report 2026-09-16, the isolation command, and what each test pins). Isolation: `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` (DB up).
- **The echo oracle, recalled** (mock `_history_echo`, byte-stable): `history: N prior messages; last answer tail: <LAST 24 CHARS of the most recent prior assistant message's content, or "none">; thinking: yes|no` — N = non-system messages before the LAST user message (the current question excluded); checked BEFORE the DEFLECT_MODE branch, so the echo fires whatever gate branch the turn takes (the owner's questions may deflect — that's fine, the marker is in the USER message).
- **Tests:**
1. `test_cold_start_echo_shows_no_phantom_history` — fresh conversation; ask `echo my history` as the FIRST message → the answer bubble contains `history: 0 prior messages; last answer tail: none; thinking: no` (the cold-start pin: no phantom prior turns).
2. `test_owner_scenario_three_turns_carry_the_full_prior_history` — the owner's exact scenario, echo marker APPENDED to turns 2 and 3 (their words preserved verbatim as the prefix):
- T1: `What is my name?` → R1 (the mock's deterministic answer — read R1's raw text from the `bor.chat.v1` record's brain entry, NOT from the rendered DOM).
- T2: `What did I just ask you? echo my history` → R2's bubble text must contain `history: 2 prior messages; last answer tail: {R1[-24:]}; thinking: no` (R1 = the record's brain text; `thinking: no` — T1 never triggered the thinking marker). **THE regression pin: the owner's bug renders this as `0 prior messages` / `last answer tail: none`.**
- T3: `What was the previous question? echo my history` → R3's bubble text must contain `history: 4 prior messages; last answer tail: {R2[-24:]}` (R2 = the echo answer itself — also from the record).
- Read the expected tails from the localStorage record AFTER each turn persists (the `test_llm_history.py` pattern — the record the client saved IS what the client sends next, so what the record shows is what the model received).
- If this test fails while tasks 01's layers passed: the bug is in the CLIENT mapping (suspects, in order: the `conversation.slice(0,-1)` sites, the phase-49 retry / phase-53 stale-regen pop paths, the record persistence timing — `frontend/assets/app.js` L2187/L2247). Fix minimally (D13), keep this test + `tests/e2e/test_llm_history.py` green, note the root cause for task 03's `VERDICT.md`.
2. Run the suite in isolation — green (or pass-after-fix).
## Testing & Quality
- E2E (the task IS the test): the full browser wire, byte-exact via the existing echo oracle.
- Coverage: `--no-cov` suite; it exercises `app/` (chat endpoint, history mapping) for real — the `app/` >90% gate is unaffected.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` green in isolation (DB up) — both tests
- [ ] The regression pin holds: turn 2's echo shows `2 prior messages` + R1's exact tail; turn 3 shows `4 prior messages` + R2's exact tail (or the client bug is fixed + pinned)
- [ ] `tests/e2e/test_llm_history.py` (phase 74) still green in isolation (regression)
- [ ] The layer-3 outcome is noteable for task 03's `VERDICT.md` (pass, or pass-after-fix with root cause)
@@ -0,0 +1,34 @@
# Task 03 — Verdict (fix or pin), full gate, atomic commit
**Phase:** `108_history_wire_check` · **Source:** `TODO.md` L4 — "Just check if there's a bug, there may not be and this was user error"; AGENTS.md rules 8/9 — the test gates are non-negotiable.
## Objective
Record the phase's verdict with its evidence, run the complete quality gate, and land the single `--no-gpg-sign` commit — tests-only if no bug was found (the owner's expected outcome), fix + tests if one layer reproduced.
## Work
1. **The verdict** — read the layer outcomes from tasks 01-02 (their test results + any fix notes):
- **All three layers green (no fix needed):** the wire is proven complete at the trimmer, the endpoint, and the full browser wire → verdict **NO BUG**: the owner's reported instance was model behavior/user error. The pins stay as the permanent guard (a future regression that drops the first turn fails layer 1, 2, or 3).
- **A layer reproduced (pass-after-fix):** the bug is fixed at that layer → verdict **BUG FOUND + FIXED at <layer>**, with the one-line root cause.
- Write `.agents/phases/todo/108_history_wire_check/VERDICT.md` BEFORE the commit: the three layer outcomes (pass / pass-after-fix + root cause / fail-should-not-occur), the verdict, and the evidence (which test names carry the pins). Keep it short — it is the durable record the owner asked for ("just check").
2. **The full gate** (DB up: `podman compose up -d db`; every command must pass before the commit):
- `uv run pytest` — green.
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL >90%.
- E2E in isolation: `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` (NEW) and `uv run pytest tests/e2e/test_llm_history.py -v --no-cov` (phase-74 regression).
- `uv run ruff check . && uv run pyright` — clean.
- No-regression spot check: `git diff --stat` shows ONLY the files this phase may touch — `tests/**`, `VERDICT.md`, `.agents/phases/**`, and (only if a bug was fixed) the single reproducing layer's file. If the diff shows anything else, stop and fix the scope before committing.
3. **The commit** (exactly one, `--no-gpg-sign`, per the 00_phase.md branch):
- no bug: `git add tests/ .agents/phases/ && git commit --no-gpg-sign -m "test(chat): history-wire verification pins — TODO L4 verdict: no bug (model behavior)"`
- bug found: `git add <fixed files> tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(chat): <layer> — follow-up turns carry the full prior history (TODO L4)"`
4. Move the phase directory: `mv .agents/phases/todo/108_history_wire_check .agents/phases/complete/` (the pipeline gate does this on success — do it only after the commit, and match how prior phases recorded the move).
## Testing & Quality
- This task runs, not writes, the gate: every command above must pass before the commit exists.
- Coverage: **>90%** on `app/` (TOTAL line of the `term-missing` report).
## Completion Criteria
- [ ] `VERDICT.md` in the phase dir: layer outcomes + verdict (no bug / fixed-at-<layer> + root cause) + the pin test names
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
- [ ] `tests/e2e/test_history_wire_check.py` + `tests/e2e/test_llm_history.py` green in isolation (DB up)
- [ ] `uv run ruff check . && uv run pyright` clean; the diff is scoped to this phase's allowed files
- [ ] Exactly one new commit with the verdict-branch message, `--no-gpg-sign`; `git status` clean afterwards (only gitignored runtime artifacts aside)
- [ ] Phase dir at `.agents/phases/complete/108_history_wire_check/`
@@ -0,0 +1,37 @@
# Phase 108 Verdict — History wire check (TODO.md L4)
**Verdict: NO BUG.** The follow-up history wire is proven complete at all
three layers, for the owner's exact scenario, byte-exact via the phase-74
`echo my history` oracle (D14 — no new marker). The reported instance
("What did I just ask you?" → "This is the first question you've asked")
was **model behavior / user error**, exactly the owner's own hypothesis
("Just check if there's a bug, there may not be"). No `app/` or
`frontend/` changes were needed — the phase ships tests-only (D13).
## Layer outcomes
| Layer | What it isolates | Outcome | Pins (evidence) |
|-------|------------------|---------|-----------------|
| 1 — unit: the trimmer | `history_to_messages` (`app/rag/prompts.py`) | **PASS** — the owner's 2-turn history (Q1 "What is my name?" / R1 "Your name is Reese.") under the DEFAULT budgets (40 turns / 24 000 chars, no overrides) survives whole: chronological, roles mapped, thinking → `reasoning_content` only when non-empty (A4) | `tests/unit/test_history.py::test_short_two_turn_history_kept_whole_and_chronological`, `::test_two_turn_history_thinking_mapping` |
| 2 — integration: the endpoint | `POST /api/chat` splice, both branches (`app/api/chat.py`) | **PASS** — the SAME 2-turn history through the real endpoint (follow-up "What did I just ask you?") makes the LLM request exactly `[system, user Q1, assistant R1, user Q2]`; SSE `done`; log line carries `history_msgs=2`. Branch-agnostic by design (phase 74 pinned the splice on both branches) | `tests/integration/test_chat_api.py::test_endpoint_two_turn_history_reaches_the_llm` |
| 3 — E2E: the full browser wire | client record → `conversation.slice(0, -1)` → body → LLM (`frontend/assets/app.js`) | **PASS** — the owner's verbatim 3-message scenario in a real browser: T2's echo = `history: 2 prior messages` + R1's exact 24-char tail; T3's echo = `history: 4 prior messages` + R2's exact tail (tails read from the persisted `bor.chat.v1` record — what the client saved is what it sends). The missing-first-turn bug would render T2 as `0 prior messages` / `last answer tail: none`; it does not | `tests/e2e/test_history_wire_check.py::test_cold_start_echo_shows_no_phantom_history`, `::test_owner_scenario_three_turns_carry_the_full_prior_history` |
## Why "no bug" is sound
- Layers 1 and 2 prove the server drops nothing: a short history is
orders of magnitude under both budgets, and the captured LLM request on
the real endpoint carries the complete prior exchange, chronological.
- Layer 3 proves the client sends what it saved: the echo's `N prior
messages` count and `last answer tail` are computed by the mock from
the request itself, and both matches hold byte-exactly for turns 2 and 3.
- The deterministic mock leaves no room for "the model forgot" at the
wire level — the model (turbo) simply receives the full history and
occasionally mis-reports it. The owner's third message working correctly
is consistent with model non-determinism over a complete wire.
## The pins stay as the permanent guard
A future regression that drops the first turn — in the trimmer, the
endpoint splice, or the client's record→history mapping (including the
phase-49 retry / phase-53 stale-regen pop paths) — fails layer 1, 2, or 3
immediately.
@@ -0,0 +1,52 @@
# Phase 109 — Never-frozen turn: re-expanding thinking block + the persistent in-turn loader
**Source:** `TODO.md` L3 (owner 2026-09-16): "Thinking can happen after the model starts responding. This sometimes results in a the chat appearing 'frozen' because the model responds, calls a tool, then continues thinking without re-expanding the thinking block. There should be a visual that the chat is still progressing regardless of what state it's in (some kind of loader will do)."
**Story:** n/a (owner request — extends the phase 17/48/87 thinking/tool feedback under the PLAN §7.4 never-stale contract; the phase's E2E proves the reported repro no longer freezes).
**Context (traced 2026-09-16):** the turn's visible feedback is state-driven in `frontend/assets/app.js`: the `UI_STATE` machine (L340-345: `idle`/`thinking`/`streaming`/`error`) is owned by `setUiState` (L1243) — the typing bubble (a `#typing-indicator` message with the animated `.bubble.typing` dots, `addTyping` L859, the 10s elapsed-seconds clock L1137) shows ONLY in `thinking`; `inFlight = thinking|streaming` drives the Stop button (L1251-1255); `#send-status` is the sole a11y live region (L347-353; visual elements are `aria-hidden` — the L1791 house pattern). The reported freeze, exactly as the owner described it: the `delta` handler (L2357-2370) runs `setUiState(streaming)` on the FIRST delta — which `removeTyping()`s the dots — then `closeThinkingBlock(wrap)` (L905-909: "auto-collapse; idempotent, **never reopens**"). A LATER `thinking` frame (the next agent round — the model answered, called a tool, then thinks again) hits the `thinking` handler (L2268-2301), which only appends to the collapsed block's `.thinking-text` — nothing is visible: the answer text is static, the dots are gone, the scratchpad is closed → the chat reads as frozen. Every OTHER state already has a cue: pre-delta thinking = live open block + dots; tool = the `.tool-call` line with the phase-87 `(Ns)` elapsed counter (plus relabeled dots pre-delta); streaming = growing text; retry = the status line. The post-delta thinking gap is the ONE uncovered state — and the owner wants a constant cue anyway ("regardless of what state it's in"). The CSS lives in `frontend/assets/styles.css` (the typing-dots rules there; `prefers-reduced-motion` is house law, §7.2). House patterns: unit pins read the assets as text (`tests/unit/test_frontend_tool_states.py` / `test_frontend_feedback.py` already pin the thinking-block + typing behavior); the mock LLM has `THINKING_TRIGGER = "think out loud"` (mock_llm.py L500 — streams ~700 chars of `reasoning_content` ahead of content) and the multi-round `tool_calls` markers (L77-191); the house rule is that a marker/regex change lands WITH its consuming task (PLAN §4).
## Objective
The chat never reads as frozen: (1) a `thinking` frame RE-OPENS the thinking block after the answer has started (the block is open-while-thinking / closed-while-answering — the reported symptom, fixed at the handler), and (2) a compact persistent loader is visible for the ENTIRE active turn (send → terminal frame) in the composer status area — the constant progress cue the owner asked for, driven by the single `setUiState` owner so it can never go stale (§7.4).
## Dependencies
- `17_thinking_display` (complete) — the thinking block, the follow-the-tail pin contract (`THINKING_NEAR_BOTTOM_PX`), the restore-path collapsed rendering; the block's unit pins live in the `test_frontend_*` suites this phase extends.
- `48_stop_generation` (complete) — the `inFlight`/Stop-button state ownership the loader's single-owner toggle joins.
- `87_big_read_progress` (complete) — the `.tool-call` line + the `armToolLineClock`/`settleToolLine` `(Ns)` counter (the "at least one cue" inventory's tool entry).
- `06_loading_feedback` (complete) — the UI state machine + the never-stale feedback contract this phase extends (every state keeps a defined UI).
## Design (shared by all tasks — the executor reads this, not the chat)
- **D15 — The thinking block becomes a TOGGLE (task 01).** The `thinking` SSE handler gains `block.open = true` after `ensureThinkingBlock(wrap)` (idempotent — a no-op while already open, so the pre-delta live flow is byte-identical in behavior); the `delta` handler KEEPS its `closeThinkingBlock(wrap)`. The contract flips from "never reopens" to **open-while-thinking, closed-while-answering** — the block reflects the model's current activity in every agent round. `closeThinkingBlock`'s docstring/comment updates (the "never reopens" claim is gone — the delta handler closes, the thinking handler opens). The follow-the-tail pin logic (L2293: `block.open && isThinkingNearBottom(textEl)` measured BEFORE the re-render) is UNCHANGED — it already keys off `block.open`, so a re-opened block resumes pinned tail-following exactly like the live pre-delta block. The phase-14 RESTORE path (`renderStoredMessage` L1517-1518) still renders stored blocks collapsed — untouched.
- **D16 — The turn loader (task 02): a static shell element, single-owner visibility.**
- `frontend/index.html` — ONE static element in the composer's status row (next to the `#send-status` live region): `<div id="turn-loader" class="turn-loader" aria-hidden="true" hidden></div>` — static markup, hidden by default (no JS-built HTML — the createElement/textContent house rule; no document-derived data anywhere near it).
- `frontend/assets/app.js` — `setUiState` (L1243) is the SOLE owner, exactly like the existing `is-stop` toggle: `turnLoader.hidden = !inFlight` (shown iff `uiState ∈ {thinking, streaming}`). Every terminal path funnels through `setUiState` (done → `idle`, error → `error`, stop/timeout → `error`/`idle` per the existing handlers), so the loader CANNOT be left visible in a terminal state — the §7.4 never-stale guarantee comes from the single-owner pattern, not from per-handler cleanup.
- `frontend/assets/styles.css` — `.turn-loader` next to the typing-dots rules: compact, reuses the EXISTING typing-dot animation (same keyframes/dot styling — no new animation family), provenance comment citing phase 109 + `TODO.md` L3, and a `prefers-reduced-motion` variant mirroring the typing dots' treatment (static dots, no pulse). Contrast N/A (the dots are decorative — `aria-hidden` + the `#send-status` announcer carry meaning; §7.2 "text + color, never color alone" — the state TEXT stays in `#send-status`).
- **The invariant (unit + E2E):** while a turn is active, at least one visible progress cue is ALWAYS present — the loader (constant, D16), the open thinking block (thinking frames, D15), the tool-line `(Ns)` counter (tool frames, phase 87), or the growing answer text (streaming). D16 makes it true by construction; task 03 proves the reported repro (delta → tool → thinking-after-delta) no longer freezes.
- **NOT touched:** the `UI_STATE` set, the `SEND_STATUS` copy, the typing bubble's own lifecycle (it still shows only pre-delta, per the phase-17 contract — the loader is a SEPARATE constant cue, not a re-homing of the dots), `#send-status` (unchanged — still the sole a11y announcer), the mock's EXISTING markers, the server (this is a pure UI phase — `app/` is untouched, so `app/` coverage is a regression check only), `PLAN.md`, completed phases.
## Tasks
1. `01_thinking_block_reexpand.md` — the `thinking` handler re-opens the collapsed block; the delta handler keeps closing; the "never reopens" narrative updated; unit pins (read-the-assets pattern).
2. `02_turn_active_loader.md` — the static `#turn-loader` element, the `setUiState` single-owner toggle, the CSS (reused dot animation + reduced-motion + provenance), unit pins.
3. `03_e2e_and_gate.md` — the new mock marker forcing the reported `delta → tool → thinking-after-delta` sequence (lands WITH this task), the dedicated E2E `tests/e2e/test_turn_progress_loader.py` (isolation), the a11y pass, the full gate, the atomic commit.
## Testing & Quality
- Unit — `tests/unit/test_frontend_turn_loader.py` (NEW, the house read-the-assets-as-text pattern): task 01 pins — the `thinking` handler contains the `block.open = true` re-open (after `ensureThinkingBlock`), the `delta` handler still calls `closeThinkingBlock`, `closeThinkingBlock`'s docstring no longer claims "never reopens", the restore path (`renderStoredMessage`) still sets `block.open = false`; task 02 pins — `index.html` carries exactly one `#turn-loader` with `aria-hidden="true"` + `hidden`, `setUiState` is the SOLE writer of `turnLoader.hidden` (cross-file single-owner check: `turnLoader.hidden` appears nowhere else in `app.js`), the `.turn-loader` CSS rule exists next to the typing rules with the reduced-motion variant + the phase-109 provenance comment.
- E2E (mandatory, A16) — `tests/e2e/test_turn_progress_loader.py` (task 03): `uv run pytest tests/e2e/test_turn_progress_loader.py -v --no-cov` with the DB up.
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing` — unchanged by this UI phase; the gate is the regression check).
## Completion Criteria
- [ ] A `thinking` frame after the answer has started RE-OPENS the thinking block (the block is open-while-thinking / closed-while-answering; pre-delta flow + restore path unchanged — unit-pinned).
- [ ] `#turn-loader` is visible for the entire active turn and hidden in every terminal state — owned solely by `setUiState` (unit-pinned single-owner + the E2E's start/mid/end samples).
- [ ] E2E green in isolation: the reported repro (delta → tool → thinking-after-delta) shows the re-opened scratchpad with the new thinking text, the loader visible throughout, hidden after `done`; `#send-status` carries the state text (the loader is `aria-hidden`).
- [ ] The phase-17/48/87/6 regressions green in isolation: `uv run pytest tests/e2e/test_thinking_display.py -v --no-cov`, `tests/e2e/test_stop_generation.py -v --no-cov`, `tests/e2e/test_big_read_progress.py -v --no-cov`, `tests/e2e/test_loading_feedback.py -v --no-cov`; `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **D15 — The thinking block is a toggle, not a one-way door (owner-instructed: "continues thinking without re-expanding the thinking block" is THE reported defect).** `thinking` frames open the block (idempotent), `delta` frames close it; the follow-the-tail pin contract and the restore path are unchanged. No new block, no new state — the existing scratchpad reflects the model's current activity in every round.
- **D16 — The constant cue is a separate static loader, owned by `setUiState` (owner-instructed: "a visual that the chat is still progressing regardless of what state it's in — some kind of loader will do").** A static `#turn-loader` in the composer status row (reused typing-dot animation, `aria-hidden`, `#send-status` stays the sole announcer), shown iff `inFlight` — the single-owner pattern makes a stale loader impossible. The typing bubble's own pre-delta lifecycle is NOT re-homed (the phase-17 contract stands); the loader ADDS the constant the owner asked for.
## Commit
```bash
git add frontend/index.html frontend/assets/app.js frontend/assets/styles.css tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(chat): never-frozen turn — re-expanding thinking block + the persistent in-turn loader"
```
@@ -0,0 +1,28 @@
# Task 01 — The thinking block becomes a toggle: re-open on `thinking` frames, close on `delta`
**Phase:** `109_turn_progress_loader` · **Source:** `TODO.md` L3 — "the model responds, calls a tool, then continues thinking without re-expanding the thinking block."
## Objective
Fix the reported freeze at its source: a `thinking` frame that arrives after the answer has started re-opens the collapsed thinking block (D15) — the scratchpad is visible exactly while the model is thinking, in every agent round — while the `delta` handler keeps closing it and nothing else about the block's lifecycle changes.
## Work
1. `frontend/assets/app.js` — the `thinking` SSE handler (L2268-2301): after `const block = ensureThinkingBlock(wrap);`, add `block.open = true;` (idempotent — while the block is already open (the pre-delta live flow) this is a no-op, so that flow's behavior is unchanged; after a `delta` closed it, this re-opens it for the new round's thinking). Update the handler's comment to state the toggle contract (D15): open-while-thinking, closed-while-answering — phase 109, `TODO.md` L3.
- The follow-the-tail logic below it (`const pinned = block.open && isThinkingNearBottom(textEl);`, L2293) is UNCHANGED — it already reads `block.open` before the re-render, so a re-opened block resumes pinned tail-following exactly like the live pre-delta block.
2. `frontend/assets/app.js` — `closeThinkingBlock` (L905-909): keep the function exactly as-is (the `delta` handler still calls it, L2369); update its docstring/comment — the "never reopens" claim is replaced by the toggle contract (the `thinking` handler re-opens; the `delta` handler closes).
- `renderStoredMessage` (L1517-1518 — the phase-14 restore path) is UNTOUCHED: stored blocks still render collapsed.
3. `tests/unit/test_frontend_turn_loader.py` (NEW — the house read-the-assets-as-text pattern; copy the file header/docstring conventions from `tests/unit/test_frontend_tool_states.py`):
- `test_thinking_handler_reopens_the_collapsed_block` — the `thinking` handler source contains `block.open = true` positioned AFTER the `ensureThinkingBlock(wrap)` line (order asserted — the block must exist before it opens).
- `test_delta_handler_still_closes_the_block` — the `delta` handler still calls `closeThinkingBlock(wrap)` (the close side of the toggle survives).
- `test_close_thinking_block_docstring_says_toggle_not_one_way` — `closeThinkingBlock`'s comment no longer contains "never reopens"; the toggle contract (open-while-thinking / closed-while-answering) is documented (assert the new wording, e.g. it names the `thinking` handler's re-open).
- `test_restore_path_still_collapses_stored_blocks` — `renderStoredMessage` still sets `block.open = false` (phase-14 contract regression).
- Cross-file: the `THINKING_NEAR_BOTTOM_PX` pin + `isThinkingNearBottom` logic are untouched (assert the pre-render `block.open &&` guard line still exists in the handler).
4. Run `uv run pytest tests/unit/test_frontend_turn_loader.py -v` + the existing frontend suites that pin this area (`tests/unit/test_frontend_tool_states.py tests/unit/test_frontend_feedback.py tests/unit/test_frontend_scroll.py tests/unit/test_big_read_progress.py`) — green.
## Testing & Quality
- Unit: the toggle contract pinned at the source level (the house pattern for `app.js` behavior — no browser).
- Coverage: **>90%** on `app/` (unchanged — pure frontend task; the gate is the regression check).
## Completion Criteria
- [ ] The `thinking` handler re-opens the block (unit-pinned, correct order); the `delta` handler still closes it; the "never reopens" narrative is gone (unit-pinned)
- [ ] The restore path + the follow-the-tail pin logic are untouched (unit-pinned regressions)
- [ ] The existing frontend unit suites green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,37 @@
# Task 02 — The persistent in-turn loader: static `#turn-loader`, single-owner visibility in `setUiState`
**Phase:** `109_turn_progress_loader` · **Source:** `TODO.md` L3 — "There should be a visual that the chat is still progressing regardless of what state it's in (some kind of loader will do)."
## Objective
Add the constant progress cue (D16): a compact animated loader in the composer status row that is visible for the ENTIRE active turn (send → terminal frame), owned solely by `setUiState` so it can never be left stale — and hidden in every terminal state by construction.
## Work
1. `frontend/index.html` — ONE static element in the composer's status row (the L290-300 region holding `#char-count` / `#send-btn` / `#send-status` — inspect the actual markup and place it adjacent to `#send-status` so it reads as the status line's companion):
```html
<div id="turn-loader" class="turn-loader" aria-hidden="true" hidden></div>
```
Static markup, `hidden` by default (idle on load). No JS-built HTML anywhere (the createElement/textContent house rule — this element is never constructed in JS).
2. `frontend/assets/app.js` — the single-owner toggle (D16):
- At the top of the module with the other element lookups (near `const sendStatus = document.querySelector("#send-status");` L307): `const turnLoader = document.querySelector("#turn-loader");`
- Inside `setUiState` (L1243), next to the existing `sendBtn.classList.toggle("is-stop", inFlight);` (L1251-1255): `turnLoader.hidden = !inFlight;` — shown iff `uiState ∈ {thinking, streaming}`. This is the SOLE writer of `turnLoader.hidden` in the file: every terminal path (done → `idle`, error → `error`, stop/timeout → the existing error/idle landings) funnels through `setUiState`, so the loader is hidden in every terminal state BY CONSTRUCTION — the §7.4 never-stale guarantee, no per-handler cleanup (that is the point; comment it that way).
- Do NOT touch the typing bubble's lifecycle (`addTyping`/`removeTyping` stay exactly as-is — the phase-17 pre-delta contract stands; the loader is a separate constant cue), the `SEND_STATUS` copy, or `#send-status` (still the sole a11y announcer — the loader is `aria-hidden` decoration, the L1791 house pattern).
3. `frontend/assets/styles.css` — the `.turn-loader` rule NEXT TO the typing-dots rules (find the `.bubble.typing` / dots animation block):
- Compact horizontal three-dot indicator, REUSING the existing typing-dot keyframes/dot styling (same animation name — no new animation family; sized down for the status row).
- Provenance comment: phase 109, `TODO.md` L3 — the constant in-turn progress cue; decorative (`aria-hidden`), `#send-status` carries the meaning.
- A `prefers-reduced-motion` variant mirroring the typing dots' treatment (static dots, no pulse — §7.2 house law).
4. `tests/unit/test_frontend_turn_loader.py` (EXTEND the file task 01 created):
- `test_index_html_carries_exactly_one_turn_loader` — `frontend/index.html` contains exactly ONE `id="turn-loader"`, with `aria-hidden="true"` and the `hidden` attribute (hidden by default).
- `test_set_ui_state_is_the_sole_owner_of_the_loader` — in `app.js`: `turnLoader.hidden` appears EXACTLY ONCE, inside `setUiState` (the cross-file single-owner check — grep the file text; any second write site fails the test, keeping the never-stale guarantee structural).
- `test_loader_css_reuses_the_typing_animation_and_reduced_motion` — the `.turn-loader` rule exists in `styles.css` after (or adjacent to) the typing-dots rules, references the SAME animation name as the typing dots, and a `prefers-reduced-motion` block covers it (static, no pulse); the provenance comment names phase 109.
- `test_loader_is_aria_hidden_and_status_untouched` — the loader element is `aria-hidden`; `#send-status`'s attributes are unchanged in `index.html` (still the live region — the house a11y split).
5. Run `uv run pytest tests/unit/test_frontend_turn_loader.py tests/unit/test_frontend_tool_states.py tests/unit/test_theme_frontend.py -v` (the theme suite parses `styles.css` — a new CSS rule must not break the built-in-theme pins) — green.
## Testing & Quality
- Unit: the single-owner invariant + the markup/CSS contract pinned at the source level (the house pattern).
- Coverage: **>90%** on `app/` (unchanged — pure frontend task; the gate is the regression check).
## Completion Criteria
- [ ] `#turn-loader` exists exactly once in `index.html` (static, `aria-hidden`, hidden by default); `setUiState` is its sole visibility owner (unit-pinned)
- [ ] The CSS reuses the typing-dot animation, has the reduced-motion variant + the phase-109 provenance comment (unit-pinned); the theme CSS-parsing suites stay green
- [ ] The typing bubble's lifecycle, `SEND_STATUS`, and `#send-status` are byte-unchanged in behavior (the existing frontend suites green)
- [ ] `uv run ruff check . && uv run pyright` clean; full `uv run pytest` green
@@ -0,0 +1,40 @@
# Task 03 — E2E: the reported repro (delta → tool → thinking-after-delta) + full gate + commit
**Phase:** `109_turn_progress_loader` · **Source:** `TODO.md` L3 — "the model responds, calls a tool, then continues thinking without re-expanding the thinking block" + "a visual that the chat is still progressing regardless of what state it's in"; AGENTS.md rules 4/8/9.
## Objective
Prove the fix end to end with a dedicated Playwright suite: a deterministic mock sequence that replays the owner's exact repro (answer starts, tool call, thinking AFTER the answer) must show the re-opened scratchpad, the loader visible throughout, and clean terminal states — then run the full gate and land the atomic commit.
## Work
1. `tests/e2e/mock_llm.py` — ONE new marker (the house rule: marker/regex changes land WITH their consuming task — this task; document it in the module docstring next to the existing markers):
- A new `*_TRIGGER` constant + branch (checked like the other user-message markers, BEFORE the DEFLECT_MODE branch) whose question forces the reported sequence with BAKED-IN DELAYS (mid-turn windows of ≥1 s each, so Playwright assertions are deterministic — the `slow_llm.py` precedent for deliberate pacing):
- model call 1: ~2 s pre-delay (model latency — the loader's start-state window), then a short `content` delta (2-3 chunks; NO reasoning), then an `ls` `tool_calls` delta (synthetic id, no arguments — the L77-81 pattern), `finish_reason: "tool_calls"`.
- model call 2 (after the server's `tool_result`): `reasoning_content` chunks (~10 × ~0.3 s), then a `content` delta (2-3 chunks ending in a DISTINCTIVE final sentence the tests can match), then a FINAL `reasoning_content` chunk (3 × ~0.3 s), then finish.
- The server is position-independent over the wire (each `reasoning_content` chunk → a `thinking` SSE frame, each `content` chunk → a `delta` frame — `app/rag/llm.py` L572+), so the resulting SSE is exactly `delta → tool → tool_result → thinking → delta → thinking → done` — the owner's repro, deterministic.
2. `tests/e2e/test_turn_progress_loader.py` (NEW — copy the app-server + fixture idiom from `tests/e2e/test_llm_history.py`: module-scoped mock-LLM app, fixture-docs import, `login`, per-test fresh conversation; module docstring: story n/a — owner request 2026-09-16, the isolation command, the marker contract, and what each test pins). Isolation: `uv run pytest tests/e2e/test_turn_progress_loader.py -v --no-cov` (DB up). Each test sends the marker question in its OWN fresh conversation (one full turn per test):
- `test_loader_visible_from_send_through_the_tool_gap` — right after the send (inside call 1's 2 s pre-delay window): `#turn-loader` is VISIBLE (the thinking state, no frame yet); then wait for the `.tool-call` line to appear (the turn is provably in flight) → the loader is STILL visible, the tool line carries the phase-87 elapsed counter, and `#send-status` carries a state text (not empty).
- `test_thinking_block_reopens_after_delta_with_visible_loader` — wait until the thinking block is open with non-empty `.thinking-text` AND the answer bubble already carries call 1's content (i.e. the post-delta re-open — THE reported symptom's state): assert the block is `open`, the new thinking text is VISIBLE in it, and the loader is STILL visible (the frozen window is gone). Then wait for the terminal state (call 2's distinctive final sentence in the bubble, or the send button back to "Send"): the loader is HIDDEN, the block is still open (the LAST frame was thinking), `.thinking-text` is non-empty, the bubble contains BOTH call 1's and call 2's content, and the send button reads "Send" (not "Stop").
- `test_loader_a11y_and_reduced_motion` — after a full turn: the loader element is `aria-hidden="true"` in the DOM (the `#send-status` live region remains the sole announcer — assert its post-done text follows the `SEND_STATUS` idle shape, i.e. not stuck on a mid-turn label); then, in a context with `reducedMotion: "reduce"` (Playwright context option), send a second turn and assert the loader is still visible mid-turn (the reduced-motion variant renders the static dots — the CSS rule, not the visibility, is what changes).
3. Run the suite in isolation — all three tests green.
4. **The full gate** (DB up; every command passes before the commit):
- `uv run pytest` — green.
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL >90%.
- Regressions in isolation (the feedback-state history this phase extends): `uv run pytest tests/e2e/test_thinking_display.py -v --no-cov`, `tests/e2e/test_stop_generation.py -v --no-cov`, `tests/e2e/test_big_read_progress.py -v --no-cov`, `tests/e2e/test_loading_feedback.py -v --no-cov`.
- `uv run ruff check . && uv run pyright` — clean.
- Scope check: `git diff --stat` shows only `frontend/index.html`, `frontend/assets/app.js`, `frontend/assets/styles.css`, `tests/**`, `.agents/phases/**` (this is a UI phase — NO `app/` changes; if the diff shows any, stop and fix the scope).
5. **The commit** (exactly one, `--no-gpg-sign`):
```bash
git add frontend/index.html frontend/assets/app.js frontend/assets/styles.css tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(chat): never-frozen turn — re-expanding thinking block + the persistent in-turn loader"
```
6. Move the phase directory: `mv .agents/phases/todo/109_turn_progress_loader .agents/phases/complete/` (the pipeline gate does this on success — do it only after the commit, matching how prior phases recorded the move).
## Testing & Quality
- E2E: the owner's repro replayed deterministically (the marker's baked delays make every window assertion race-free); the a11y split (visual loader + `#send-status` announcer) + the reduced-motion variant pinned.
- Coverage: **>90%** on `app/` (unchanged by this UI phase — the gate is the regression check).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_turn_progress_loader.py -v --no-cov` green in isolation (DB up) — all three tests
- [ ] The reported repro is pinned: post-delta thinking re-opens the block with visible text while the loader stays visible; terminal states are clean (loader hidden, button "Send", status not stuck)
- [ ] The regression suites (`test_thinking_display`, `test_stop_generation`, `test_big_read_progress`, `test_loading_feedback`) green in isolation; `uv run pytest` green; coverage TOTAL >90%; ruff + pyright clean
- [ ] Exactly one new commit with the phase message, `--no-gpg-sign`; the diff scoped to `frontend/` + `tests/` + `.agents/phases/`; `git status` clean afterwards
- [ ] Phase dir at `.agents/phases/complete/109_turn_progress_loader/`
@@ -0,0 +1,101 @@
# Phase 10 — Story: Clickable Document Viewer
**Story:** `.agents/user_stories/document-viewer.md`
**Context:** `.agents/PLAN.md` §4 (API), §7 (UI/UX)
## Goal
Every document Brain cites — and every row in the Sources table — opens
in the browser: a dark-themed viewer page fed by a stateless content
endpoint served from the database.
## Dependencies
08 (dark theme + tokens — the viewer inherits them) and 09 (multi-format
corpus; content in every format must be viewable). 01–07 as the base.
## Locked decisions
No anchors changed. Adds one stateless endpoint under `/api` (A10) —
PLAN §4 was updated 2026-08-21 with owner permission. No new technology
(A11): vanilla HTML/CSS/JS, content rendered by the existing local
escape-first markdown renderer.
## Implementation steps
1. **Schema + endpoint** (`app/schemas.py`, `app/api/docs.py`):
`DocContent {source, path, title, format, content, indexed_at,
chunks}`; `GET /api/documents/content?source=…&path=…` — look up
`documents` by `(source, path)` **in the database only** (no
filesystem access → no path-traversal surface; `../`-style values
simply aren't rows → 404). `format` = lowercased path suffix. 404 →
`{detail: "document not found"}`.
2. **Shared renderer** — extract the ~60-line escape-first markdown
renderer from `frontend/assets/app.js` into `frontend/assets/
markdown.js` (local static — no CDN); `index.html` and the new viewer
page both load it via relative `<script src>`. If a unit test
inspects the renderer inside `app.js`, update it to inspect
`markdown.js` (rendering behavior must not change).
3. **Viewer page** `frontend/document.html` (+ small
`frontend/assets/document.js`): read `source`/`path` query params,
fetch the endpoint, render:
- Header: back link (SVG arrow + "Sources"; `history.length > 1` →
`history.back()`, else navigate to `/sources.html`), `#doc-title`,
meta row (`#doc-meta`): source badge, `.format-badge` (mono), mono
path, indexed date, chunk count.
- Content `#doc-content`: `md`/`markdown` → shared renderer into a
≤46rem centered column; any other format → escaped text in
`<pre class="doc-raw">` (mono, `overflow-x: auto`, full width).
- 404 → `#doc-not-found` card (no emoji) + "Open Sources" link.
- A11y: landmarks (`<header>`/`<main>`), skip link, focus moved to
main on load, `aria-live="polite"` around the load→content swap,
visible labels, `:focus-visible` ring, Phase-08 tokens (all pairs
already ≥4.5:1).
4. **Chat chips** (`frontend/assets/app.js`):
`chip.href = "/document.html?source=" + encodeURIComponent(s.source) +
"&path=" + encodeURIComponent(s.path)`; `target="_blank"
rel="noopener"`; keep the existing title/aria-label truncation logic.
5. **Sources table** (`frontend/assets/sources.js`): path cell →
`<a class="doc-link">` to the same URL, `target="_blank"
rel="noopener"`, keep the `title` full-path attribute; style:
`--brand-ink`, underline on hover/focus.
6. **No-CDN integration test:** extend the existing local-asset test to
cover `/document.html` (Phase 07 extended it to `/sources.html` —
same pattern).
7. **README:** UI section — cited documents open in the browser.
## Testing & Quality
- **Unit:** viewer URL builder (query-encoding of paths containing
spaces/slashes); 404 mapping; format-from-suffix (incl. `.markdown`
and no-suffix fallback).
- **Integration:** content endpoint 200 (all fields, seeded doc) / 404
(unknown path; traversal-style `path=../../etc/passwd` → 404, no leak);
no-CDN on `/document.html`; renderer extraction keeps the existing
frontend tests green.
- **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` —
**>90%** on `app/`.
- **No regressions:** existing story E2E suites (02–08) green in
isolation (story 03's chip test asserts chip presence/text, not the
href — verify; the new href behavior is covered by this story's suite).
## Playwright Execution Phase
Run ONLY this story's suite:
```bash
uv run pytest tests/e2e/test_document_viewer.py -v --no-cov
```
Implements the story mapping: source chip opens a **new tab** viewer with
title/content/format badge; Sources row link opens the viewer (yaml in a
`pre`); md `<script>` fixture renders escaped (no execution); unknown doc
→ not-found state + Sources link; dark theme + all assets local.
## Success criteria
- [ ] chip click → new tab → full document (any format)
- [ ] Sources table path links work
- [ ] XSS-safe rendering (escaped) proven in E2E
- [ ] 404 state designed, no console crash
- [ ] unit + integration green, coverage >90%, ruff + pyright green
- [ ] existing story E2E suites green in isolation
- [ ] committed
## Commit
```bash
git add -A && git commit --no-gpg-sign -m "feat(ui): clickable document viewer — open any cited document in the browser from chat chips and the sources table"
```
@@ -0,0 +1,72 @@
# Phase 106 — Fix SSE chat stream DB connection pinning (SEC-14-04)
**Source:** `.agents/VULNS.md` — SEC-14-04 (Medium, open): "In-flight SSE chat stream pins a DB connection for the whole turn → pool-exhaustion DoS (~15 streams, one token user)" (CWE-770/400).
**Story:** n/a (security remediation).
**Context:** `app/api/chat.py` uses `db: Session = Depends(get_db)` which holds one DB connection for the entire SSE stream lifetime (embedding → retrieval → agent loop with tool calls → query_log write). The agent loop (`app/rag/agent.py::run_agent`) receives the same session and uses it for every tool round (ls/read/grep DB lookups). With the default implicit pool of 5+10=15, a single slow user can exhaust all connections by opening 15+ SSE streams. The sync/upload/git_sources house pattern (`SessionLocal()` → do work → `close()`) is the proven short-lived session model already used throughout the codebase.
## Objective
Eliminate the SSE-stream DB-connection pinning that causes pool-exhaustion DoS (SEC-14-04): (1) make pool parameters explicit via env vars, (2) refactor `run_agent` and the chat endpoint to use short-lived DB sessions per DB step instead of one long-lived session, and (3) add an optional concurrency cap so the pool is never saturated even if individual steps take time. After this phase, the app remains functional under concurrent load and the pool is never exhausted by slow streams.
## Dependencies
- `99_kb_tree_table_and_back_nav` (todo) — pipeline predecessor (execution order) only; no code dependency (this phase touches `app/db.py`, `app/api/chat.py`, `app/rag/agent.py`, `app/config.py`, `app/schemas.py` — none of which phase 99's files reach; its suites must stay green unchanged).
## Design (shared by all tasks — the executor reads this, not the chat)
### Pool configuration (A1, task 01)
- `app/config.py`: add `db_pool_size: int = Field(default=5)` and `db_pool_max_overflow: int = Field(default=10)`. Add `db_pool_recycle: int = Field(default=3600)` (one hour, prevents stale connections). The `__post_init__` validator ensures `db_pool_size >= 1` and `db_pool_max_overflow >= 0`.
- `app/db.py`: pass `pool_size=settings.db_pool_size`, `max_overflow=settings.db_pool_max_overflow`, `pool_recycle=settings.db_pool_recycle` to `create_engine()`. The existing `pool_pre_ping=True` and `future=True` stay.
- Env vars: `BOR_DB_POOL_SIZE`, `BOR_DB_POOL_MAX_OVERFLOW`, `BOR_DB_POOL_RECYCLE`.
### Short-lived sessions in run_agent (A2, task 02)
- **The core problem:** `run_agent(llm, db, ...)` receives a `Session` from the chat endpoint and holds it for the entire agent loop (all rounds, all tool calls). Every tool call (`ls`, `read`, `grep`) executes SQL on this session, and the session is never closed until the stream ends.
- **The fix:** Change `run_agent` to accept a session factory (`Callable[[], Session]`) instead of a `Session`. For each DB operation (tool execution), create a short-lived session, execute the operation, close the session. The LLM chat history (messages list) is already in-memory and needs no DB.
- **Signature change:** `run_agent(llm, db_factory, ..., *, max_rounds, ...)` where `db_factory = lambda: SessionLocal()`. All internal DB accessors (`ls_top`, `ls_folder`, `find_document`, `all_documents`, `find_path_candidates`) already take `Session` — they are called inside the factory closure.
- **The chat endpoint:** remove `db: Session = Depends(get_db)`. Create `db_factory = lambda: SessionLocal()` at the top of the `stream()` generator. Pass `db_factory` to `run_agent`. For the retrieval steps (`load_steering_notes`, `load_kb_overview`, `retrieve`), create short-lived sessions inline (the existing pattern).
- **query_log write:** create a short-lived session, add + commit + close (the existing pattern in sync).
- **AgentHolder:** unchanged — it only tracks in-memory state (`read_docs`, `tool_calls`, `scaffold_stripped`).
### Chat concurrency cap (A3, task 03)
- `app/api/chat.py`: a module-level `asyncio.Semaphore` initialized to `max(1, settings.chat_max_concurrent)` (default 10). The `chat` endpoint acquires the semaphore before starting the stream and releases it when the stream ends (in a `finally` block). If the semaphore is exhausted, return 503 "Too many concurrent chat turns — try again."
- `app/config.py`: add `chat_max_concurrent: int = Field(default=10)`. Validator: `>= 1`.
- Env var: `BOR_CHAT_MAX_CONCURRENT`.
### NOT touched
- `app/rag/llm.py` — LLM client unchanged.
- `app/rag/retriever.py` — retrieval functions unchanged (they take `Session` as before).
- `app/rag/prompts.py` — prompt building unchanged.
- `app/rag/scaffolding.py` — scaffolding filter unchanged.
- `app/rag/suggestions.py` — suggestions unchanged.
- `app/models.py` — no model changes.
- `alembic/` — no migrations needed.
- Frontend — no UI changes.
- Completed phase E2E suites — behavior is preserved (same API contract, same SSE frames).
## Tasks
1. `01_pool_config_db.py` — explicit pool kwargs in `app/db.py` via `app/config.py` settings + tests.
2. `02_short_lived_sessions_agent.py` — refactor `run_agent` to use a session factory + short-lived sessions per DB step + chat endpoint DB refactor + integration tests.
3. `03_chat_concurrency_cap.py` — `asyncio.Semaphore` concurrency cap on `/api/chat` + config + integration tests + E2E.
## Testing & Quality
- Unit — `tests/unit/test_db_pool_config.py` (new, task 01): `create_engine` receives the correct pool kwargs from settings; `SessionLocal` is still callable. `tests/unit/test_agent_short_lived_sessions.py` (new, task 02): `run_agent` with a mock session factory — verify that each tool call creates a new session (the factory is called per-DB-operation), sessions are closed after use, and the agent loop completes correctly.
- Integration — `tests/integration/test_chat_db_sessions.py` (new, task 02): end-to-end chat turn (both deflected and grounded paths) — verify DB sessions are created and closed per step (not held across the stream); a grounded turn with tool calls uses separate sessions per tool round; query_log is written correctly. `tests/integration/test_chat_concurrency.py` (new, task 03): concurrent chat requests — verify the semaphore limits concurrent turns; excess requests get 503; released slots are reused.
- E2E (mandatory, A16) — `tests/e2e/test_chat_db_pool.py` (task 03), run in isolation with the DB up: `uv run pytest tests/e2e/test_chat_db_pool.py -v --no-cov`.
- Regression: all existing `test_chat*.py` and `test_agent*.py` suites stay green (behavior preserved).
- Coverage: **>90%** on `app/` (the validate.sh gate).
## Completion Criteria
- [ ] `app/db.py::create_engine` receives explicit `pool_size`, `max_overflow`, `pool_recycle` from settings; default values match the previous implicit behavior (5+10).
- [ ] `run_agent` accepts a session factory (not a `Session`) and creates short-lived sessions for each DB operation; the chat endpoint creates `db_factory = lambda: SessionLocal()` and passes it.
- [ ] A grounded chat turn with tool calls: each tool round uses a separate DB session that is closed after the tool result is produced — no session is held across rounds.
- [ ] A concurrency cap (`BOR_CHAT_MAX_CONCURRENT`, default 10) limits concurrent `/api/chat` turns; excess requests get 503.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run pytest tests/e2e/test_chat_db_pool.py -v --no-cov` green in isolation (DB up); regression suites green; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **A1 — pool defaults match previous implicit behavior (owner-confirmed 2026-09-14).** `pool_size=5`, `max_overflow=10`, `pool_recycle=3600`. These are the SQLAlchemy defaults (5+10) plus a 1-hour recycle to prevent stale connections. The operator can override via env vars.
- **A2 — short-lived sessions per DB step, not per round (owner-confirmed 2026-09-14).** Each individual DB operation (one tool call, one retrieval query, one steering-notes load) gets its own session. This is the sync/upload house pattern already used throughout the codebase.
- **A3 — concurrency cap is configurable, defaults to 10 (owner-confirmed 2026-09-14).** The cap prevents pool exhaustion even if individual steps take longer than expected. The operator can increase it via `BOR_CHAT_MAX_CONCURRENT`.
## Commit
```bash
git add app/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(rag): eliminate SSE chat stream DB connection pinning — short-lived sessions, explicit pool config, concurrency cap (SEC-14-04)"
```
@@ -0,0 +1,56 @@
# Task 01 — Explicit pool kwargs in `app/db.py` via settings
**Phase:** `106_fix_sse_db_pool_exhaustion` · **Source:** SEC-14-04 (Medium, open): pool-exhaustion DoS via SSE stream DB connection pinning.
## Objective
Make the database connection pool parameters explicit and configurable via environment variables, replacing the implicit SQLAlchemy defaults. This is a safe, additive change — no behavior change, just explicit configuration with the same defaults.
## Work
1. `app/config.py` — add three new settings fields to the `Settings` dataclass (after the existing DB-related fields, near `database_url`):
```python
#: Connection pool size for the primary Postgres engine (SEC-14-04).
#: Default 5 — matches SQLAlchemy's built-in default.
db_pool_size: int = Field(default=5)
#: Maximum overflow connections beyond pool_size (SEC-14-04).
#: Default 10 — matches SQLAlchemy's built-in default.
db_pool_max_overflow: int = Field(default=10)
#: Seconds before a pooled connection is recycled (SEC-14-04).
#: Default 3600 (1 hour) — prevents stale connections.
db_pool_recycle: int = Field(default=3600)
```
Add a `__post_init__` validator (or use Field validators) to ensure `db_pool_size >= 1` and `db_pool_max_overflow >= 0`. Raise `ValueError` with a descriptive message if violated.
2. `app/db.py` — update the `create_engine()` call to pass the pool kwargs:
```python
settings = get_settings()
engine = create_engine(
settings.database_url,
pool_pre_ping=True,
future=True,
pool_size=settings.db_pool_size,
max_overflow=settings.db_pool_max_overflow,
pool_recycle=settings.db_pool_recycle,
)
```
(The `get_settings()` call was already happening indirectly via `get_settings().database_url`; make it explicit by assigning to a variable first.)
3. `app/config.py` — update the `.env.example` documentation (add the three new `BOR_DB_POOL_*` vars with their defaults and a comment about SEC-14-04).
4. Tests — `tests/unit/test_db_pool_config.py` (NEW):
- Default values: `Settings().db_pool_size == 5`, `db_pool_max_overflow == 10`, `db_pool_recycle == 3600`.
- Custom values: `Settings(db_pool_size=10, db_pool_max_overflow=20, db_pool_recycle=1800)` round-trips correctly.
- Validator: `db_pool_size=0` raises `ValueError`; `db_pool_max_overflow=-1` raises `ValueError`.
- Engine kwargs: `create_engine()` is called with the correct pool parameters (verify by inspecting the engine's pool configuration or by mocking `create_engine` and checking the call args).
5. Run `uv run pytest tests/unit/test_db_pool_config.py -v --no-cov && uv run pytest tests/unit/ -q` — green.
## Testing & Quality
- Unit: `tests/unit/test_db_pool_config.py` — defaults, custom values, validators, engine kwargs.
- Coverage: **>90%** on new/modified code.
## Completion Criteria
- [ ] `app/config.py` has `db_pool_size`, `db_pool_max_overflow`, `db_pool_recycle` with correct defaults and validators
- [ ] `app/db.py::create_engine` receives explicit pool kwargs
- [ ] `tests/unit/test_db_pool_config.py` passes (defaults, custom values, validators)
- [ ] `uv run ruff check . && uv run pyright` clean
- [ ] no behavior change in existing tests
@@ -0,0 +1,90 @@
# Task 02 — Refactor `run_agent` and chat endpoint to short-lived sessions
**Phase:** `106_fix_sse_db_pool_exhaustion` · **Source:** SEC-14-04 (Medium, open): SSE stream pins DB connection for the whole turn.
## Objective
Replace the long-lived DB session (held via `Depends(get_db)` across the entire SSE stream) with short-lived sessions per DB operation. This eliminates the pool-exhaustion vulnerability by ensuring no DB connection is held longer than a single SQL operation.
## Work
1. `app/rag/agent.py` — refactor `run_agent` to accept a session factory instead of a `Session`:
- Change the signature: replace `db: Session` with `db_factory: Callable[[], Session]`.
- The `db_factory` is a callable that returns a new session (e.g., `lambda: SessionLocal()`).
- In the agent loop, for each tool call that needs DB access, create a short-lived session:
```python
with db_factory() as tool_db:
result = execute_tool_call(tool_name, arguments, tool_db, ...)
```
- The `execute_tool_call` (or the inline tool dispatch) receives `tool_db` (a short-lived session), executes the tool's DB operations, and returns the result. The session is closed when the `with` block exits.
- The agent loop's message history (messages list) is in-memory and needs no DB — unchanged.
- `AgentHolder` is unchanged — it only tracks in-memory state.
- Update the module docstring to reflect the new signature.
2. `app/api/chat.py` — refactor the `chat` endpoint:
- Remove `db: Session = Depends(get_db)` from the function signature.
- Inside the `stream()` generator, create the session factory:
```python
from app.db import SessionLocal
db_factory = lambda: SessionLocal()
```
- For retrieval steps (`load_steering_notes`, `load_kb_overview`, `retrieve`), replace the direct `db` usage with short-lived sessions:
```python
with SessionLocal() as step_db:
steering_notes = load_steering_notes(step_db)
with SessionLocal() as step_db:
kb_overview = load_kb_overview(step_db)
with SessionLocal() as step_db:
chunks = retrieve(step_db, request.message, question_vec)
```
(Note: these can be separate sessions because they are independent reads. If they need to be in the same transaction, use one session — but they are all reads, so separate is fine and safer.)
- Pass `db_factory` to `run_agent` instead of `db`.
- For the `query_log` write (step 4), use a short-lived session:
```python
with SessionLocal() as log_db:
log_db.add(QueryLog(...))
log_db.commit()
```
- Update the module docstring to reflect the short-lived session pattern.
3. `app/rag/agent.py` — update all internal DB accessor calls inside the tool dispatch to use the session passed from the caller (which is now a short-lived session, not the long-lived one):
- The tool dispatch (inline in `run_agent` or in helper functions) receives the tool name and arguments, creates a session via `db_factory()`, calls the accessor, and closes the session.
- Example pattern for `read` tool:
```python
if tool_name == "read":
with db_factory() as tool_db:
doc = find_document(tool_db, source, path)
if doc:
result = doc.content[:settings.read_max_chars]
holder.read_docs.append(doc)
else:
result = _no_document_refusal(tool_db, combined)
```
- Same pattern for `ls` and `grep` tools.
- For `grep` on the whole KB (unscoped), use `all_documents(db_factory())` — one short-lived session for the bulk read.
4. Tests — `tests/unit/test_agent_short_lived_sessions.py` (NEW):
- Mock `db_factory` to track calls: verify that `db_factory()` is called for each tool execution (not just once at the start).
- Verify that sessions returned by `db_factory` are closed after use (use a mock that tracks `close()` calls).
- Verify that the agent loop completes correctly with a mock LLM and a mock DB factory.
- Test the deflected path (no tools, no DB factory usage beyond retrieval).
5. Tests — `tests/integration/test_chat_db_sessions.py` (NEW):
- Deflected turn: verify that retrieval uses short-lived sessions (steering notes, kb overview, retrieve each get their own session).
- Grounded turn with tool calls: verify that each tool round uses a separate session; sessions are closed after each tool result.
- Query log write: verify the query_log row is created correctly with a short-lived session.
- DB failure mid-stream: verify the error path works correctly with short-lived sessions.
6. Run `uv run pytest tests/unit/test_agent_short_lived_sessions.py tests/integration/test_chat_db_sessions.py -v --no-cov` — green.
## Testing & Quality
- Unit: `tests/unit/test_agent_short_lived_sessions.py` — verify db_factory is called per-operation, sessions are closed, agent loop completes.
- Integration: `tests/integration/test_chat_db_sessions.py` — deflected and grounded turns use short-lived sessions; query_log writes correctly.
- Coverage: **>90%** on new/modified code.
## Completion Criteria
- [ ] `run_agent` accepts `db_factory: Callable[[], Session]` instead of `db: Session`
- [ ] Each tool call in the agent loop creates its own short-lived session via `db_factory()` and closes it after the tool result is produced
- [ ] The chat endpoint no longer uses `Depends(get_db)`; retrieval steps and query_log write use short-lived sessions
- [ ] `tests/unit/test_agent_short_lived_sessions.py` passes
- [ ] `tests/integration/test_chat_db_sessions.py` passes
- [ ] `uv run ruff check . && uv run pyright` clean
- [ ] no behavior change in existing tests
@@ -0,0 +1,194 @@
# Task 03 — Chat concurrency cap + E2E test suite
**Phase:** `106_fix_sse_db_pool_exhaustion` · **Source:** SEC-14-04 (Medium, open): pool-exhaustion DoS via SSE stream DB connection pinning.
## Objective
Add a configurable concurrency cap on `/api/chat` to prevent pool exhaustion even if individual DB steps take longer than expected. Also write the dedicated E2E test suite and run all regression gates.
## Work
1. `app/config.py` — add the concurrency cap setting:
```python
#: Maximum concurrent chat turns allowed (SEC-14-04).
#: Default 10 — prevents pool saturation from too many simultaneous streams.
chat_max_concurrent: int = Field(default=10)
```
Add a validator: `chat_max_concurrent >= 1`, raise `ValueError` otherwise.
2. `app/api/chat.py` — add the semaphore:
- Module-level import: `import asyncio`.
- Module-level variable (initialized lazily to avoid import-time side effects):
```python
_chat_semaphore: asyncio.Semaphore | None = None
def _get_chat_semaphore() -> asyncio.Semaphore:
global _chat_semaphore
if _chat_semaphore is None:
settings = get_settings()
_chat_semaphore = asyncio.Semaphore(max(1, settings.chat_max_concurrent))
return _chat_semaphore
```
- In the `chat` endpoint (before the `stream()` generator definition), acquire the semaphore:
```python
sem = _get_chat_semaphore()
```
- Inside the `stream()` generator, wrap the entire body in a semaphore acquire/release:
```python
async def stream() -> AsyncIterator[str]:
await sem.acquire()
try:
# ... existing stream body ...
finally:
sem.release()
```
- If the semaphore cannot be acquired immediately (all slots taken), return 503 before entering the generator:
```python
# At the top of the chat endpoint, before defining stream():
try:
sem = _get_chat_semaphore()
# We can't do a non-blocking acquire in a sync function, so use a different approach:
# Check current semaphore value vs max, or use a try/except pattern
except Exception:
...
```
Actually, since `chat()` is an `async def`, we can do a non-blocking acquire:
```python
sem = _get_chat_semaphore()
try:
await asyncio.wait_for(sem.acquire(), timeout=0.001) # non-blocking check
except asyncio.TimeoutError:
return JSONResponse(status_code=503, content={"detail": "Too many concurrent chat turns — try again."})
```
Wait — this is racy (another request could slip in between the check and the actual acquire). Better approach: always acquire (blocking), but check if we're at the limit before starting:
Actually, the cleanest approach for FastAPI async endpoints:
```python
sem = _get_chat_semaphore()
async def stream() -> AsyncIterator[str]:
await sem.acquire()
try:
# ... existing stream body ...
finally:
sem.release()
```
And at the top of the `chat` function (before `stream()` is defined), add a pre-check:
```python
# Pre-check: if the semaphore is fully occupied, reject immediately
# (This is a best-effort check; the semaphore inside stream() is the real gate.)
if sem._value == 0:
return JSONResponse(status_code=503, content={"detail": "Too many concurrent chat turns — try again."})
```
Hmm, `_value` is implementation-specific. Let me use a cleaner approach: use a counter instead of a semaphore for the pre-check, or just always acquire and let it block (the stream will start when a slot opens). Actually, the simplest correct approach:
```python
sem = _get_chat_semaphore()
async def stream() -> AsyncIterator[str]:
await sem.acquire()
try:
# ... existing stream body ...
finally:
sem.release()
return StreamingResponse(stream(), media_type="text/event-stream", headers=SSE_HEADERS)
```
This is correct: the semaphore blocks until a slot is available. The pre-check for 503 is optional — if we want to reject immediately, we can use `sem.acquire(blocking=False)` in a try/except:
```python
# At the top of chat(), after sem = _get_chat_semaphore():
try:
sem.acquire(blocking=False) # non-blocking
except asyncio.InvalidStateError:
# Semaphore not ready yet (shouldn't happen, but be safe)
pass
else:
# Successfully acquired — we need to release it because stream() will acquire again
sem.release()
# Now let stream() acquire it properly
```
Actually this is getting complicated. Let me use the simplest correct approach: always acquire in the stream, and add a separate counter for the pre-check:
```python
_chat_semaphore: asyncio.Semaphore | None = None
_chat_active: int = 0 # thread-safe counter for pre-check
def _get_chat_semaphore() -> asyncio.Semaphore:
...
@router.post("/chat")
async def chat(...):
sem = _get_chat_semaphore()
max_concurrent = get_settings().chat_max_concurrent
# Pre-check: reject if we're already at capacity
if _chat_active >= max_concurrent:
return JSONResponse(
status_code=503,
content={"detail": "Too many concurrent chat turns — try again."}
)
async def stream() -> AsyncIterator[str]:
nonlocal _chat_active
_chat_active += 1
try:
await sem.acquire()
try:
# ... existing stream body ...
finally:
sem.release()
finally:
_chat_active -= 1
return StreamingResponse(stream(), media_type="text/event-stream", headers=SSE_HEADERS)
```
This is clean: `_chat_active` is the pre-check counter (fast path), `sem` is the actual gate (ensures we never exceed the limit even under race conditions). The counter is incremented before the semaphore acquire and decremented in the outer `finally`.
3. `app/config.py` — update `.env.example` to document `BOR_CHAT_MAX_CONCURRENT`.
4. Tests — `tests/integration/test_chat_concurrency.py` (NEW):
- Two concurrent requests: both succeed (within the cap).
- N+1 concurrent requests where N = `chat_max_concurrent`: N succeed, 1 gets 503.
- After the first N complete, the (N+1)th request succeeds (slot freed).
- Use `httpx.AsyncClient` with `anyio` or `pytest-asyncio` for concurrency.
5. Tests — `tests/e2e/test_chat_db_pool.py` (NEW, Playwright E2E):
- Open multiple concurrent browser pages, each sending a chat request.
- Verify that at most `chat_max_concurrent` requests are active simultaneously.
- Verify that excess requests get a 503 response (or wait and eventually succeed).
- Use the mock LLM (`E2E_REAL_LLM=1` not set) for determinism.
- Run in isolation: `uv run pytest tests/e2e/test_chat_db_pool.py -v --no-cov`.
6. Regression — run the full existing test suite:
- `uv run pytest tests/unit/ -q` — green.
- `uv run pytest tests/integration/ -q` — green.
- `uv run pytest tests/e2e/test_chat*.py tests/e2e/test_agent*.py -v --no-cov` — green in isolation.
7. Run the full gate:
```bash
uv run pytest --cov=app --cov-report=term-missing
uv run ruff check . && uv run pyright
```
All green.
## Testing & Quality
- Integration: `tests/integration/test_chat_concurrency.py` — concurrent requests, 503 on excess, slot reuse.
- E2E: `tests/e2e/test_chat_db_pool.py` — Playwright suite for concurrency cap verification.
- Regression: all existing chat and agent test suites stay green.
- Coverage: **>90%** on `app/` (the validate.sh gate).
## Completion Criteria
- [ ] `app/config.py` has `chat_max_concurrent` with default 10 and validator `>= 1`
- [ ] `/api/chat` rejects with 503 when `chat_max_concurrent` turns are active
- [ ] Released slots are reused — a waiting request starts when a slot frees up
- [ ] `tests/integration/test_chat_concurrency.py` passes
- [ ] `tests/e2e/test_chat_db_pool.py` passes in isolation (DB up)
- [ ] All regression suites green
- [ ] `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
- [ ] `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,42 @@
# Phase 111 — Chat error banner: a real Retry button (TODO L1)
**Source:** `TODO.md` L3–21 — "L1 — Chat error banner: 'Try again' is plain text, not a button (2026-09-15, brain-of-reese interactive test)"
**Story:** n/a (interactive-test follow-up fix; extends the phase-06 loading-feedback and phase-49/53 retry assets).
**Context:** `frontend/index.html:112` renders `#kb-banner` (chat page only — the Sources/document pages do not render it) with `#kb-banner-text`; `frontend/assets/app.js` — `showErrorBanner(detail)` (L2084) writes `${detail} ${ERROR_HINT}` as **plain text**; `ERROR_HINT` (L362) begins "Try again — …" so "Try again" reads as a clickable action but is not. `retryLastTurn(wrap)` (L2131) re-asks the last question in place (phase 49); `#stale-regenerate` (index.html:134, handler at app.js ~L1898) is the existing banner-button → `retryLastTurn` pattern.
## Objective
Give the chat-view error banner a real Retry control after a failed/dropped turn: the banner shows a visible Retry button that re-runs the last question without re-typing (reusing the phase-49 redo-in-place and the stale-banner button pattern). The banner text stops mimicking a button, and every existing `showErrorBanner` caller (share, save-doc, stale chat) keeps working text-only — the Retry button appears only on failed chat turns.
## Dependencies
- `110_fix_sse_db_pool_exhaustion` (complete) — pipeline predecessor (execution order) only; no code dependency (this phase touches `frontend/index.html`, `frontend/assets/app.js`, `frontend/assets/styles.css`, and frontend unit tests).
## Design (shared by all tasks — the executor reads this, not the chat)
- **Banner button (task 01):** add `<button type="button" class="banner-retry" id="banner-retry" hidden>` inside `#kb-banner` (after `#kb-banner-text`), mirroring the `#stale-regenerate` markup (same refresh SVG + visible "Retry" label). Hidden by default; `showErrorBanner(detail, opts)` gains an optional second arg — when the caller flags the error as a **failed chat turn** (the UI state-machine path at app.js:1281 `if (state === UI_STATE.error) showErrorBanner(errorDetail)`), the button is revealed and wired to `retryLastTurn(lastBrainWrap)` — the same last-brain-bubble targeting the `#stale-regenerate` handler uses. No retryable brain bubble → no button.
- **Copy:** `ERROR_HINT` becomes "If this persists, check the LLM is reachable." — the "Try again —" prefix moves to the button (the text must no longer read as a fake control).
- **Non-turn callers** (share failures L1857/L1870/L1892, save-doc L735/L742, stale L1620/L1950/L1962, …) pass no opts → text-only banner, no button — no behavior change for them.
- **CSS:** `.banner-retry` in `styles.css` reuses the `.stale-regenerate` pill look (same component family); the banner keeps `role="alert"`.
- **NOT touched:** the per-answer Retry pill (phase 49), the stale banner, `retryLastTurn` itself, the server, and the RAG/document views (no `#kb-banner` there — no split needed, ASSUMPTION in task 01).
## Tasks
1. `01_banner_retry_button.md` — banner Retry button markup + handler + hint-copy fix.
2. `02_banner_retry_tests.md` — frontend unit tests for the button's presence/handler + stale hint-copy assertions updated.
## Testing & Quality
- Unit: `tests/unit/test_frontend_banner_retry.py` (new, task 02) — house-style source assertions: `#banner-retry` exists in the `#kb-banner` markup (hidden by default, `type="button"`); `showErrorBanner` wires the click → `retryLastTurn`; the button is revealed only on the turn-error path; `clearErrorBanner` re-hides it; `ERROR_HINT` no longer starts with "Try again".
- E2E: no new file — the turn-error path is exercised by the existing `tests/e2e/test_llm_retry.py` and `tests/e2e/test_smoke.py` suites, which must stay green (no banner behavior change for non-turn callers).
- Coverage: **>90%** on `app/` (validate.sh gate; the frontend JS is pinned by the source-assertion unit tests — no app/ code changes in this phase).
## Completion Criteria
- [ ] After a dropped/failed chat turn, `#kb-banner` shows a visible Retry button; clicking it re-runs the last question without re-typing.
- [ ] Share/save-doc/stale-chat errors show a text-only banner (no button) — unchanged.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **A1 — Retry = re-ask the last question in place via the existing `retryLastTurn` (owner-confirmed 2026-09-14, roadmap confirmation).** No new retry mechanism; the phase-49 redo-in-place is reused.
- **A2 — the button is offered only on the UI state-machine's turn-error path (a dropped/failed chat turn); all other banner callers stay text-only (owner-confirmed 2026-09-14).** Matches L1's acceptance ("after a dropped/failed turn …").
## Commit
```bash
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(ui): give the chat error banner a real Retry button that re-asks the last question"
```
@@ -0,0 +1,34 @@
# Task 01 — Banner Retry button: markup, handler, hint-copy fix
**Phase:** `111_chat_banner_retry` · **Source:** `TODO.md:3–18` — "L1 — Chat error banner: 'Try again' is plain text, not a button … `showErrorBanner()` (`frontend/assets/app.js`, `ERROR_HINT` ~line 362) renders `${detail} ${ERROR_HINT}` as **plain text** into `#kb-banner-text` … Suggested fix: give the chat-view error banner a real Retry control that re-asks the last question (the stale-chat banner already has the pattern: `#stale-regenerate` → `retryLastTurn`; the phase-49 Retry pill asset exists). Keep `#kb-banner` dual-use working for the RAG view, or split the two banners if the RAG view's banner has different recovery semantics."
## Objective
The chat error banner gets a real Retry button (turn failures only) that re-asks the last question through the existing `retryLastTurn` redo-in-place; the hint copy stops mimicking the button.
## Work
1. `frontend/index.html` — inside `#kb-banner` (L112), add after `<span id="kb-banner-text">`:
```html
<button type="button" class="banner-retry" id="banner-retry" hidden>
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M21 12a9 9 0 1 1-9-9c2.52 0 4.93 1 6.74 2.74L21 8"/><path d="M21 3v5h-5"/></svg>
<span>Retry</span>
</button>
```
(the same refresh SVG the `#stale-regenerate` button at L134 uses). Add a comment block: revealed only for failed chat turns (task 01 of this phase); hidden for every other banner caller.
2. `frontend/assets/app.js`:
- `ERROR_HINT` (L362): change to `"If this persists, check the LLM is reachable."` (the action moves to the button).
- `showErrorBanner(detail, opts = {})` (L2084): keep the single-arg behavior byte-identical; when `opts.retryable` is true AND a retryable last brain bubble exists (the same lastBrainWrap lookup the `#stale-regenerate` handler at ~L1898 uses), unhide `#banner-retry` and bind its click **once** to `() => retryLastTurn(lastBrainWrap)`; re-binding on every reveal must be guarded (one listener per button lifetime). `clearErrorBanner()` re-hides the button.
- The turn-error path (L1281 `if (state === UI_STATE.error) showErrorBanner(errorDetail)`): pass `{ retryable: true }`. Every other caller (L735, L742, L1620, L1857, L1870, L1892, L1950, L1962, …) is left unchanged.
- If `retryLastTurn` would no-op (no retryable bubble), do not reveal the button — reveal only when a bubble exists.
3. `frontend/assets/styles.css` — `.banner-retry`: same pill treatment as `.stale-regenerate` (color, border, hover, `focus-visible` ring per the theme), laid out inline after the banner text (the `.kb-banner` flex row + gap already handles spacing).
4. ASSUMPTION: no banner split — `#kb-banner` exists only in `frontend/index.html` (the chat page); `document.html`/the Sources pages do not render it, so "keep dual-use" is trivially satisfied and the RAG view is untouched.
5. ASSUMPTION: the button is revealed only on the UI state-machine's turn-error path (a dropped/failed chat turn) — non-turn errors (share, save-doc, stale chat) stay text-only (locked A2).
## Testing & Quality
- Unit: `tests/unit/test_frontend_banner_retry.py` (added by task 02 — this task ships the code, task 02 ships the pin).
- Coverage: n/a (frontend) — the validate.sh `app/` gate must stay green.
## Completion Criteria
- [ ] `#kb-banner` contains `#banner-retry` (hidden by default); a failed chat turn reveals it; clicking re-asks the last question without re-typing.
- [ ] `ERROR_HINT` no longer contains "Try again".
- [ ] No non-turn call site passes `retryable` (grep the call sites).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,23 @@
# Task 02 — Unit tests for the banner Retry button
**Phase:** `111_chat_banner_retry` · **Source:** `TODO.md:19–21` — "Acceptance: after a dropped/failed turn, the banner shows a visible Retry button that re-runs the last question without re-typing; unit test for the banner's button presence/handler in the frontend test suite."
## Objective
Pin the banner contract in the frontend unit suite: the button's presence, its handler, its reveal condition, and the new hint copy.
## Work
1. `tests/unit/test_frontend_banner_retry.py` (new) — house-style source assertions (pattern: `tests/unit/test_frontend_feedback.py`):
- `frontend/index.html`: `#banner-retry` exists inside the `#kb-banner` block, `hidden` by default, `type="button"`, with a visible "Retry" label.
- `frontend/assets/app.js`: `showErrorBanner` binds the click handler to `retryLastTurn`; the `UI_STATE.error` turn path passes the retryable flag; `clearErrorBanner` re-hides the button.
- `ERROR_HINT` does not start with "Try again".
2. Grep the whole `tests/` tree for the old hint copy (`Try again — if this persists`) and update any stale assertion (loading-feedback and banner-related frontend tests).
3. Run the full unit + integration suite — no regressions (in particular `tests/unit/test_frontend_*.py`).
## Testing & Quality
- Unit: the new file above (≥4 assertions across markup/handler/reveal/copy).
- Coverage: **>90%** on `app/` unchanged (no app/ code touched by this phase).
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_frontend_banner_retry.py -v` green.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] No test anywhere asserts the old "Try again — …" hint.
@@ -0,0 +1,46 @@
# Phase 112 — Honesty gate: FTS hits need cosine corroboration (TODO L2a/b + README nit)
**Source:** `TODO.md` L23–86 — "L2 — Deflection test: non-KB question got answered parametrically + unrelated source chips (2026-09-15, brain-of-reese interactive test)" — part a (L36–44), part b (L46–55) + the stochastic follow-up (L64–75), the README nit (L77–81), acceptance (L83–86). Part c (chips on non-grounded answers, L57–62) is delivered by phase 113.
**Story:** the completed story 04 (`04_story_honest_deflection`) — this phase fixes the A8 gate eagerness the live deflection test exposed.
**Context:** `app/api/chat.py::plan_turn` (L272–277): HIGH when `best_cosine >= settings.relevance_threshold (0.62) or fts_hits > 0` — a single weak FTS token hit (suspected: the token "capital" inside a quest file) promoted a non-KB question into grounded mode and injected two irrelevant docs into the HIGH prompt, which then tempted the model into a parametric answer. The follow-up one-tap re-run of the identical question produced a **clean, textbook deflection** with the same docs injected → HONESTY GATE compliance is stochastic across runs; the deterministic lever is the gate (don't inject irrelevant docs), not prompt copy alone.
## Objective
Make the A8 honesty gate deterministic against weak lexical hits: an FTS hit flips the turn to HIGH (grounded) only when the vector signal corroborates it (best cosine clears a new `lexical_support_floor`); below the floor the turn stays LOW (deflected) even with FTS matches. The owner-confirmed locked-prompt contract decision (treat the stochastic disclosed-general-knowledge behavior as acceptable — documented, prompt text byte-identical) and the stale README deflection copy are fixed in the same phase.
## Dependencies
- `111_chat_banner_retry` (todo) — pipeline predecessor (execution order) only; no code dependency.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Gate rule (task 01):** HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`). LOW otherwise — including the fts>0 / cosine<floor case (the Mongolia case). `lexical_support_floor` is a new setting (default 0.35, env `BOR_LEXICAL_SUPPORT_FLOOR`), validated `0 <= floor <= relevance_threshold`. Rationale: a real lexical match on a genuinely similar doc (cosine ≥ floor) still grounds; a single weak token match with vector-unsupported docs no longer promotes. `plan_turn`'s docstring (the A8 bullets) and the A8 entry in `.agents/PLAN.md` are updated with the revision note (house precedent: "A8 revised 2026-08-21" — an owner-confirmed change to a LOCKED decision is recorded in the plan, not silently deviated from).
- **No schema/API change:** `query_log.top_score` / `fts_hits` are recorded exactly as today (observability unchanged); `TurnPlan` shape unchanged; the LOW branch (deflect prompt, weak-hit titles, derived suggestions) unchanged.
- **Prompt contract (task 03):** owner decision (iii) — the model's stochastic disclosed-general-knowledge answer (when misleading docs are injected) is documented as acceptable; the `app/rag/prompts.py` module docstring (the house location for locked-prompt revision history, e.g. the 2026-08-22 note) gains a dated entry; the prompt strings stay byte-identical (LOCKED verbatim); the README deflection section notes the behavior.
- **README nit (task 04):** the quoted deflection opening *"I haven't done anything like that"* was removed in the 2026-08-22 locked-prompt revision — the two README spots (L11, L575) are updated to describe the current behavior (admit no notes + 2–3 concrete alternative questions).
## Tasks
1. `01_gate_meaningful_fts.md` — the `plan_turn` gate fix + `lexical_support_floor` setting + the A8 plan revision note.
2. `02_gate_tests.md` — unit pins for the HIGH/LOW quadrants on weak single-token FTS hits + the E2E deflection check.
3. `03_prompt_contract_documentation.md` — document the owner decision (iii) (prompts.py docstring + README); prompt text unchanged.
4. `04_readme_deflection_copy.md` — README stale deflection copy (L11, L575) updated.
## Testing & Quality
- Unit: `tests/unit/test_chat_gate.py` (existing — extend) + a new quadrant file (task 02): cosine ≥ threshold → HIGH regardless of FTS; fts>0 + cosine ≥ floor → HIGH; **fts>0 + cosine < floor → LOW** (the new behavior, the Mongolia regression pin); no hits → LOW.
- E2E: `tests/e2e/test_honest_deflection.py` (existing — extend, task 02): a known-out-of-KB question (mock LLM, so the test pins the gate not the model) → deflected, no citation chips, 2–3 alternative questions. Run in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov`.
- Regression: `tests/unit/test_chat_gate.py`, `tests/e2e/test_chat_rag.py`, `tests/e2e/test_retrieval_quality.py` stay green.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] A weak single-token FTS hit with vector-unsupported docs (cosine < floor) → LOW/deflected (unit-pinned).
- [ ] A known-out-of-KB question produces no false citations and 2–3 concrete alternative questions (E2E).
- [ ] `app/rag/prompts.py` prompt strings byte-identical to pre-phase (test-pinned); README deflection copy matches current behavior.
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — lever: cosine corroboration; `lexical_support_floor` default 0.35, env-tunable via `BOR_LEXICAL_SUPPORT_FLOOR` (owner-confirmed 2026-09-14, roadmap confirmation).** Of the TODO's three options (rank threshold / stopword-short-token exclusion / cosine corroboration), cosine corroboration is the deterministic one; the tests pin the decision logic, not the default value.
- **A2 — locked-prompt contract: option (iii) — treat the stochastic disclosed-general-knowledge behavior as acceptable and document it (owner-confirmed 2026-09-14, roadmap confirmation).** The prompt text is unchanged (LOCKED verbatim); the deterministic protection is the gate (A1). Options (i) tighten copy / (ii) amend the prompt via the plan remain open to a future owner decision.
- **A3 — the A8 entry in `.agents/PLAN.md` is amended in this phase with a dated revision note** (owner-confirmed 2026-09-14) — house precedent for recording owner-confirmed LOCKED-decision changes.
## Commit
```bash
git add app/ tests/ .agents/ README.md && git commit --no-gpg-sign -m "fix(rag): require cosine-corroborated FTS hits before the honesty gate flips HIGH — document the disclosed-answer behavior, refresh README deflection copy"
```
@@ -0,0 +1,30 @@
# Task 01 — Gate: FTS hits must be cosine-corroborated
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:36–44` — "a) **Honesty gate too eager** (`app/api/chat.py` L272–277, A8 revised 2026-08-21): HIGH/grounded when `best_cosine >= threshold` OR `fts_hits > 0`. A single weak FTS token hit (suspected: the token 'capital' inside a quest file) promotes a non-KB question into grounded mode and injects two irrelevant top-docs into the HIGH prompt — which then tempts the model into a parametric answer instead of deflection. Consider: require FTS hits to be *meaningful* (e.g. rank threshold, stopword/short-token exclusion, or cosine corroboration) before flipping to HIGH; or run the LOW prompt when top-docs score below a usefulness bar." (+ the follow-up, L64–75: "the deterministic lever is the gate (don't inject irrelevant docs — part a), not prompt copy alone")
## Objective
An FTS hit flips `plan_turn` to HIGH only when the best cosine clears the new `lexical_support_floor`; a weak single-token hit with vector-unsupported docs stays LOW (deflected).
## Work
1. `app/config.py` — add `lexical_support_floor: float = Field(default=0.35)` (env `BOR_LEXICAL_SUPPORT_FLOOR`) next to `relevance_threshold` (L119), with validation `0 <= lexical_support_floor <= relevance_threshold` (mirror the file's existing validator style); document it in `.env.example`.
2. `app/api/chat.py::plan_turn` — replace the gate line (`if best_cosine >= settings.relevance_threshold or fts_hits > 0:`):
```python
lexical_supported = fts_hits > 0 and best_cosine >= settings.lexical_support_floor
if best_cosine >= settings.relevance_threshold or lexical_supported:
...
```
Update the `plan_turn` docstring's gate section (the A8 bullets): HIGH when `best_cosine >= threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); LOW otherwise — a lexical-only hit without vector support deflects (A8 revised 2026-09-14).
3. `.agents/PLAN.md` — amend the A8 locked-decision entry with a dated revision note (house precedent: "A8 revised 2026-08-21"): "A8 revised 2026-09-14 (owner-confirmed, TODO L2a): an FTS hit flips HIGH only when `best_cosine >= lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`); lexical-only hits without vector support deflect."
4. `app/rag/prompts.py` — prompt strings NOT touched (LOCKED verbatim).
5. ASSUMPTION: the chosen lever is cosine corroboration (one of the TODO's three listed options); floor default 0.35 (≈ half the 0.62 threshold) is tunable via env against the live KB — the unit/E2E tests pin the decision logic, not the default value (locked A1).
## Testing & Quality
- Unit: covered by task 02 (`tests/unit/test_chat_gate.py` extension) — this task's code must keep it green.
- Coverage: **>90%** on `app/` including the new setting and the modified gate line.
## Completion Criteria
- [ ] `plan_turn`: fts_hits>0 + best_cosine < floor → LOW (`TurnPlan.deflected` True, LOW prompt, suggestions derived from weak-hit titles).
- [ ] fts_hits>0 + floor <= best_cosine < threshold → HIGH (the new "corroborated lexical" path).
- [ ] best_cosine >= threshold → HIGH regardless of FTS (unchanged); no hits → LOW (unchanged).
- [ ] `.env.example` documents `BOR_LEXICAL_SUPPORT_FLOOR`; the config validation rejects floor > threshold.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,27 @@
# Task 02 — Pin the gate's HIGH/LOW decision on weak FTS hits
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:83–86` — "Acceptance: a known-out-of-KB question (LLM-known, e.g. capitals, sports results) produces no false citations, follows the HONESTY GATE (or the amended contract), and unit/E2E tests pin the gate's HIGH/LOW decision on a weak single-token FTS hit."
## Objective
Unit tests pin all four gate quadrants — especially the new fts>0 / cosine<floor → LOW quadrant (the Mongolia/"capital" regression); the E2E deflection story asserts a known-out-of-KB question deflects with no false citations and 2–3 alternatives.
## Work
1. `tests/unit/test_chat_gate.py` (existing — extend; reuse its chunk-building helpers) — the quadrants:
- cosine ≥ threshold, fts=0 → HIGH (unchanged).
- fts>0, cosine ≥ floor (e.g. 0.50 with default settings) → HIGH (corroborated lexical — the new path).
- **fts>0, cosine < floor** (e.g. one `fts_hit=True` chunk with cosine 0.10 — the "capital" case) → LOW: `deflected=True`, LOW prompt, `suggestions` non-empty, the weak docs do not enter a HIGH prompt.
- no chunks → LOW (unchanged).
- boundary: cosine exactly at the floor → HIGH (`>=`, mirroring the threshold's convention); config with floor > threshold → validation error.
2. `tests/e2e/test_honest_deflection.py` (existing — extend): a test asking a known-out-of-KB question (e.g. "What is the capital of Mongolia?" — LLM-known, absent from the fixture KB; the mock LLM's deflection path keeps the test deterministic on the gate, not on model compliance) → the done frame is `deflected: true`, `sources` is empty (no false citations), `suggestions` has 2–3 items.
3. Run the E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` (DB up).
4. Regression: `tests/unit/test_chat_gate.py` (all), `tests/e2e/test_chat_rag.py`, `tests/e2e/test_retrieval_quality.py` green.
## Testing & Quality
- Unit: the quadrant table (the new quadrant is the regression pin for TODO L2a).
- E2E: the deflection story extension.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] All quadrants green, including fts>0 + cosine<floor → LOW.
- [ ] E2E: known-out-of-KB question → deflected, zero source chips, 2–3 alternatives.
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,21 @@
# Task 03 — Document the disclosed-answer contract (owner decision iii)
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:46–55, 64–75` — "b) **Model violates the locked prompt** (Rule 1 'Answer ONLY from the provided document context', Rule 3 HONESTY GATE): with irrelevant docs injected it answered from general knowledge. The disclosure is better UX than silence, but the contract says no pretending to know + 2-3 concrete alternative questions — decide whether to (i) tighten the prompt copy … (ii) amend the locked prompt via the plan to explicitly permit disclosed general-knowledge answers, or (iii) treat the observed behavior as acceptable and document it. Owner decision required — the prompt text is locked verbatim (change through the plan, not here)." + the follow-up: "HONESTY GATE compliance is **stochastic** across runs (run 1: parametric answer; run 2: perfect deflection). Implication for the fix direction: the deterministic lever is the gate …, not prompt copy alone; a small local model cannot be relied on to obey Rules 1/3 100% when handed misleading context."
## Objective
Record the owner-confirmed decision (iii): the stochastic disclosed-general-knowledge answer (when misleading docs are injected) is acceptable and documented — the prompt text stays byte-identical (LOCKED verbatim); the deterministic protection is the task-01 gate fix.
## Work
1. `app/rag/prompts.py` — module docstring (the house location for the locked-prompt revision history, e.g. the 2026-08-22 note): add a dated entry recording the 2026-09-15 interactive-test finding (parametric "Ulaanbaatar" answer with a disclosure, on the Mongolia question with two irrelevant docs injected; the clean textbook deflection on the identical one-tap re-run — stochastic compliance) and the owner decision (2026-09-14, roadmap confirmation): treat the disclosed general-knowledge answer as acceptable; the deterministic lever is the gate (A8 revised, task 01); options (i)/(ii) remain open to a future plan amendment. Prompt strings: byte-identical.
2. `README.md` — in the deflection paragraph (the one task 04 rewrites), one sentence: with a small local model, a rare turn may answer from general knowledge with an explicit disclosure when retrieval was borderline — the gate (phase 112) minimizes this; the disclosure is surfaced, never silent.
3. Prompt-lock pin: if no existing test byte-pins the prompt text, add a small `tests/unit/test_prompt_lock.py` asserting the HIGH/LOW prompt constants against pre-phase anchor strings (the executor extracts the pre-phase values when writing the test — e.g. exact prefix/suffix + total length, so any byte change fails).
4. ASSUMPTION: the documentation lives in the prompts.py docstring (the existing revision-history location) + the README — no new docs file.
## Testing & Quality
- Unit: the prompt-lock pin (work item 3).
- Coverage: n/a (docs-only change) — the suite stays green.
## Completion Criteria
- [ ] `app/rag/prompts.py` docstring carries the dated decision entry; the prompt strings are byte-identical (test-pinned).
- [ ] The README deflection section notes the rare disclosed-answer behavior.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,20 @@
# Task 04 — README: fix the stale deflection copy
**Phase:** `112_honesty_gate_weak_hits` · **Source:** `TODO.md:77–81` — "Plus a docs nit: the README still promises the exact deflection copy *'I haven't done anything like that'* — the mandated deflection opening was removed in the 2026-08-22 locked-prompt revision (`app/rag/prompts.py` module docstring). Update the README's 'If it doesn't have notes…' paragraph to match current behavior."
## Objective
The README describes the current deflection behavior (admit no notes + 2–3 concrete alternative questions) instead of the removed mandated opening.
## Work
1. `README.md` L11 ("If it doesn't have notes for your question, it admits it: *'I haven't done…*") — rewrite the quoted copy to match the current locked-prompt behavior: it admits it has no notes on that and offers 2–3 concrete alternative questions about things it DOES have notes on. No exact-copy promise (the opening is no longer mandated).
2. `README.md` L575 ("**Honest deflection** (the amber *'I haven't done anything like that'*…") — same update; keep the amber-banner description accurate.
3. Grep the README for any other occurrence of the old quoted opening and update it.
## Testing & Quality
- Docs-only: no code tests. Grep `tests/` for `haven't done anything` — if any test asserts the old copy, update it to the new behavior.
- Coverage: n/a.
## Completion Criteria
- [ ] `grep -rn "haven't done anything" README.md` → no hits (the quoted opening is gone).
- [ ] The deflection paragraph matches current behavior (admit + 2–3 concrete alternatives).
- [ ] `uv run pytest` green.
@@ -0,0 +1,46 @@
# Phase 113 — Source chip quality: usefulness bar + related-docs tier (TODO L5 + L2c)
**Source:** `TODO.md` L101–147 — "L5 — Recurring weak-hit source chips: the 2nd chip is often noise the answer never used (2026-09-15, brain-of-reese interactive test)" — root-cause chain (L125–129), suggested directions (L131–142), acceptance (L144–146) — plus `TODO.md` L57–62 (L2 part c): "c) **Misleading chips on non-grounded answers** (`chat.py:252` — `done.sources` = weak hits when deflected; by design, but visually a citation). … At minimum: never render them as answer citations."
**Story:** n/a (interactive-test follow-up fix; extends the phase-09 retrieval-quality and phase-05 chip assets).
**Context:** `top_n_docs = 2` (`app/config.py:114`) forces two docs into `plan.docs`; `done.sources` (`app/api/chat.py` ~L796, `ChatDoneEvent.sources`) carries every entry and `appendSources` (`frontend/assets/app.js:1369`) chips them all with identical visual weight — "the answer used this" vs "this also scored" is indistinguishable. Deflected turns carry weak hits in `sources` "by design" but they render as citations. Owner-confirmed direction: the deterministic **usefulness bar** (server-side) + the **visual split** (UI) — cite-gated chips (parsing the model's cited paths) are rejected for now (stochastic with a small model).
## Objective
A document earns a citation slot only when its retrieval signal is vector-corroborated (or the agent explicitly read it via a tool call); everything else that scored is demoted to a clearly secondary "nearby docs" row that never reads as a citation. For a single-document question the turn shows exactly one citation chip; a deflected turn shows no citation chips at all.
## Dependencies
- `112_honesty_gate_weak_hits` (todo) — the gate fix stops weak hits being injected into the HIGH prompt; this phase stops weak docs earning a `done.sources` slot. Same workstream, ordered after the gate.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Usefulness bar (task 01):** new settings `source_usefulness_floor: float = 0.35` (env `BOR_SOURCE_USEFULNESS_FLOOR`, validated `0 <= floor <= relevance_threshold`, mirroring phase 112's floor) and `related_max_docs: int = 2` (env `BOR_RELATED_MAX_DOCS`, validated `>= 0`). In `plan_turn`, retrieval docs are tiered: **cited** = distinct parent docs (best fused-score order, at most `top_n_docs`) whose best hit-chunk **cosine** clears the floor; **related** = the next scored distinct docs (at most `related_max_docs`) that did not clear it. Agent-read docs (`holder.read_docs`, the phase-37 agent tool reads) always stay cited — the model read them via tool calls, so they were used by definition. `TurnPlan` gains `related_docs: list[Document] = []`; the tiering is a new `select_documents_tiered(chunks, n, floor, related_cap) -> tuple[list[Document], list[Document]]` in `app/rag/retriever.py`, with `select_documents` becoming a thin wrapper (legacy behavior byte-identical for existing callers/tests). `query_log.sources` is unchanged (it records retrieval, not citations — locked A3).
- **Done frame (task 01):** `ChatDoneEvent` (`app/schemas.py`) gains `related: list[SourceRef] = []` — additive; old clients ignore unknown fields (house contract, PLAN §4). Built from `plan.related_docs` with the same (source, path) dedupe against the cited list as `cited_docs` already does.
- **UI split (task 02):** `frontend/assets/app.js` — new `appendRelated(wrap, related)` renders a `.msg-meta.related-docs` row under the bubble (only when `related` is non-empty): a small de-emphasized label "Nearby docs, in case:" + one link per doc with the class `related-doc` (NOT `source-chip`) — same `documentUrl(...)` href and left-click → `openDocumentModal` behavior as citation chips, visually secondary (reduced opacity/size/dashed border via theme variables; link contrast ≥4.5:1, WCAG 2.1 AA). The done-frame handler (~L2421) also calls `appendRelated(wrap, ev.related)`; the restored-chat path (~L1561) likewise when the stored payload carries `related` (pre-phase chats don't — graceful). Deflected turns: `ev.sources` is empty (the server change) → no chips; the weak hits arrive in `ev.related` → the row only.
- **NOT touched:** the citation-chip component (`.source-chip` / `appendSources`) for the cited tier; the suggestion chips; `top_n_docs` (ceiling, not quota); the phase-37 agent-read dedupe; the Sources/RAG pages.
## Tasks
1. `01_usefulness_bar_sources.md` — retriever tiering + `TurnPlan.related_docs` + `ChatDoneEvent.related` + the two settings.
2. `02_secondary_related_docs_ui.md` — the related-docs row in the chat UI (app.js + styles.css); deflected turns show no chips.
3. `03_chip_filter_tests.md` — unit pins on the four observed live shapes + E2E chip-count assertions.
## Testing & Quality
- Unit: `tests/unit/test_retriever.py` (extend — the tiering table), `tests/unit/test_source_chip_quality.py` (new, task 03 — the four observed shapes), the done-frame schema tests (`related` defaults `[]`; old payloads without the field still parse).
- E2E: `tests/e2e/test_source_chip_quality.py` (new, task 03; run in isolation: `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov`) — a known single-source question → exactly one citation chip; a deflected question → zero `.source-chip` elements (the row, if any, is `.related-doc`, never `.source-chip`).
- Regression: `tests/e2e/test_retrieval_quality.py`, `test_honest_deflection.py`, `test_chat_rag.py`, `test_sources_midstream_bug.py` stay green.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] For a single-document question, the turn shows one citation chip (E2E).
- [ ] A weak 2nd doc renders only in the de-emphasized related row, never as a `.source-chip` (unit + E2E).
- [ ] A deflected turn renders zero citation chips (the weak hits, if any, live in the related row).
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — direction: usefulness bar (server, deterministic) + visual split (UI); cite-gated chips rejected for now (owner-confirmed 2026-09-14, roadmap confirmation).**
- **A2 — citation slot = vector-corroborated retrieval doc (best hit-chunk cosine >= `source_usefulness_floor`, default 0.35, env-tunable) OR agent-read doc; `top_n_docs` stays a ceiling, not a quota (owner-confirmed 2026-09-14).**
- **A3 — `query_log.sources` keeps recording the full retrieval (observability); `done.sources` records only the cited tier (owner-confirmed 2026-09-14).**
- **A4 — the related tier is capped at 2 docs (`related_max_docs`, env `BOR_RELATED_MAX_DOCS` — owner-confirmed 2026-09-14).**
## Commit
```bash
git add app/ tests/ frontend/ .agents/phases/ && git commit --no-gpg-sign -m "feat(rag): tier sources by a usefulness bar — weak hits become a de-emphasized related-docs row, never citation chips"
```
@@ -0,0 +1,28 @@
# Task 01 — Usefulness bar: tier done.sources into cited + related
**Phase:** `113_source_chip_quality` · **Source:** `TODO.md:125–129, 137–140` — "Root cause chain: `top_n_docs = 2` (`app/config.py` L114) forces retrieval to return two documents, `done.sources` carries both (`app/api/chat.py` L252), and the UI chips every entry without distinguishing 'the answer used this' from 'this also scored'." + "**Usefulness bar on the 2nd doc** — only include a document in `done.sources` when its fused/cosine score clears a threshold (a single weak FTS token hit should not earn a citation slot); `top_n_docs` stays a ceiling, not a quota." + `TODO.md:57–62` (L2 part c: "`done.sources` = weak hits when deflected; by design, but visually a citation … At minimum: never render them as answer citations.")
## Objective
Retrieval docs are tiered at the honesty gate: cited (vector-corroborated, ≤ `top_n_docs`) vs related (scored but under the floor, ≤ `related_max_docs`); the SSE done frame carries both; agent-read docs always stay cited.
## Work
1. `app/config.py` — add `source_usefulness_floor: float = Field(default=0.35)` (env `BOR_SOURCE_USEFULNESS_FLOOR`) and `related_max_docs: int = Field(default=2)` (env `BOR_RELATED_MAX_DOCS`); validators: `0 <= source_usefulness_floor <= relevance_threshold`, `related_max_docs >= 0`; `.env.example` entries.
2. `app/rag/retriever.py` — add `select_documents_tiered(chunks, n, floor, related_cap) -> tuple[list[Document], list[Document]]`:
- rank distinct parent docs by best fused score (the existing `select_documents` ordering), tracking each doc's best hit-chunk cosine;
- **cited** = the docs whose best-chunk cosine >= `floor`, up to `n` (ceiling — a single strong doc yields one cited doc);
- **related** = the next docs in rank order (any cosine, including 0.0 lexical-only), up to `related_cap`, never overlapping the cited list.
- `select_documents` becomes a wrapper: `cited, _ = select_documents_tiered(chunks, n, 0.0, 0)` — floor 0.0 + cap 0 keeps the legacy "any score, top-N" behavior byte-identical for existing callers/tests.
3. `app/api/chat.py` — `TurnPlan` gains `related_docs: list[Document] = []` (after `docs`); `plan_turn` calls `select_documents_tiered(chunks, settings.top_n_docs, settings.source_usefulness_floor, settings.related_max_docs)` → `docs`, `related_docs` (both the HIGH and LOW branches — deflected turns: the weak hits fall to related, cited is usually empty). The done-frame build (~L796): add `related=[SourceRef(source=d.source, path=d.path, title=d.title) for d in <plan.related_docs deduped against cited_docs by (source, path)>]` — the same dedupe pattern `cited_docs` already uses.
4. `app/schemas.py` — `ChatDoneEvent` gains `related: list[SourceRef] = []` (additive; docstring note: old clients ignore unknown fields, PLAN §4).
5. `query_log.sources` — unchanged (locked A3).
6. ASSUMPTION: the bar is on the **cosine** of the doc's best hit chunk, not the RRF fused score — the fused `score` is a rank key, not a similarity; a lexical-only hit has cosine 0.0 and is vector-unsupported by definition (consistent with the phase-112 gate; locked A2).
## Testing & Quality
- Unit: `tests/unit/test_retriever.py` (extend) — the tiering table: both clear → both cited; strong + weak → 1 cited + 1 related; both weak → 0 cited + 2 related; related cap respected; `select_documents` wrapper legacy behavior unchanged (existing tests stay green without edits).
- Unit: the existing done-frame/schema tests — `related` defaults to `[]`; a payload without the field still parses (back-compat).
- Coverage: **>90%** on `app/` including the new function.
## Completion Criteria
- [ ] `plan_turn` returns tiered docs; HIGH and LOW branches both populated correctly.
- [ ] The done frame carries `related` (≤ `related_max_docs`, deduped against cited); old frames (no field) parse.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,27 @@
# Task 02 — UI: the related-docs row (never a citation chip)
**Phase:** `113_source_chip_quality` · **Source:** `TODO.md:141–142` — "**Visual split** — keep both, but render uncited/weak docs as a clearly secondary 'related docs' row, not citation chips." + `TODO.md:59–62` — "When the answer is disclosed general knowledge (or a deflected turn), the chips should be visually de-emphasized / labeled (e.g. 'nearby docs I have, in case'), or omitted when the brain says it didn't use them. At minimum: never render them as answer citations."
## Objective
The chat bubble renders the cited tier exactly as today (`.source-chip` via `appendSources`) and the related tier as a clearly secondary labeled row; a deflected turn renders zero citation chips.
## Work
1. `frontend/assets/app.js`:
- new `appendRelated(wrap, related)` next to `appendSources` (L1369): early-return when empty; a `.msg-meta.related-docs` row (`role="list"`, `aria-label="Nearby docs, in case"`) + a small `<span class="related-docs-label">Nearby docs, in case:</span>` + one link per doc — class `related-doc` (NOT `source-chip`), the same `documentUrl(s.source, s.path, "/")` href and left-click → `openDocumentModal(s.source, s.path, link)` behavior, `title`/`aria-label` carrying the full path.
- the done-frame handler (~L2421, next to `appendSources(wrap, ev.sources)`): also `appendRelated(wrap, ev.related)`.
- the restored-chat path (~L1561): same, when the stored payload carries `related` (pre-phase saved chats don't — the row is simply absent, graceful).
- deflected turns: `ev.sources` is empty (the task-01 server change) → `appendSources` no-ops; the weak hits arrive in `ev.related` → row only.
2. `frontend/assets/styles.css` — `.related-docs` (muted row: smaller font, theme-variable color — link text contrast ≥4.5:1, WCAG 2.1 AA), `.related-doc` (dashed border, no hover elevation of the citation chips; `focus-visible` ring), `.related-docs-label` (small caps or muted small text); the row stacks below the citation `.msg-meta` row with the existing gap.
3. Frontend unit test (house source-assertion style — lives in `tests/unit/test_source_chip_quality.py`, extended by task 03): `appendRelated` exists and uses `related-doc` (assert `source-chip` is NOT in the `appendRelated` body); the row renders only when related is non-empty; the label text is present; the done handler calls `appendRelated`.
4. ASSUMPTION: label copy "Nearby docs, in case:" (the TODO's suggested wording, trimmed).
5. ASSUMPTION: pre-phase saved chats (stored payload without `related`) restore exactly as today — no related row.
## Testing & Quality
- Unit: the frontend source-assertion tests (task 03's file).
- E2E: pinned by task 03.
- Coverage: n/a (frontend) — the `app/` gate stays green.
## Completion Criteria
- [ ] A related doc renders only as `.related-doc` in the labeled row — never as `.source-chip`.
- [ ] A deflected turn (mock) renders zero `.source-chip` elements under the bubble.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,31 @@
# Task 03 — Pin the chip contract: the four observed shapes + E2E chip counts
**Phase:** `113_source_chip_quality` · **Source:** `TODO.md:110–123, 144–146` — the four observed live cases (1. "What is the capital of Mongolia?" → `Trooper_Nagraz.pl` + `Trooper_Begzei.pl`, both unrelated; 2. phase-gate question answered from `brain-of-reese/.agents/validate.sh` → second chip `ServMon/README.md` unused; 3. Trooper_Nagraz question answered from `Trooper_Nagraz.pl` → second chip `Trooper_Byzin.pl` uncited; 4. meta question about the conversation's own history → chips `app/api/suggestions.py` + `108_history_wire_check/00_phase.md`, neither used) + "Acceptance: for a single-document question, the turn shows one citation chip; a unit test pins `done.sources` filtering (or the chip renderer's cite-gate) on the four observed shapes; E2E asserts chip count for a known single-source question."
## Objective
A unit test pins the cited/related tiering on the four live shapes; the E2E suite asserts the visible chip counts.
## Work
1. `tests/unit/test_source_chip_quality.py` (new) — model each of the four observed shapes as a `plan_turn`/done-frame fixture (retrieval chunks with controlled cosine/`fts_hit`/fused `score`) and assert the tiering:
1. **both docs weak** (cosine < floor, the Mongolia case) → `sources` empty, `related` ≤ 2.
2. **one strong + one weak** (the validate.sh case: `validate.sh` cosine ≥ floor, `ServMon/README.md` below) → exactly 1 in `sources`, the weak one in `related`.
3. **the Nagraz case** — same shape, different fixtures (`Trooper_Nagraz.pl` strong, `Trooper_Byzin.pl` weak) → 1 cited, 1 related.
4. **the meta/history question** (no doc clears the floor, the agent reads nothing) → `sources` empty, `related` ≤ 2; assert the frame shape that the UI renders as row-only (the rendering is pinned by task 02's source tests + the E2E).
5. **agent-read exemption**: a doc under the floor that is in `holder.read_docs` (agent tool read) still lands in `sources` (cited).
- plus the frontend source-assertion tests from task 02 work item 3 (same file).
2. `tests/e2e/test_source_chip_quality.py` (new; conftest/mock-LLM/fixture-KB pattern per `tests/e2e/test_retrieval_quality.py`):
- a known single-source question (a fixture-KB question whose answer comes from one doc) → the done bubble has **exactly one** `.source-chip`.
- a deflected question (known-out-of-KB) → **zero** `.source-chip`; if a `.related-docs` row exists, its links are `.related-doc`, never `.source-chip`.
- if the fixture KB cannot produce a strong+weak two-tier shape, say so in the test docstring and rely on the unit table for that shape.
3. Run in isolation: `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (DB up).
4. Regression: `tests/e2e/test_retrieval_quality.py`, `test_honest_deflection.py`, `test_chat_rag.py` green.
## Testing & Quality
- Unit: the four-shape table (the acceptance pin) + the agent-read exemption.
- E2E: the chip-count assertions (the acceptance pin).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] The four observed shapes are unit-pinned (plus the agent-read exemption).
- [ ] E2E: single-source question → exactly one citation chip; deflected turn → zero.
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,45 @@
# Phase 114 — Embed question length: truncation + accurate error (TODO L6)
**Source:** `TODO.md` L149–181 — "L6 — 4,000-char question clamp exceeds the embed model's input cap → misleading 'couldn't reach the embedding model' error (2026-09-15, brain-of-reese interactive test)"
**Story:** n/a (interactive-test follow-up fix; extends the phase-67 LLM-retry and phase-06 loading-feedback assets).
**Context:** The composer clamps at 4,000 chars; `app/api/chat.py:423` embeds the **full** question via `llm.embed_one`; aipi's litellm rejects the ~903-token input with **HTTP 500**: "input (903 tokens) is too large to process. increase the physical batch size (current batch size: 512)". The single-text path in `app/rag/llm.py` (`_embed_batch` → `_TooLarge`, ~L279–284) turns that into `EmbeddingError("a single …-char chunk exceeded the endpoint's per-request input token cap — lower BOR_CHUNK_TARGET_CHARS and re-import")` — an **import-oriented** message — and the chat endpoint's catch-all (~L428–444) maps EVERY `EmbeddingError` to "I couldn't reach the embedding model — please try again." Both diagnoses are wrong (reachability is fine; the chunker constant is irrelevant to a question). The chunker's own `HARD_MAX_CHARS = 1200` (`app/rag/chunker.py:51`, ~1024 tokens at ~1.4 chars/token) shows the question path never got the same treatment.
## Objective
Every legal question (≤ the UI clamp) is embeddable: the embed step gets a bounded prefix of the question (the chunker's 1200-char budget) while the full question still reaches the LLM prompt; and if the input is still too large (a smaller-cap model, a misconfiguration), the turn fails with an accurate "question too long" error — no false reachability diagnosis, no wasted retries — and the banner carries the phase-111 Retry button.
## Dependencies
- `111_chat_banner_retry` (todo) — L6's acceptance: "the L1 'Try again' button fix should also apply to this banner" — the too-long error flows through the same turn-error state machine, so the phase-111 Retry button is offered on it.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Truncation (task 01):** new setting `embed_question_max_chars: int = 1200` (env `BOR_EMBED_QUESTION_MAX_CHARS`, default = the chunker's `HARD_MAX_CHARS` budget, validated `> 0`). The chat embed step (chat.py:423) embeds `request.message[:settings.embed_question_max_chars]`; the LLM prompt build is unchanged (the full question still reaches the model). Questions shorter than the budget are byte-identical to today.
- **Error mapping (task 02):** `app/rag/llm.py` — new `EmbeddingInputTooLargeError(EmbeddingError)` subclass; the single-text `_TooLarge` branch of `_embed_batch` raises it (same message text — the importer path is byte-identical, it still catches `EmbeddingError`). The chat endpoint catches `EmbeddingInputTooLargeError` **before** `EmbeddingError` inside the phase-67 retry loop → no retry (a deterministic failure — locked A3) → terminal `ChatErrorEvent` with `detail="Question too long — trim it and re-ask."` and a new optional `hint` field: `hint="The app reached the embedding model fine — only the question length is the problem."` `ChatErrorEvent` gains `hint: str | None = None` (additive; PLAN §4 old-client ignore contract). The frontend's phase-111 reworked `showErrorBanner(detail, opts)` shows `opts.hint` when the frame carries one, else the default `ERROR_HINT`.
- **Retry:** the too-long frame flows through the turn-error state machine → the phase-111 banner Retry button is offered (re-asking is the user's call after trimming; the composer clamp still applies).
- **NOT touched:** the importer's embed path and its batch-halving `_TooLarge` behavior/error copy, the 4,000-char composer clamp (locked A2 — truncation, not a lower clamp), the reachability-failure retry semantics (phase 67 — byte-identical).
## Tasks
1. `01_embed_truncation.md` — the bounded-prefix embed + the setting.
2. `02_too_long_error_mapping.md` — `EmbeddingInputTooLargeError`, the chat-path mapping, `ChatErrorEvent.hint`, the frontend hint support.
3. `03_embed_length_tests.md` — the unit pins + the 4,000-char E2E.
## Testing & Quality
- Unit: `tests/unit/test_embed_question_length.py` (new, task 03) — truncation (long → prefix embedded, LLM prompt carries the full text; short → byte-identical), error mapping (too-large failure → exact detail + hint, no retry frame, one attempt; transport failure → legacy reachability path with retries — the regression pin), the config validator.
- E2E: `tests/e2e/test_embed_question_length.py` (new, task 03; run in isolation: `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov`) — a 4,000-char question (the composer clamp) streams to done (mock LLM), no error banner.
- Regression: `tests/e2e/test_llm_retry.py`, `test_oneshot_llm_retry.py`, `test_chip_sizing_question_cap.py` (the 4,000-char counter) stay green.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] A 4,000-char question embeds (bounded prefix) and the turn succeeds; the LLM prompt carries the full question.
- [ ] A too-large embed failure (forced in a unit test) → the accurate "Question too long" frame + the reachability-fine hint; the banner offers the phase-111 Retry button.
- [ ] A reachability embed failure behaves byte-identically to pre-phase (retries + old copy).
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — both fixes combined: 1200-char embed truncation (default = the chunker budget, env-tunable) + the precise too-long error mapping (owner-confirmed 2026-09-14, roadmap confirmation).**
- **A2 — the 4,000-char composer clamp stays (owner-confirmed 2026-09-14) — truncation, not a lower clamp.**
- **A3 — a too-large embed failure is NOT retried (deterministic failure) — it short-circuits the phase-67 retry loop (owner-confirmed 2026-09-14).**
## Commit
```bash
git add app/ tests/ frontend/ .agents/phases/ && git commit --no-gpg-sign -m "fix(rag): embed a bounded question prefix (1200-char budget) and map the embed too-large failure to an accurate too-long error with a reachability-fine hint"
```
@@ -0,0 +1,21 @@
# Task 01 — Embed a bounded question prefix
**Phase:** `114_embed_question_length` · **Source:** `TODO.md:151–164, 168–170` — "Repro: type/paste a question to the UI maximum (the composer clamps at 4,000 chars — char counter shows '4000/4000 — character limit') and send. Result, **100% reproducible**: the turn dies pre-token with the banner 'I couldn't reach the embedding model — please try again.' … a short question embeds fine (HTTP 200), but the 4,000-char question (~903 tokens) gets **HTTP 500** from aipi … So the maximum legal question length exceeds the embed model's maximum legal input — and the chunker's own 1200-char cap (set to stay under the ~1024-token per-request cap) shows the question path never got the same treatment." + "**Truncate for embedding** — embed a bounded prefix of the question (e.g. the same 1200-char budget as chunks) while the full question still reaches the LLM prompt."
## Objective
The chat embed step embeds at most `embed_question_max_chars` (default 1200 — the chunker's `HARD_MAX_CHARS` budget) of the question; the full question still reaches the LLM prompt.
## Work
1. `app/config.py` — add `embed_question_max_chars: int = Field(default=1200)` (env `BOR_EMBED_QUESTION_MAX_CHARS`), validator `> 0`; `.env.example` entry with a comment citing the chunker rationale (`app/rag/chunker.py:30–51` — ~1.4 chars/token, stays under the ~1024-token per-request cap).
2. `app/api/chat.py` — the embed step (~L423): `question_vec = await llm.embed_one(request.message[: settings.embed_question_max_chars])`. Everything downstream is unchanged: retrieval runs on the prefix vector (intended — the prefix is the question's head); the LLM prompt build (`hist` + the full `request.message`) is untouched; the per-turn log line is untouched (`question=%r` logs the full text).
3. One-line comment at the call site: the prefix is bounded to the embed model's input cap (the chunker budget); the full question still reaches the LLM prompt (TODO L6).
4. ASSUMPTION: the budget is a setting (env-tunable), default 1200 — not a hard-coded constant — so a model with a larger/smaller cap is accommodated without a code change (locked A1).
## Testing & Quality
- Unit: `tests/unit/test_embed_question_length.py` (new, task 03) — a question > the budget → `embed_one` receives exactly the prefix (mock LLM client); the LLM request messages carry the full question; a question ≤ the budget → byte-identical call.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] A 4,000-char question → `embed_one` called with the 1200-char prefix; the LLM request carries the full 4,000-char message.
- [ ] A short question → no behavior change (byte-identical call).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,31 @@
# Task 02 — Map the too-large embed failure to an accurate error
**Phase:** `114_embed_question_length` · **Source:** `TODO.md:171–173` — "**Map the 500 to a precise error** — detect the 'too large' embed failure and surface 'question too long — trim it' (and fix the ERROR_HINT for this case: reachability is fine)." + `TODO.md:179–181` — "Acceptance: a 4,000-char question either succeeds (truncated embedding) or fails with an accurate too-long error; unit test pins the error mapping; the L1 'Try again' button fix should also apply to this banner."
## Objective
A deterministic "input too large" embed failure surfaces as a precise "question too long" terminal error with a hint that reachability is fine — no false reachability diagnosis, no wasted retries; the error banner carries the phase-111 Retry button (the turn-error path).
## Work
1. `app/rag/llm.py` — add `class EmbeddingInputTooLargeError(EmbeddingError)` (near `EmbeddingError`, L42), with a docstring: the single-text input exceeded the endpoint's token cap — deterministic, not a reachability failure. In the single-text `_TooLarge` branch of `_embed_batch` (~L279–284): raise `EmbeddingInputTooLargeError(<the existing import-oriented message>)` instead of plain `EmbeddingError` — the message text is **identical** (the importer path is byte-identical; it still catches `EmbeddingError`, and the subclass is a drop-in).
2. `app/schemas.py` — `ChatErrorEvent` gains `hint: str | None = None` (additive; docstring: the client shows the hint in place of its default reachability hint when present; old clients ignore the field — PLAN §4).
3. `app/api/chat.py` — the embed-failure handling (~L428–444, inside the phase-67 retry `while` loop): catch `EmbeddingInputTooLargeError` **before** `EmbeddingError` → do NOT restart (locked A3 — deterministic) → `settled = True`, log an error line (the existing format plus a `too-large` marker), and yield:
```python
ChatErrorEvent(
detail="Question too long — trim it and re-ask.",
hint="The app reached the embedding model fine — only the question length is the problem.",
).model_dump()
```
The existing `EmbeddingError` branch (reachability) is unchanged, including the retry semantics and the old copy.
4. `frontend/assets/app.js` — the phase-111 reworked `showErrorBanner(detail, opts)`: honor `opts.hint` — `bannerText.textContent = detail ? \`${detail} ${opts.hint ?? ERROR_HINT}\` : (opts.hint ?? ERROR_HINT)`. The SSE error-frame handler in the stream state machine (~L1281): pass `{ retryable: true, hint: ev.hint }` when the frame carries a hint.
5. ASSUMPTION: detail copy "Question too long — trim it and re-ask." (the TODO's "question too long — trim it", phrased as a banner sentence); hint copy as in work item 3.
6. ASSUMPTION: no retry on too-large (locked A3) — the phase-67 retry loop is for transient failures; a size failure is guaranteed to repeat.
## Testing & Quality
- Unit: `tests/unit/test_embed_question_length.py` (new, task 03) — force the `_TooLarge` branch (a fake httpx response: HTTP 500 + a "too large to process" body) → the chat SSE stream yields exactly one error frame with the precise detail + hint and **no** retry frame; a transport failure (no "too large" signature) → the legacy reachability path with the retry loop and old copy (the regression pin).
- Coverage: **>90%** on `app/` including the new exception class and branch.
## Completion Criteria
- [ ] A too-large embed failure → the frame `{type: "error", detail: "Question too long — trim it and re-ask.", hint: "…fine…"}` — no "couldn't reach" copy, no retry frame.
- [ ] A reachability embed failure → byte-identical to pre-phase (retries + old copy).
- [ ] The frontend shows the frame's hint when present; the banner offers the phase-111 Retry button on this error.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,25 @@
# Task 03 — Unit + E2E pins for the 4,000-char question
**Phase:** `114_embed_question_length` · **Source:** `TODO.md:179–181` — "Acceptance: a 4,000-char question either succeeds (truncated embedding) or fails with an accurate too-long error; unit test pins the error mapping; the L1 'Try again' button fix should also apply to this banner."
## Objective
The acceptance is pinned: a full-clamp (4,000-char) question succeeds end-to-end (truncated embed), and the too-long mapping is unit-pinned.
## Work
1. `tests/unit/test_embed_question_length.py` (new):
- **truncation:** a mock `LLMClient` records the `embed_one` input; a 4,000-char question → exactly the prefix (default budget); the chat request to the LLM carries the full question; a 100-char question → byte-identical call.
- **error mapping:** a fake embed transport returning HTTP 500 + "too large to process" body for the input → the chat SSE stream yields exactly one error frame with the precise detail + the reachability-fine hint and no retry frame; short input + a 500 WITHOUT the "too large" signature → the legacy reachability path (retry frames + old copy) — the regression pin.
- **config:** the `embed_question_max_chars` default (1200) and validator.
2. `tests/e2e/test_embed_question_length.py` (new; the `tests/e2e/` conftest + mock-LLM pattern): type a 4,000-char question into the composer (the counter shows "4000/4000") → send → the turn streams to done (mock LLM) — no error banner.
3. Run in isolation: `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov` (DB up).
4. Regression: `tests/e2e/test_llm_retry.py`, `test_oneshot_llm_retry.py`, `test_chip_sizing_question_cap.py` (the 4,000-char counter) stay green.
## Testing & Quality
- Unit: as above (the acceptance pin: the error mapping).
- E2E: the 4,000-char success path (the acceptance pin: the truncated embed).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] A 4,000-char question → a successful turn (E2E); the embed input was the prefix (unit).
- [ ] The too-long mapping is unit-pinned (exact frame, no retry).
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,45 @@
# Phase 115 — Doc drafts: Discard + DELETE route + title fix (TODO L7)
**Source:** `TODO.md` L183–209 — "L7 — 'Save as doc' has no Discard: orphan drafts are invisible and un-deletable (2026-09-15, brain-of-reese interactive test)"
**Story:** n/a (interactive-test follow-up fix; extends the phase-59/75 doc-draft and save-as-doc assets).
**Context:** "Save as doc" (`frontend/assets/app.js::saveAsDoc`, L720) POSTs `/api/doc-drafts` (201) → `/doc-edit.html?draft=<token>`. The edit screen offers exactly one action: **"Push to docs branch"**. The drafts router (`app/api/doc_drafts.py`) has POST/GET/PUT/POST-push only — no DELETE, no TTL/pruning; a draft created by an accidental click (or a tester) sits orphaned in the DB forever — invisible (no UI lists drafts) and only consumable by actually pushing a doc to the repo. Side observation: the draft's default title comes from `defaultDocTitle()` (app.js:619) — the **last user record** in the conversation — and after a Retry redo-in-place (phase 49) the redone answer sits at the end, so its title came from an unrelated trailing question (a junk 4,000-char test question, not the question the answer answered). The body (full session transcript) is correct by design; only the title derivation mismatches.
## Objective
An orphaned doc draft can be discarded from the edit screen (a new admin-gated `DELETE /api/doc-drafts/{token}` + a Discard control), and the draft's default title is the question the saved answer actually answered (its paired user record) — fixing the retry-redo mismatch.
## Dependencies
- `114_embed_question_length` (todo) — pipeline predecessor (execution order) only; no code dependency.
## Design (shared by all tasks — the executor reads this, not the chat)
- **DELETE route (task 01):** `DELETE /api/doc-drafts/{token}` → **204**. The whole router already sits behind `require_admin` (phase 59, `dependencies=[Depends(require_admin)]` at L63) — the new route inherits it; the uuid4 token is the screen's credential (same trust model as GET/PUT/push). Unknown token → 404 via the existing `_get_draft_or_404` helper. No migration (a row delete); no push-side state (the git push happens only on push).
- **Discard UI (task 02):** the doc-edit screen — a "Discard draft" control next to "Push to docs branch" (secondary/danger treatment per the theme). `frontend/assets/doc-edit.js`: `confirm()` (destructive + irreversible — no undo exists), `DELETE /api/doc-drafts/${token}` (the same token the screen already uses for GET/PUT) → 204 → `location.assign("/")` (back to the chat page). Non-204 → the page's existing inline-error pattern, no navigation.
- **Title fix (task 03):** `defaultDocTitle(wrap)` — takes the saved brain bubble's wrap (the `.save-as-doc-btn`'s bubble); the title is the text of the user bubble **paired** with that brain bubble (the nearest preceding user message in the DOM conversation flow), falling back to the current last-user-record-in-`conversation` logic when no wrap is given or no paired user bubble is found (first-turn edge / DOM mismatch). The `DOC_TITLE_MAX` slice + "Note" fallback are unchanged.
- **NOT touched:** the push flow (byte-identical), the draft body (`buildSessionTranscript` — full session, correct by design), the drafts schema (no field change), no TTL/pruning (locked A1 — out of scope).
## Tasks
1. `01_delete_draft_route.md` — `DELETE /api/doc-drafts/{token}` (204 / 404 / admin-gated).
2. `02_discard_ui.md` — the Discard control on the doc-edit screen wired to the route.
3. `03_draft_title_fix.md` — the title from the answer's own question (the paired user record).
4. `04_draft_discard_tests.md` — the API + frontend + E2E pins.
## Testing & Quality
- Integration: `tests/integration/test_doc_drafts_api.py` (existing — extend, task 04): DELETE removes the row (204; subsequent GET 404); unknown token → 404; the admin gate applies (same assertions the sibling routes use).
- Unit (frontend, house source-assertion style, task 04): the Discard control's presence/handler (confirm → DELETE → 204 → redirect; non-204 → inline error, no navigation); the `defaultDocTitle` pairing logic + the call site.
- E2E: `tests/e2e/test_save_doc_session.py` (existing — extend, task 04): the discard flow (save → edit screen → discard → confirm → back on the chat, draft gone); a save-as-doc after a Retry redo-in-place → the title matches the redone answer's own question. Run in isolation: `uv run pytest tests/e2e/test_save_doc_session.py -v --no-cov`.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] An orphaned draft can be discarded from the edit screen; the draft row is gone afterward (integration).
- [ ] The title of a save-as-doc after a retry redo matches the redone answer's own question (E2E).
- [ ] The push flow is byte-identical (regression green).
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — scope: the Discard control + the DELETE route + the title fix; NO TTL/pruning (owner-confirmed 2026-09-14, roadmap confirmation — the discard covers the acceptance; pruning can be a future phase).**
- **A2 — the DELETE route is admin-gated by the router-level `require_admin` (phase 59) with the uuid4 token as the credential — same trust model as the sibling routes (owner-confirmed 2026-09-14).**
## Commit
```bash
git add app/ tests/ frontend/ .agents/phases/ && git commit --no-gpg-sign -m "feat(docs): discard doc drafts from the edit screen (DELETE /api/doc-drafts/{token}) + derive the draft title from the answer's own question"
```
@@ -0,0 +1,37 @@
# Task 01 — DELETE /api/doc-drafts/{token}
**Phase:** `115_doc_draft_discard` · **Source:** `TODO.md:185–199, 201–204` — "Clicking 'Save as doc' on an answer POSTs `/api/doc-drafts` (201) and navigates to `/doc-edit.html?draft=<token>`. The edit screen offers exactly one action: **'Push to docs branch'**. There is no Discard/cancel control, the drafts API has no DELETE route (`app/api/doc_drafts.py`: POST, GET, PUT, POST /push only) and no TTL/pruning. A draft created by an accidental click (or a tester) sits orphaned in the DB forever — invisible (no UI lists drafts) and only consumable by actually pushing a doc to the repo." + "Suggested fix: add a Discard control to `/doc-edit.html` wired to a new `DELETE /api/doc-drafts/{token}` (admin-gated, token = the screen's credential) …"
## Objective
A new admin-gated `DELETE /api/doc-drafts/{token}` removes a draft row (204) so an orphaned draft can be discarded instead of only pushed.
## Work
1. `app/api/doc_drafts.py` — add (after the `update_draft` PUT route, before the push route — or at the file's route-order idiom):
```python
@router.delete("/{token}", status_code=status.HTTP_204_NO_CONTENT)
def delete_draft(
token: uuid.UUID,
db: Session = Depends(get_db), # noqa: B008
) -> None:
"""Discard a draft (the edit screen's Discard control, phase 115).
Admin-gated like the whole router (phase 59); the uuid4 token is
the screen's credential — after a successful discard, GET/PUT/push
all 404.
"""
row = _get_draft_or_404(db, token)
db.delete(row)
db.commit()
```
Match the file's existing import style (check whether `status` from `fastapi` is already imported; use the file's idiom for the 204 response).
2. No migration, no schema change, no other route touched.
3. ASSUMPTION: 204 No Content (no body) — the token is a one-way credential; nothing else references the row (no FK targets, no push-side state).
## Testing & Quality
- Integration: extended in task 04 (`tests/integration/test_doc_drafts_api.py`).
- Coverage: **>90%** on `app/` including the new route.
## Completion Criteria
- [ ] `DELETE /api/doc-drafts/{token}` → 204; the subsequent `GET` → 404.
- [ ] Unknown token → 404; the admin gate applies exactly like the sibling routes.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,33 @@
# Task 02 — Discard control on the doc-edit screen
**Phase:** `115_doc_draft_discard` · **Source:** `TODO.md:201–204` — "Suggested fix: add a Discard control to `/doc-edit.html` wired to a new `DELETE /api/doc-drafts/{token}` (admin-gated, token = the screen's credential), or a TTL/prune for stale drafts; …" + `TODO.md:207–208` — "Acceptance: an orphaned draft can be discarded from the edit screen; the draft row is gone afterward (API test) …"
## Objective
The doc-edit screen offers a Discard control: confirm → DELETE → back to the chat.
## Work
1. The doc-edit template (find the exact HTML file — the edit screen that renders "Push to docs branch"; `frontend/doc-edit.html` or the template it uses): add next to the push control:
```html
<button type="button" id="discard-draft" class="discard-draft"
title="Delete this draft permanently — this cannot be undone">Discard draft</button>
```
visually secondary to the push button (the theme's muted/danger treatment).
2. `frontend/assets/doc-edit.js` — a handler near the push handler:
- `if (!confirm("Discard this draft? This cannot be undone.")) return;`
- `fetch(\`/api/doc-drafts/${token}\`, { method: "DELETE" })` (the same token the screen already uses for GET/PUT, per its existing load code ~L136–143);
- 204 → `location.assign("/")` (back to the chat page);
- non-204 → the page's existing inline-error pattern (message + no navigation, no crash).
3. `frontend/assets/styles.css` — `.discard-draft`: the secondary/danger button style (contrast ≥4.5:1, `focus-visible` ring per the theme), laid out next to the push button.
4. Frontend unit tests (house source-assertion style, shipped in task 04's `tests/unit/test_frontend_doc_draft_discard.py`): the button's presence; `confirm(...)` before the DELETE; 204 → redirect; non-204 → inline error, no navigation.
5. ASSUMPTION: after a successful discard the user lands on the chat page (`/`) — the draft has no other home (no drafts list exists).
6. ASSUMPTION: a native `confirm()` is acceptable for this one destructive action (the codebase has no custom dialog asset — if the executor finds one in the theme, use it instead).
## Testing & Quality
- Unit: the frontend source-assertion tests (task 04).
- E2E: the discard flow (task 04, `tests/e2e/test_save_doc_session.py`).
- Coverage: n/a (frontend) — the `app/` gate stays green.
## Completion Criteria
- [ ] The edit screen shows "Discard draft" next to the push control; the confirm dialog appears; a 204 returns the user to `/`.
- [ ] A failed DELETE (e.g. a 404 race) shows the inline error; no navigation; no crash.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,26 @@
# Task 03 — Draft title from the answer's own question
**Phase:** `115_doc_draft_discard` · **Source:** `TODO.md:193–199` — "Side observation (edge case, same test): the draft's default title is the user record *immediately preceding* the saved answer. After a Retry redo-in-place (L-see app.js `retryLastTurn`), the redone answer sits at the end of the conversation, so its save-as-doc title came from an unrelated trailing question (the doc was titled with a junk 4,000-char test question, not the question the answer answered). Body is the full session transcript (correct by design); title derivation is just the mismatch." + `TODO.md:203–204` — "consider deriving the default title from the question the answer actually answered (its paired user record) rather than the preceding record."
## Objective
The draft's default title is the user question paired with the saved brain bubble (its own question) — after a Retry redo-in-place the title matches the redone answer's question.
## Work
1. `frontend/assets/app.js` — `defaultDocTitle()` (L619) → `defaultDocTitle(wrap)`:
- when *wrap* (the brain bubble) is given, walk the DOM conversation flow backwards from *wrap* to the **nearest user message bubble** (confirm the exact user-bubble class — the `.msg` variant used for user turns) and use its text;
- fall back to the current last-user-record-in-`conversation` logic when *wrap* is absent or no paired user bubble is found (first-turn edge / DOM mismatch);
- the `DOC_TITLE_MAX` slice + whitespace collapse + "Note" fallback are unchanged.
2. `saveAsDoc(btn)` (L720) — pass the bubble: `const title = defaultDocTitle(btn.closest(<the bubble class>))` — the save button lives in the bubble's meta (the `addSaveAsDocButton` code ~L704–710 shows the exact ancestor; use the same class `lastBrainWrap` uses at L577).
3. The `docSlug`/path logic is unchanged (it derives from the title).
4. Frontend unit tests (house source-assertion style, task 04's file): `defaultDocTitle` takes a wrap arg and prefers the paired user bubble over the last conversation record; the `saveAsDoc` call site passes the bubble ancestor.
5. ASSUMPTION: the pairing is DOM-structural (nearest preceding user bubble), not index-based — the redo-in-place reorders the DOM, and the structural pair IS the answer's question by construction.
## Testing & Quality
- Unit: the frontend source-assertion tests (task 04).
- E2E: the title-after-retry pin (task 04, `tests/e2e/test_save_doc_session.py`).
- Coverage: n/a (frontend) — the `app/` gate stays green.
## Completion Criteria
- [ ] A save-as-doc on a normal (non-redone) answer → the same title as pre-phase (no regression).
- [ ] A save-as-doc after a Retry redo-in-place → the title is the redone answer's own question (E2E).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,30 @@
# Task 04 — API, frontend, and E2E pins for the discard + title
**Phase:** `115_doc_draft_discard` · **Source:** `TODO.md:207–209` — "Acceptance: an orphaned draft can be discarded from the edit screen; the draft row is gone afterward (API test); title of a save-as-doc after a retry redo matches the redone answer's own question."
## Objective
The acceptance is pinned at all three layers: the API (the row is gone), the frontend (the control + handler + the pairing), and the E2E (the full discard flow + the title after a retry redo).
## Work
1. `tests/integration/test_doc_drafts_api.py` (existing — extend):
- DELETE an existing draft → 204; the subsequent GET → 404.
- DELETE an unknown token → 404.
- the admin gate: the same assertions the sibling routes use (mirror the existing test's auth fixtures — anonymous → 401 / non-admin → 403 per the router's `require_admin`).
2. `tests/unit/test_frontend_doc_draft_discard.py` (new, house source-assertion style):
- the doc-edit template carries `#discard-draft`; `doc-edit.js` calls `confirm(...)` before the `DELETE /api/doc-drafts/` fetch; 204 → the redirect; non-204 → the inline error, no navigation.
- `app.js`: `defaultDocTitle` takes a wrap arg and prefers the paired user bubble; the `saveAsDoc` call site passes the bubble ancestor.
3. `tests/e2e/test_save_doc_session.py` (existing — extend):
- **the discard flow:** ask (mock LLM) → save as doc → the edit screen → click Discard → confirm → back on the chat page; an API check (test client) confirms the draft row is gone (GET 404).
- **the title after a retry:** ask → the answer → Retry (redo-in-place) → save as doc on the redone answer → the edit screen's title field value == the redone question (not an unrelated trailing question).
4. Run in isolation: `uv run pytest tests/e2e/test_save_doc_session.py -v --no-cov` (DB up).
5. Regression: `tests/integration/test_doc_drafts_api.py` (all), the existing `tests/e2e/test_save_doc_session.py` tests green.
## Testing & Quality
- Integration: the DELETE contract (the acceptance: "the draft row is gone afterward (API test)").
- Unit: the frontend pins.
- E2E: the flow + the title (the acceptance).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] All the acceptance pins green (API row gone; title after a retry).
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,39 @@
# Phase 116 — Document modal: themed code-block scrollbar (TODO L3)
**Source:** `TODO.md` L88–99 — "L3 — Document modal: native unstyled horizontal scrollbar in the code block (cosmetic) (2026-09-15, brain-of-reese interactive test)"
**Story:** `document-viewer.md` — the modal belongs to the document-viewer story (the same-page chip viewer, phase 26).
**Context:** The document modal (the almost-fullscreen chip viewer, `.doc-modal` — `frontend/assets/styles.css:4117+`) shows the raw content in a code block; long lines (e.g. a `quest::say(...)` line in a quest `.pl` file) overflow horizontally and reveal the **browser-native, unstyled scrollbar** (light-gray bar) — it clashes with the dark theme, and long lines clip at the right edge.
## Objective
The modal's code block scrolls horizontally with a themed scrollbar (`scrollbar-color` for Firefox, `::-webkit-scrollbar` pseudos for Chromium), consistent with the dark theme; long lines stay unwrapped (code stays code) and scroll instead of clipping.
## Dependencies
- `115_doc_draft_discard` (todo) — pipeline predecessor (execution order) only; no code dependency (CSS + one E2E file).
## Design (shared by all tasks — the executor reads this, not the chat)
- **Theme tokens (task 01):** two CSS custom properties in the theme's token block: `--scrollbar-thumb` (a muted theme color, ≥3:1 against the track — a scrollbar is a UI component, non-text contrast AA) and `--scrollbar-track` (near the code-block background). Scope: the modal's raw-content code element **only** (confirm the exact selector from `frontend/assets/document-modal.js` / the `.doc-modal` rules) — no global scrollbar restyle (out of scope; the TODO asks only about the modal).
- **Rules (task 01):** on the modal code block: `overflow-x: auto` (confirm present — add if the element relies on an ancestor), `scrollbar-width: thin`, `scrollbar-color: var(--scrollbar-thumb) var(--scrollbar-track)` (Firefox), and the Chromium pair: `::-webkit-scrollbar { height: 8px }`, `::-webkit-scrollbar-track { background: var(--scrollbar-track) }`, `::-webkit-scrollbar-thumb { background: var(--scrollbar-thumb); border-radius: 4px }`.
- **E2E (task 02):** the document-viewer story gains a check: open the modal on a document with a long line → the code block is horizontally scrollable (`scrollWidth > clientWidth`; a scroll action moves it) → a screenshot to `.agents/screenshots/` (house convention) as the visual record.
## Tasks
1. `01_themed_scrollbar.md` — the theme tokens + the scoped scrollbar rules.
2. `02_scrollbar_e2e_check.md` — the E2E scroll check + screenshot.
## Testing & Quality
- E2E: `tests/e2e/test_document_viewer.py` (existing — extend, task 02); run in isolation: `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov`.
- CSS: no unit layer for CSS — the E2E check + screenshot are the gate; the dark-theme suite (`tests/e2e/test_dark_tech_theme.py`) must stay green (no token collision).
- Coverage: n/a (no app/ code change) — the validate.sh gate stays green.
## Completion Criteria
- [ ] The modal's code block scrolls horizontally; the scrollbar is themed — the screenshot in `.agents/screenshots/` shows no native light-gray bar.
- [ ] No global scrollbar change (the new selector is scoped under `.doc-modal` — grep); the other pages are visually unchanged (regression suites green).
- [ ] `uv run pytest` green; coverage >90%; the e2e file green in isolation; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — keep horizontal scroll (NO line-wrap) for the code content (owner-confirmed 2026-09-14, roadmap confirmation — the TODO's "consider wrapping long lines per content type" is rejected for code: wrapping breaks code readability; the themed scrollbar is the fix).**
## Commit
```bash
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "style(ui): theme the document modal's code-block horizontal scrollbar (scrollbar-color + webkit pseudos)"
```
@@ -0,0 +1,23 @@
# Task 01 — Themed scrollbar for the modal code block
**Phase:** `116_modal_scrollbar_theme` · **Source:** `TODO.md:90–96` — "In the document modal (the almost-fullscreen chip viewer), the raw-content code block overflows horizontally and reveals the **browser-native, unstyled scrollbar** (light-gray bar) — it clashes with the dark theme, and long lines clip at the right edge (observed on a quest `.pl` file whose `quest::say(...)` line exceeds the modal width). Style the scrollbar to match the theme (`scrollbar-color` for Firefox, `::-webkit-scrollbar` pseudos for Chromium), or consider wrapping long lines per content type."
## Objective
The modal's raw-content code block gets a themed horizontal scrollbar (both engine families), scoped to the modal.
## Work
1. `frontend/assets/styles.css` — find the modal code block's exact selector (the `.doc-modal` raw-content `<pre>`/code element — how `frontend/assets/document-modal.js` renders the content; the `.doc-modal-panel` rules start ~L4135):
- ensure `overflow-x: auto` on the scrolling element (add it if the element relies on an ancestor for the overflow);
- add the scoped rules: `scrollbar-width: thin`, `scrollbar-color: var(--scrollbar-thumb) var(--scrollbar-track)`, and `::-webkit-scrollbar { height: 8px }` / `::-webkit-scrollbar-track { background: var(--scrollbar-track) }` / `::-webkit-scrollbar-thumb { background: var(--scrollbar-thumb); border-radius: 4px }`.
2. `frontend/assets/styles.css` — define `--scrollbar-thumb` / `--scrollbar-track` in the theme's token block (derive from existing theme colors — the thumb must be visibly distinct from the track, ≥3:1 non-text contrast).
3. Do NOT restyle scrollbars elsewhere — the new selector stays scoped under `.doc-modal` (the TODO's scope).
4. ASSUMPTION: no line-wrap (locked A1, phase level) — the code stays unwrapped and scrolls.
## Testing & Quality
- E2E: the scroll check (task 02).
- Coverage: n/a (CSS) — the suite stays green.
## Completion Criteria
- [ ] The modal code block's horizontal scrollbar is themed for Chromium (webkit pseudos) and Firefox (`scrollbar-color`) — the task-02 screenshot shows no native light-gray bar.
- [ ] No other page's scrollbar changes (the new selector is scoped under `.doc-modal`).
- [ ] `uv run pytest` green (in particular `tests/e2e/test_dark_tech_theme.py`).

Some files were not shown because too many files have changed in this diff Show More