Compare commits

..
127 Commits
Author SHA1 Message Date
ducoterra bef24e05e2 phase: 123_chat_image_questions
Build and Push Containers / build-and-push-app (push) Successful in 1m54s
Build and Push Containers / build-and-push-db (push) Failing after 13s
All gates green. Verification complete.

**Phase 123 — final verification pass (all 4 tasks already in `complete/`)**

- Verified the full implementation is in the working tree: `app/api/chat_images.py` (upload/serve pair), `ChatRequest.image`/`ChatMessage.image` (path-validated, omitted-when-None), toggle-off + stale-file hinted error frames, `build_user_content` multimodal build at both sites (chat.py deflected branch + `run_agent`), config-gated composer attach/preview/upload-then-send, restore + shared rendering, CSP `img-src 'self' data:` carve-out, mock-LLM capture buffer.
- `uv run pytest` → **2796 passed**, exit 0 (unit + integration).
- `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (29/4615 missed; phase-123 modules 99–100%).
- `uv run pytest tests/e2e/test_chat_image_questions.py -v --no-cov` → **5 passed** in isolation.
- `uv run ruff check . && uv run pyright` → clean (0 errors).

**Completion criteria:** (1) attach→send→multimodal text+image to the model, bubble/reload/shared all render it, saved chat stores the PATH with `"base64" not in json.dumps(stored)` — **verified** (E2E tests 1–4 + integration round-trip); (2) `BOR_IMAGES=false` — control hidden, exact hinted error frame, zero model calls / no query_log row — **verified** (E2E test 5 + integration); (3) text-only byte-identical (`content` stays a plain `str`) — **verified** (unit + integration); (4) all gates green — **verified**; (5) commit + phase move — left to the harness per pipeline rules (no `git add`/`commit` run).

No defects found; no live-infrastructure changes (repo + local dev DB only). **Next pending phase: none** — 123 is the last phase in `todo/`.
2026-09-25 05:19:18 -04:00
ducoterra a19d78d284 phase: 122_image_documents
Build and Push Containers / build-and-push-app (push) Successful in 1m57s
Build and Push Containers / build-and-push-db (push) Failing after 13s
**Phase 122 (image documents) — final verification pass: all green. No code changes were needed; defects found: none.**

**Verified (implementation already complete in working tree, reviewed end-to-end):**
- Toggle (`BOR_IMAGES`/`BOR_IMAGE_EXTENSIONS`/`BOR_IMAGE_DIR`, off by default) + `GET /api/config` `images` flag
- Ingest: bytes digest, `image_dir` persistent copy, `content = summary = vision description` (chat-model call; only text embedded), fail-soft skip + `images_failed` counter
- Serve/display: `/api/documents/{id}/image` route (404 matrix), viewer `<img>` + description, Sources 48px lazy thumbnails, chat inline source figure (alt = summary), agent `read` marker
- Prune guard: images-off syncs never prune `is_image` docs

**Test / lint / coverage (exact commands & outcomes):**
- `uv run pytest` → exit 0 (green; note: pytest 9.1.1 `-q` omits the final count line in output — exit code authoritative)
- `uv run pytest --cov=app --cov-report=term-missing` → **2715 passed, exit 0, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors, 0 warnings, 0 informations"
- `uv run pytest tests/e2e/test_image_documents.py -v --no-cov` → **4 passed, exit 0** (isolation)

**Completion criteria:** (1) images=true → described/embedded/displayed docs: ✅ (E2E + integration) · (2) images=false byte-identical + image docs survive sync: ✅ (E2E negative app + unit/integration) · (3) viewer + chat rendering with alt text; failed description skips + logs, sync completes: ✅ · (4) test/lint/coverage gates: ✅ · (5) commit + phase move: deferred to harness per this pass's rules (working tree left uncommitted).

**Notable deviation (pre-existing, documented in code):** image route uses `require_user` (phase-79 posture, same gate as the document content endpoint) rather than the phase text's "public" parenthetical — matches the endpoint it mirrors.

**Next pending phase:** `123_chat_image_questions`.
2026-09-25 01:54:23 -04:00
ducoterra 0f77e9a876 phase: 121_git_source_tokens
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Build and Push Containers / build-and-push-db (push) Failing after 14s
**Phase 121 final verification pass — all green** (all 4 tasks already in `complete/`; verified, no defects found, no changes needed)

- Verified implementation vs phase design: migration `0021` (reversible, round-tripped via `alembic downgrade base` + `upgrade head` → head `0021`), `GitSource.token` column, `normalize_credential`/`clone_url_for`/`sanitize_url`, clone callers switched (`sync.py`, `import_docs.py`), masked token fields in add form + editor, `extra="forbid"` output shapes
- Tests: `uv run pytest` → 2662 passed, 0 failed (exit 0); `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (≥90% gate)
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov` → **4 passed**

Completion criteria:
1. Private repo (UI add or pasted embedded-token URL) clones with injected token; token absent from every API response, page text, title attr, and full HTML — **PASS** (integration raw-JSON assertions + E2E `_assert_token_nowhere`)
2. Legacy embedded-token rows still clone from stored URL; output sanitized — **PASS** (`test_sync_legacy_row_clones_with_original_stored_url`, `test_get_masks_legacy_embedded_token_row`, env-fallback masking)
3. Public/local sources byte-identical — **PASS** (verbatim-URL + no-userinfo-unchanged tests)
4. pytest / coverage / ruff / pyright — **PASS** (see above)
5. Commit + phase move — harness responsibility; task files already in `complete/`, changes left in working tree (no commit made, per protocol)

Notable: no deviations; DB left at head, functional. Next pending phase: **122_image_documents** (then 123_chat_image_questions).
2026-09-24 20:51:39 -04:00
ducoterra 3a0fc3db05 phase: 120_failed_turn_retry
Build and Push Containers / build-and-push-app (push) Successful in 3m5s
Build and Push Containers / build-and-push-db (push) Failing after 2m13s
All verification complete. Final report:

**Phase 120 — Failed-turn retry: verification pass (all 3 tasks were done; final verification + 1 regression fix)**

**Verified:** `ChatMessage.failed`/`error` (≤500, `extra="forbid"` intact); `finalizeFailedTurn` funnel on the 3 failure paths (catch-else, stream-drop guard, zero-frame fallback) with `failed: true` + capped detail + `markLastRetryable`; `appendFailedNote` restore branch (Save-as-doc/Tune excluded); `showErrorBanner`/`retryLastTurn` byte-pinned untouched; only the three paths persist `failed: true` (grep + unit pin); no test asserts the old broken behavior.

**Defect found & fixed (rule 7):** a real navigate-away mid-turn let the browser's teardown fetch rejection (TypeError, not AbortError) leak into the failed funnel, persisting a phantom failed brain record — `test_sources_midstream_bug.py::test_no_orphan_brain_message_when_navigated_before_first_token` failed (2 `.msg` after reload) and violated the phase-20 navigate-away convention. Fixed: turn-scoped `leftThePage` flag (set unconditionally on `pagehide`, reset in `runTurn`) skips the funnel in the catch-else branch; pinned by new unit test `test_navigate_away_is_not_a_failed_turn`. No phase-overview/PLAN/todo/complete files touched; no commits made.

**Gates (exact):**
- `uv run pytest` → 2577 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL 4271 stmts, 99% (>90%)
- `uv run pytest tests/e2e/test_failed_turn_retry.py -v --no-cov` → 4 passed (isolated)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- Regression E2E, isolated: `test_sources_midstream_bug.py` 6/6 (was 5/6); `test_llm_retry`/`test_tool_scaffolding_guardrails`/`test_stop_generation`/`test_navbar_refresh` 17/17

**Completion criteria:** (1) network error → banner + in-bubble Retry, re-ask without re-typing ✅ (E2E A); (2) refresh restores failed bubble + working Retry, no "new chat" ✅ (E2E C); (3) stopped/successful turns byte-identical ✅ (negative E2E, stop suite, byte-identity units); (4) pytest/coverage/lint/types ✅; (5) commit + phase move — left to the harness per pass rules.

**Notable:** deviation = the regression fix above (a navigation is not a failed turn; phase-20 partial-persist convention restored). Next pending phase: `121_git_source_tokens`.
2026-09-24 18:50:36 -04:00
ducoterra 0ff1f8c4d6 chore(agent): phase roadmap from TODO.md — 4 phases (120–123)
Protocol B append: failed-turn retry (L3–4), git source tokens (L5),
image documents (L6 ingest), chat image questions (L6 chat side).
TODO.md items now live in .agents/phases/todo/ and the file is cleared.

LLM-Generated: true
2026-09-24 16:27:44 -04:00
ducoterra a5b63f83ad phase: 119_name_signal_read_chips
Build and Push Containers / build-and-push-app (push) Successful in 2m1s
Build and Push Containers / build-and-push-db (push) Successful in 18s
All verification complete. Final report:

**Phase 119 final verification pass — all criteria verified, one stale pin fixed.**
- Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry.
- Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged.
- New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2.
- Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed).
- Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met).
- Next pending phase: **none** — `todo/` holds only phase 119.
2026-09-16 15:50:48 -04:00
ducoterra 795fb56425 add phase 119
Build and Push Containers / build-and-push-app (push) Successful in 15s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-16 10:05:52 -04:00
ducoterra 9820c361b0 phase: 118_summary_seed_context
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 14s
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.

- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
2026-09-16 06:57:49 -04:00
ducoterra 21aad84a6d chore(agent): phase roadmap from TODO.md — phase 118 summary seed context (top-5 summary seeding, all-doc summaries)
Build and Push Containers / build-and-push-app (push) Successful in 17s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-15 22:43:01 -04:00
ducoterra c851d1a1c5 phase: 117_tool_call_compact
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification complete — every gate green. Final report:

**Phase 117 final verification pass — all criteria verified** (tasks 01–04 were already in `complete/`; I re-verified the full gate myself, no code changes needed)
- **Verified:** `appendToolLine` wraps lines in `details.tool-calls-disclosure` (open live, `closeToolCalls` at delta/done/stop + restore-closed, count via textContent); `addToolLines` parity (closed); `.tool-call` deboxed / `code` dechipped; new disclosure/summary CSS; only the one pinned CSS assert updated
- `uv run pytest` → **2466 passed**
- `uv run pytest --cov=app --cov-report=term-missing` → **99%** on `app/` (>90% floor)
- E2E in isolation: `test_tool_call_compact.py` **4 passed** (fold/expand/deboxed-flow/restored-folded); `test_agent_document_tools.py` **4 passed**; `test_big_read_progress.py` **4 passed** (live disclosure open + `.tool-elapsed` visible); `test_thinking_display.py` **5 passed**; `test_smoke.py` **3 passed**
- `uv run ruff check . && uv run pyright` → clean (pyright 0 errors, 0 warnings)
- Completion criteria: pins 1–4 ✅ (E2E above); live "calling tool" state byte-identical ✅ (`test_frontend_tool_states.py` 10 passed incl. `test_calling_tool_label_strings`); scope ✅ — `git diff` limited to the 3 frontend assets + 2 unit tests + E2E story + phase files, **0 changes in `app/`**
- Notable: `TODO.md` carries a **pre-existing uncommitted owner edit** (new retrieval-context idea) already in the tree before this pass — left untouched, will ride along in the harness commit; pre-existing committed quirk: the brain-wave commit's styles.css comment mislabels itself "Phase 117" (cosmetic, out of this diff)
- No commit made (harness commits + moves the phase); **next pending phase: none** — `todo/` holds only phase 117
2026-09-15 18:26:38 -04:00
ducoterra 2ac3fc89c2 feat(ui): swap the in-turn loader for a brain-wave sweep left of the send button
Build and Push Containers / build-and-push-app (push) Successful in 2m11s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Phase 117 (owner request, live-mockup-confirmed): the phase-109 3-dot
cue becomes a compact ECG trace (49px, P/QRS/T) with a brand sweep
traveling the path (bwdraw, 42/140 dash segment, 0.9s loop), the
loader repositioned left of the button so its appearance never shifts
it. setUiState stays the sole owner of the loader's hidden attribute;
the reduced-motion variant stills the sweep. Unit + lifecycle-E2E pins
updated for the new contract.
2026-09-15 11:55:59 -04:00
ducoterra adf31a4a35 phase: 116_modal_scrollbar_theme
Build and Push Containers / build-and-push-app (push) Successful in 2m1s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification complete — every gate re-run independently and green. Final report:

**Phase 116 final verification pass — all green (work was already in the tree; verified, nothing to fix)**

- **Verified** `styles.css`: `--scrollbar-thumb`/`--scrollbar-track` tokens in `:root` (color-mix derivations — follow the saved theme, not in BUILTIN_COLORS; contrast 3.73:1 ≥ 3:1, recomputed independently) + scoped rules `.doc-modal .doc-raw` (overflow-x already owned by the element)
- **Verified** E2E test 8 `test_raw_code_block_themed_scrollbar`: overflow, scroll action, computed-style theme assertions, screenshot
- **Screenshots visually confirmed**: headed `doc_modal_scrollbar_themed_1280.png` shows muted rounded thumb on dark track, no native light-gray bar; headless suite shot records scrolled state (headless Chromium paints no bar UI — documented in test docstring)

**Test / lint / coverage (re-run by me):**
- `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` → **8 passed** (isolation)
- `uv run pytest --cov=app --cov-report=term` → **2457 passed, app/ 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_dark_tech_theme.py -v --no-cov` → **6 passed** (no token collision)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:**
1. Modal scrolls horizontally, scrollbar themed, screenshot shows no native bar — ✅
2. No global scrollbar change (all new selectors under `.doc-modal`; grep-confirmed), regressions green — ✅
3. pytest green, coverage >90%, e2e isolated green, ruff+pyright clean — ✅
4. `--no-gpg-sign` commit + phase move — harness job (left uncommitted in working tree, per rules)

**No defects found; no deviations. Next pending phase:** none — `todo/` contains only phase 116.
2026-09-15 06:11:38 -04:00
ducoterra 990c8adf13 phase: 115_doc_draft_discard
Build and Push Containers / build-and-push-app (push) Successful in 2m12s
Build and Push Containers / build-and-push-db (push) Successful in 14s
All green — this was the final verification pass; everything from the four completed tasks was already in the working tree and verified.

**Phase 115 — Doc drafts: Discard + DELETE route + title fix — verification report**

- Verified all 4 task deliverables present: DELETE route (`app/api/doc_drafts.py`), Discard UI (`doc-edit.html` + `doc-edit.js` + `.discard-draft` CSS), title fix (`defaultDocTitle(wrap)` pairing + `saveAsDoc` call site), and all test pins (integration, frontend unit, E2E). No code changes needed.
- **Completion criteria:**
  1. ✅ Orphaned draft discardable from edit screen; row gone — `test_delete_removes_row_and_invalidates_token` (204 → GET 404), unknown-token 404, admin-gate 403 on all routes, E2E `test_discard_draft_from_edit_screen` all pass.
  2. ✅ Title after retry redo = redone answer's own question — E2E `test_save_title_is_the_redo_question_after_retry` passes.
  3. ✅ Push flow byte-identical — `git diff` shows only the new DELETE route + module docstring; all 7 existing push tests green.
  4. ✅ `uv run pytest --cov=app` → **2457 passed**, app coverage **99%** (>90%); `uv run pytest tests/e2e/test_save_doc_session.py -v --no-cov` → **4 passed**; `uv run ruff check .` → clean; `uv run pyright` → 0 errors.
  5. ⏳ Commit + phase-dir move left to the harness (per executor rules, no `git` run; all changes left in the working tree).
- No defects found; no deviations.
- Next pending phase: none in `todo/` other than this one (`115_doc_draft_discard` is the last).
2026-09-15 05:35:05 -04:00
ducoterra 3846f26a58 phase: 114_embed_question_length
Build and Push Containers / build-and-push-app (push) Successful in 2m6s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification passes complete — the phase was already fully implemented in the working tree, and every gate is green. No defects found; no code changes were needed.

**Final verification pass — Phase 114 (embed question length):**
- Verified truncation: `chat.py:459` embeds `request.message[:settings.embed_question_max_chars]` (default 1200, `BOR_EMBED_QUESTION_MAX_CHARS`, `>0` validator); full question still reaches the LLM prompt/log.
- Verified error mapping: `EmbeddingInputTooLargeError(EmbeddingError)` (byte-identical message) caught **before** `EmbeddingError` → no retry, terminal frame `detail="Question too long — trim it and re-ask."` + reachability-fine hint; `ChatErrorEvent.hint` additive.
- Verified frontend chain: frame `hint` → `err.hint` → `setUiState(error, …, {hint})` → `showErrorBanner(…, {retryable: true})` — hint replaces default `ERROR_HINT`, phase-111 `#banner-retry` button revealed. 4,000-char clamp untouched (A2).
- `uv run pytest tests/unit/test_embed_question_length.py -v --no-cov` → 21 passed
- `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov` (isolation, DB up) → 1 passed (4,000-char question → done, no banner)
- Regression: `test_llm_retry.py` 4 passed · `test_oneshot_llm_retry.py` 2 passed · `test_chip_sizing_question_cap.py` 6 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2444 passed, TOTAL **99%** (>90% gate)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:** (1) 4,000-char question embeds prefix + full prompt ✅ · (2) too-large → accurate frame + hint + Retry button ✅ · (3) reachability failure byte-identical (retries + old copy) ✅ · (4) all gates green ✅ · (5) commit/phase-move → left to the harness per instructions (no `git add`/`commit` run).
**Deviations:** none. **Next pending phase:** `115_doc_draft_discard`.
2026-09-15 04:16:55 -04:00
ducoterra 97d663d16d phase: 113_source_chip_quality
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 15s
All gates green — no defects found; this pass was verification only.

**Phase 113 final verification pass — report**

- Verified (no code changes needed): `select_documents_tiered` cited/related tiering + `select_documents` wrapper, `TurnPlan.related_docs`, `ChatDoneEvent.related` (additive, old payloads parse), `appendRelated` UI row (`.related-doc`, never `.source-chip`), done-frame + restore-path wiring, two settings with validators, `.env.example` entries
- `uv run pytest --cov=app --cov-report=term-missing` → 2422 passed, app/ coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (isolated) → 2 passed
- Regression E2E `test_retrieval_quality.py` + `test_honest_deflection.py` + `test_chat_rag.py` + `test_sources_midstream_bug.py` → 17 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors); `bash .agents/validate.sh` → "validation OK"

Completion criteria:
1. Single-doc question → exactly one `.source-chip` (E2E): ✅ passed
2. Weak 2nd doc only in de-emphasized related row, never `.source-chip` (unit + E2E): ✅ passed
3. Deflected turn → zero citation chips, weak hits in related row: ✅ passed
4. Full suite green, coverage >90%, isolated E2E green, lint/types clean: ✅ passed
5. `--no-gpg-sign` commit + phase dir move: left to harness per pass rules (task files already in `complete/`)

No deviations. Next pending phase: `114_embed_question_length`.
2026-09-15 03:11:05 -04:00
ducoterra 1374faf136 phase: 112_honesty_gate_weak_hits
Build and Push Containers / build-and-push-app (push) Successful in 2m15s
Build and Push Containers / build-and-push-db (push) Successful in 14s
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):**

- Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`.
- Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length).
- Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented.
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW).
- E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**.
- Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors.

**Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree).

**Deviations:** none. Next pending phase: `113_source_chip_quality`.
2026-09-15 00:37:38 -04:00
ducoterra 2683128876 phase: 111_chat_banner_retry
Build and Push Containers / build-and-push-app (push) Successful in 2m24s
Build and Push Containers / build-and-push-db (push) Successful in 14s
## Phase 111 Completion Report

**Implemented/Verified:**
- `#kb-banner` contains a `<button type="button" class="banner-retry" id="banner-retry">` (hidden by default, Retry label + SVG)
- `showErrorBanner(detail, opts)` reveals the button only when `opts.retryable` is true AND `lastBrainWrap` exists
- Turn-error path passes `{ retryable: true }`; all non-turn callers (share, save-doc, stale) remain text-only
- `clearErrorBanner()` re-hides the button
- `ERROR_HINT` changed from "Try again — …" to "If this persists, check the LLM is reachable."
- `.banner-retry` CSS styled as a pill (matching `.stale-regenerate` family)
- 12 source-assertion unit tests in `tests/unit/test_frontend_banner_retry.py`

**Test / Lint / Coverage:**
- `uv run pytest tests/unit/test_frontend_banner_retry.py -v --no-cov` → 12 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2362 passed, 99% coverage
- `uv run ruff check .` → All checks passed
- `uv run pyright` → 0 errors
- `tests/e2e/test_llm_retry.py` → 4 passed (in isolation)
- `tests/e2e/test_smoke.py` → 3 passed (in isolation)

**Completion Criteria:**
- ✅ Retry button visible after failed chat turn, re-runs last question
- ✅ Non-turn callers show text-only banner (no button)
- ✅ pytest green, coverage >90%, ruff + pyright clean
- ✅ Phase dir to be moved by pipeline gate

**Next pending phase:** `112_honesty_gate_weak_hits`
2026-09-14 23:07:56 -04:00
ducoterra f37c517590 chore(agent): phase roadmap from TODO.md — 6 phases (111–116): banner retry, honesty gate, chip quality, embed length, draft discard, modal scrollbar
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 13s
2026-09-14 22:07:32 -04:00
ducoterra 2b75f3cc85 docs(todo): log live testing findings L1-L7 (brain.reeseapps.com session)
Interactive browser test session (15 tests) against the deployed
instance. Findings: L1 banner 'Try again' is plain text, L2 honesty
gate eagerness + stochastic deflection compliance, L3 unstyled modal
scrollbar, L5 recurring weak-hit 2nd source chip, L6 4000-char
question clamp exceeds the embed input cap (500 surfaced as
'unreachable'), L7 save-as-doc drafts have no discard path. Each
entry carries observed evidence, suggested fixes and acceptance.
2026-09-14 21:31:08 -04:00
ducoterra 9ce13b6290 add vulns.md
Build and Push Containers / build-and-push-app (push) Successful in 29s
Build and Push Containers / build-and-push-db (push) Successful in 13s
2026-09-14 16:01:58 -04:00
ducoterra 3a4035fc96 phase: 110_fix_sse_db_pool_exhaustion
Build and Push Containers / build-and-push-app (push) Successful in 2m14s
Build and Push Containers / build-and-push-db (push) Successful in 13s
---

**Phase 110 — Fix SSE DB Connection Pool Exhaustion (SEC-14-04): COMPLETE**

**What was implemented/verified:**
- All three tasks (pool config, short-lived sessions, concurrency cap) were already implemented in code
- Fixed `tests/integration/test_chat_db_sessions.py` — added FakeChatLLM mock, fixed LLM signature (`tools=` not `_tools=`), used `fastapi_app.dependency_overrides` instead of `client.app.dependency_overrides`
- Fixed `tests/e2e/test_chat_db_pool.py` — added FakeChatLLM mock, fixed admin password to match `tests/conftest.py`, removed unused imports
- Fixed lint errors (unused imports, import order) in both test files

**Test / lint / coverage results:**
- `uv run pytest` → 2350 passed, 1 warning, 56.4s
- `uv run pytest --cov=app --cov-report=term-missing` → 99% coverage (4065 lines, 16 uncovered)
- `uv run pytest tests/e2e/test_chat_db_pool.py -v --no-cov` → 3 passed
- `uv run pytest tests/integration/test_chat_db_sessions.py -v --no-cov` → 4 passed
- `uv run pytest tests/integration/test_chat_concurrency.py -v --no-cov` → 11 passed
- `uv run pytest tests/unit/test_db_pool_config.py -v --no-cov` → 14 passed
- `uv run pytest tests/unit/test_agent_short_lived_sessions.py -v --no-cov` → 7 passed
- `uv run ruff check .` → all checks passed
- `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:**
- [✓] `app/db.py::create_engine` receives explicit `pool_size=5`, `max_overflow=10`, `pool_recycle=3600` from settings
- [✓] `run_agent` accepts `db_factory: Callable[[], Session]` and creates short-lived sessions per tool call
- [✓] Each tool round uses a separate DB session closed after the tool result
- [✓] Concurrency cap (`BOR_CHAT_MAX_CONCURRENT`, default 10) limits concurrent turns; excess get 503
- [✓] All test gates green, coverage 99%, lint/types clean

**Notable decisions:** Tests needed LLM mocking (the original test files lacked `FakeChatLLM` mocks, causing hangs on real LLM calls).

**Next pending phase:** None — this is the last phase in `todo/`.
2026-09-14 15:55:13 -04:00
ducoterra 35d65d2f25 feat(rag): summarize single-document folders (MIN_DOCS_PER_FOLDER 2 → 1)
Build and Push Containers / build-and-push-app (push) Successful in 2m10s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Relax the phase-94 folder-summary scope rule from ≥ 2 documents to
≥ 1: a folder (or source root) is a candidate while ANY document
lives under it, so single-file folders and single-file source roots
get their own lite-written description. A row is now pruned only
when its folder loses its last document (vanishes from the
catalogue).

The constant is the single source of truth, so the flip propagates
to the generator's candidate set, the prune pass, the
missing_folder_summaries gap probe (the next sync self-heals the
new gaps), and the KB-tree summary_pending markers (1-doc folders /
sources now read "Summary pending" until their row lands).

Docstrings/comments across app/, scripts/import_docs.py, and the
E2E fixtures updated to the ≥ 1 wording. Unit + integration tests
updated to the new semantics (the pruned-below-minimum scenario is
now a folder losing its LAST doc; single-doc folders are pinned as
candidates/pending). Full suite: 2314 passed, app coverage 99%;
ruff + pyright clean; folder-summary E2E stories pass in isolation
(ls_tree_drilldown, sync_summary_visibility, kb_tree,
kb_tree_nav, document_dates, oneshot_llm_retry).
2026-09-14 08:57:57 -04:00
ducoterra 3a81793565 phase: 109_turn_progress_loader
Build and Push Containers / build-and-push-app (push) Successful in 4m39s
Build and Push Containers / build-and-push-db (push) Successful in 35s
All gates green. Final report:

**Phase 109 — final verification pass (all 3 tasks were already complete; no re-implementation needed)**

- Verified D15 (thinking-handler re-open `block.open = true` after `ensureThinkingBlock`; delta keeps closing; "never reopens" narrative removed; restore path still collapses) and D16 (single static `#turn-loader` in composer row, `aria-hidden`, sole `turnLoader.hidden = !inFlight` write in `setUiState`; CSS reuses `typing` keyframes + reduced-motion override after the main rule + provenance; global `[hidden]` guard intact; mock `TURN_PROGRESS_TRIGGER` repro marker; `app/` untouched — pure UI phase).
- Unit pins: `tests/unit/test_frontend_turn_loader.py` (9 tests) — green via full run.
- `uv run pytest tests/e2e/test_turn_progress_loader.py -v --no-cov` → **3 passed** (isolation, DB up): loader visible send→tool-gap→final-thinking, hidden after done; re-opened scratchpad carries `marker-thought-42`/`marker-final-thought-42`; `#send-status` carries state text, back to empty idle; reduced-motion context stills dots (computed `animation-name: none`) with loader still visible.
- Regressions in isolation: `test_thinking_display` 5 passed, `test_stop_generation` 3 passed, `test_big_read_progress` 4 passed, `test_loading_feedback` 5 passed.
- `uv run pytest` → **2314 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90%); `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings.
- All terminal paths (done→idle L2554, stream error→error L2543, timeout→error L2247) funnel through `setUiState` — never-stale by construction.
- No defects found; no code changes made in this pass. Commit + phase-dir move left to the harness per pipeline rules (working tree carries all changes).
- Next pending phase: `108_history_wire_check` (still in `todo/`; out of scope here).
2026-09-14 01:27:16 -04:00
ducoterra fbbd98d734 phase: 108_history_wire_check
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 12s
Final verification pass complete — everything independently re-verified green. Report:

**Phase 108 — History wire check (final verification pass; verdict: NO BUG, tests-only per D13)**

- Verified all 3 tasks complete in `.agents/phases/complete/108_history_wire_check/`; re-ran every pin and gate myself
- Layer 1 (unit): 2 owner-shape pins in `tests/unit/test_history.py` (2-turn keep-whole + A4 thinking mapping) — pass
- Layer 2 (integration): `test_endpoint_two_turn_history_reaches_the_llm` — captured LLM request = `[system, user Q1, assistant R1, user Q2]`, `history_msgs=2` — pass
- Layer 3 (E2E): `tests/e2e/test_history_wire_check.py` — owner's verbatim 3-message scenario, echo byte-exact (T2: `2 prior messages`+R1 tail; T3: `4 prior messages`+R2 tail) — pass
- `VERDICT.md` present in phase dir: layer outcomes + "no bug — model behavior/user error" + pin names

**Gates (exact results)**
- `uv run pytest --cov=app --cov-report=term-missing` → 2305 passed, TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` → 2 passed (isolation, DB up)
- `uv run pytest tests/e2e/test_llm_history.py -v --no-cov` (phase-74 regression) → 3 passed
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors

**Completion criteria:** all six met — no defects found; diff scoped to `tests/**`, `VERDICT.md`, `.agents/phases/**`, `.agents/reports/**` (no `app/`/`frontend/` changes, consistent with no-bug verdict). Per executor rules, no git commit made — left for the harness.

**Next pending phase:** `109_turn_progress_loader`
2026-09-13 23:57:54 -04:00
ducoterra 6bcee831ec phase: 107_git_full_history_dates
Build and Push Containers / build-and-push-app (push) Successful in 1m49s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All green — verification complete. Final report:

**Phase 107 — final verification pass (all tasks already in `complete/`)**

- **Verified:** `clone_or_pull` full-history fix (no `--depth` clone; `--is-shallow-repository` probe → one-time `git fetch --unshallow` self-heal → `git pull --ff-only`; D12 fail-loud) in `scripts/git_sync.py`; E2E suite `tests/e2e/test_git_source_dates.py` present; all five stale shallow/tip-date narratives corrected (no residual claims repo-wide; `docs_push.py` `--depth` fetches untouched as designed).
- **Unit+integration:** `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py tests/integration/test_import_docs_git.py tests/integration/test_sync_api.py -v --no-cov` → **66 passed** (D10 tip-date test replaced by `test_url_clone_yields_true_per_file_dates` + `test_existing_shallow_checkout_self_heals`; unit pins: clone argv w/o `--depth`, probe→unshallow→pull order, unshallow-failure → `GitSyncError`).
- **New E2E in isolation:** `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` → **4 passed** (old file 2020-06-15 ≠ tip file 2024-06-15 in `GET /api/docs`, Sources `Created`/`Updated` columns, viewer `Created` badge ISO title; folder `updated_at` subtree maxes).
- **Phase-106 regressions in isolation:** `test_document_dates.py` → 6 passed; `test_sync_button.py` → 3 passed; `test_git_sources_admin.py` → 6 passed.
- **Full gates:** `uv run pytest` → **2302 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90%); `uv run ruff check . && uv run pyright` → **clean, 0 errors**.
- **Completion criteria:** 1 ✅ non-shallow URL clone + true per-file dates · 2 ✅ self-heal / plain-pull / fail-loud · 3 ✅ both suites green, bug-pin replaced · 4 ✅ E2E green in isolation · 5 ✅ all regressions + full suite + coverage + lint · 6 ✅ narratives corrected · 7 — no commit made (harness override: changes left in working tree; task files already in `complete/`).
- **No defects found; no deviations.** Next pending phase: **108_history_wire_check**.
2026-09-13 23:21:39 -04:00
ducoterra addbd4ca08 add phase 107
Build and Push Containers / build-and-push-app (push) Successful in 34s
Build and Push Containers / build-and-push-db (push) Successful in 26s
2026-09-13 22:28:04 -04:00
ducoterra e2d08a95a9 chore(agent): phase roadmap from TODO.md, 2 phases (108 history-wire check, 109 turn progress loader) 2026-09-13 22:27:05 -04:00
ducoterra ee3efb28c9 phase: 106_document_dates
Build and Push Containers / build-and-push-app (push) Successful in 4m35s
Build and Push Containers / build-and-push-db (push) Successful in 14s
Everything is verified green. Final report:

**Phase 106 — Document dates (final verification pass; all 10 tasks already complete)**

- Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed.
- `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up)
- 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation**
- `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings**

**Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions.

- **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring).
- **Next pending phase:** none — `todo/` holds only this phase.
2026-09-13 19:28:05 -04:00
ducoterra cec819743d add phase 106 - document dates
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 14s
2026-09-13 12:18:01 -04:00
ducoterra 8476dc1e07 phase: 99_kb_tree_table_and_back_nav
Build and Push Containers / build-and-push-app (push) Successful in 1m55s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification is complete and green. Final report:

**Phase 99 — final verification pass (all 3 tasks already in `complete/`)**

- Verified the one-line Description clamp: `.kb-desc-cell` flex wrapper + `.kb-desc-text` ellipsis triad + `nowrap` column + in-cell Edit button + hover `title` + untouched level block (source diffs match D1; unit pins present)
- Verified the back-button breadcrumb nav: `applyTarget(target, push)` state-only `pushState`, no-duplicate gate, popstate adopt/reset, `bor:view-refresh` alignment before `loadTree()`, anonymous gate listener-free, router.js untouched (D2)
- `uv run pytest --cov=app`: **2180 passed**, coverage **99%** (>90% gate)
- `uv run ruff check .`: clean; `uv run pyright`: **0 errors, 0 warnings**
- `uv run pytest tests/e2e/test_kb_tree_nav.py -v --no-cov` (isolated, DB up): **4/4 passed** (back-walk, jump-then-back, fresh-nav/active-reclick, measured clamp incl. row-height parity ±4px, computed styles, full text in DOM/title/level block)
- Regression suites isolated: `test_kb_tree.py` 8✓, `test_ls_tree_drilldown.py` 3✓, `test_navbar_refresh.py` 7✓, plus `test_sync_summary_visibility.py` 3✓ (marker pin updated to `kb-desc-text kb-summary-pending` — the sanctioned class-pair change) and `test_edit_summaries.py` 4✓
- Criteria 1–4: **met** (each as above). Criterion 5 (atomic commit): left to the harness per executor rules — no `git add`/`commit` run; all changes left in the working tree
- No defects found in prior phases; no deviations. Stray `__pycache__/test_zz_smoke_tree_ui.*.pyc` is a leftover (no `.py` on disk) — inert, not touched
- Next pending phase: **none** — `todo/` will be empty once the harness moves this phase
2026-09-13 01:57:40 -04:00
ducoterra f665a83b1a phase: 98_sync_summary_visibility
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All verification complete. Final report:

**Phase 98 — Sync summary visibility: final verification pass** (all 5 tasks already complete; implementation verified against the design, no defects found, no code changes needed)

- **Implementation checked:** `SyncStatus` phase machine (4 new keys, terminal-keep counts), `on_progress` hook in `generate_folder_summaries`, `summary_pending` on `KbTreeSource`/`KbTreeFolder` + D3 rule in `build_kb_tree`, phase-aware sync labels + pending UI in `sources.js`, `.kb-summary-pending` CSS — all match decisions D1–D5.
- **Unit + integration:** `uv run pytest` → 2184 tests, 0 failed/errors (exit 0)
- **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **99%** on `app/` (criterion >90% ✓; `app/api/sync.py` and `app/rag/folder_summaries.py` at 100%)
- **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- **Phase E2E (isolation):** `uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov` → **3 passed** (phase machine, live label, pending markers + gap-fill self-heal)
- **Regression suites (each isolated, `--no-cov`):** test_kb_tree ✓, test_ls_tree_drilldown 3 ✓, test_sync_button 3 ✓, test_sync_upload_progress 4 ✓, test_oneshot_llm_retry 2 ✓, test_local_directory_sources 3 ✓
- **Completion criteria:** all 7 verified green — status phase fields + terminal semantics; `Writing KB overview…`/`Summarizing folders… (n/m)` labels (title + aria-live); pending set == `missing_folder_summaries` (integration cross-check pinned at `test_docs_api.py:428`); CLI/`ls` byte-identity (no changes to those paths, pins green); suite/coverage/lint gates; dedicated + regression E2E. Commit left to the harness per protocol (no `git add`/`commit` run).
- **Decisions/deviations:** none — no fixes were required this pass.
- **Next pending phase:** `99_kb_tree_table_and_back_nav`.
2026-09-13 00:23:05 -04:00
ducoterra 909c96c7bc minor fix to prevent null bytes from breaking sync
Build and Push Containers / build-and-push-app (push) Successful in 1m54s
Build and Push Containers / build-and-push-db (push) Successful in 19s
2026-09-12 22:12:09 -04:00
ducoterra d731169b8b phase: 105_hidden_folders_toggle
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All completion criteria verified. Everything is green.

**Phase 105 final verification pass — all criteria verified**

- Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page
- Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified)
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓
- Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation)
- Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed
- Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings)

**Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched).

**Deviations:** none — no defects found; no code changes were needed on this pass.

**Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`.
2026-09-12 21:26:30 -04:00
ducoterra ecc921098a phase: 104_chip_sizing_question_cap
Build and Push Containers / build-and-push-app (push) Successful in 1m48s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All completion criteria verified. Final report:

**Phase 104 — final verification pass: all green**
- Verified (no code changes needed): single-line ellipsized `.suggestion-chip` CSS + deleted `.maybe-try` override, `renderChips` full-text `title` + `aria-label`-when-clipped, `maxlength="4000"` + `#char-count` counter + `handleSend` over-cap guard, unit pins file, schemas boundary pins, dedicated E2E suite.
- E2E (isolation): `uv run pytest tests/e2e/test_chip_sizing_question_cap.py -v --no-cov` → **6 passed**; regressions: `test_suggestion_chips.py` 8 passed, `test_pinned_composer.py` 4 passed, `test_responsive_polish.py` 7 passed, `test_chat_history.py` 5 passed.
- `uv run pytest` → **2102 passed**; `--cov=app` → **99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- Criteria: chip E2E (single-line, clipped, title+aria-label full text) ✅; paste caps at exactly 4,000, send streams, counter hides ✅; programmatic 5,000-char fill → banner, no turn, text kept ✅; 4,000/4,001 boundary pinned + HTML maxlength == JS constant cross-file pin ✅.
- Diff scope: `frontend/`, new unit file, `tests/unit/test_schemas.py`, new E2E file, phase files — **no `app/` diff, no migration, no `shared.js` diff**.
- Deviations: 4 regression test files touched — 2 genuine DOM-pin conflicts from the new `#char-count` child (explicitly anticipated by the overview) + 3 documented **pre-existing E2E flake fixes** (smooth-scroll race, tab-walk heuristic, 10 ms timeout), each verified pre-existing on the pre-phase-104 tree.
- No commit made (harness commits per the execution protocol override).
- Next pending phase: `98_sync_summary_visibility`.
2026-09-12 19:45:00 -04:00
ducoterra 1f1c01c9f7 phase: 103_suggestions_session_openers
Build and Push Containers / build-and-push-app (push) Successful in 2m32s
Build and Push Containers / build-and-push-db (push) Successful in 12s
Phase 103 final verification pass — all green.

**Verified (all 3 tasks already in `complete/`; no code changes needed):**
- `opening_questions` in `app/api/suggestions.py` — forward walk, one opener per chat (first non-blank user msg, A3), reads raw `messages` not `title` (A4), phase-80 order/dedup/cap/seed contracts; `last_questions` name gone from `app/`+`tests/`
- Docs updated: `app/config.py` seed docstring, `.env.example` `BOR_SUGGESTIONS`, `README.md` — "session openers" wording
- Diff scope correct: only the 6 expected files + phase-file moves; `app/rag/suggestions.py` and `frontend/` untouched

**Test / lint / coverage results:**
- `uv run pytest tests/integration/test_suggestions_api.py -v` → 12 passed
- `uv run pytest tests/e2e/test_suggestion_chips.py -v --no-cov` → 8 passed in isolation (opener-only core pin included)
- `test_responsive_polish.py` → 7 passed; `test_chat_persistence.py` → 4 passed (both isolated, no edits)
- `uv run pytest --cov=app --cov-report=term-missing` → 2086 passed, TOTAL 99% (>90%); `app/api/suggestions.py` 100%
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:** all 7 ✅ (follow-ups-never-surface pin; cap-across-chats pin; seed/dedup/case/partial/A3/401 pins; E2E suites isolated; deflection chips unchanged; full suite + lint; commit + dir move left to harness per executor rules).

**Deviations:** none — no defects found; nothing changed in this pass.
**Next pending phase:** `98_sync_summary_visibility` (numeric order in `todo/`).
2026-09-12 16:37:50 -04:00
ducoterra 3b2dea5685 phase: 102_extensionless_filenames
Build and Push Containers / build-and-push-app (push) Successful in 1m38s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification complete — every gate green, no defects found in previously completed work.

**Phase 102 final verification pass — report**

Verified (all three task files present in `complete/`; working-tree implementation matches D1–D5 design):
- `match_extension` choke point in `app/rag/importer.py` (walk + `formats` counter), `doc_format` name-token badge in `app/api/docs.py`, config/`.env.example` docs, fixture `tests/fixtures/extensionless_kb/`, integration + E2E suites — all present and correct
- Completion criteria: end-to-end sync (✓ integration + E2E), case matrix incl. `mydockerfile`/`Dockerfile.dev`/`.dockerfile` exclusions (✓ unit), `formats=dockerfile:1` not `unknown` (✓ log-line assertion), badge `dockerfile`/`containerfile` + `text` fallback + suffixed unchanged (✓ unit/integration/E2E), prune-on-token-removal (✓ `pruned==2`), suffixed-path rule byte-identical (✓ single-line swap, existing cases untouched)

Test / lint results (exact commands):
- `uv run pytest --cov=app --cov-report=term-missing` → 2084 passed, **99%** coverage (>90% gate)
- `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` → 2 passed, isolated, DB up
- Regressions isolated: `test_import_documents` 3✓, `test_import_extensions_env` 2✓, `test_quadlet_jinja_import` 4✓, `test_document_viewer` 7✓, `test_kb_tree` 8✓
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings

Notable: commit intentionally not made (harness commits the phase); 102's task files already sit in `complete/`, overview stays in `todo/` for the harness.
Next pending phases: 98, 99, 103, 104, 105 (numeric next after 102: `103_suggestions_session_openers`).
2026-09-12 15:56:43 -04:00
ducoterra 4dbac1660a phase: 101_tokens_page_overhaul
Build and Push Containers / build-and-push-app (push) Successful in 1m49s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All gates verified green — no defects found, nothing to fix. Final report:

## Phase 101 — final verification pass (all tasks already complete; verified, not re-implemented)

**Verified against the working tree** (task files 01–04 present in `complete/`):
- Active/revoked table split (Status column gone, section hidden while 0 revoked, per-table search with query state surviving re-renders) in `frontend/index.html` + `tokens.js`
- `generateToken()` refuses blank/whitespace names (`Give the token a name first.`, no request); `|| "token"` fallback deleted
- Atomic rotation: `regenerate_token` service + `POST /api/tokens/{id}/regenerate` (201 new-plaintext-once / 404 / 409 / router-wide 403) + `TokenListItem.revoked_at` (D5)
- Regenerate two-step confirm UI + CSS (`.token-regenerate`, neutral hover, no new hue); A4 pins intact

**Test / lint / coverage results:**
- `uv run pytest` → **2065 passed**
- `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90% ✓)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- `uv run pytest tests/e2e/test_tokens_page.py -v --no-cov` → **4 passed** (isolation, DB up)
- Regression, each in isolation: `test_api_tokens.py` **9 passed**, `test_admin_auth.py` **6 passed**, `test_shared_header.py` **6 passed**, `test_theme_semantic_completion.py` **8 passed** (its revoked-pill pin was correctly re-scoped to the revoked table in this phase)

**Completion criteria:** 1 ✓ split+search (E2E 1–2) · 2 ✓ required name (E2E 3 + source pin) · 3 ✓ rotation end-to-end, old token refused at gate (E2E 4 + API 404/409 pinned) · 4 ✓ A4 holds (list carries no plaintext/hashes) · 5 ✓ suite/coverage/lint green · 6 ✓ E2E + regressions green in isolation · 7 commit left to the harness per executor rules (all changes uncommitted in the working tree)

**Deviations:** none. Next pending phase: `98_sync_summary_visibility`.
2026-09-12 15:16:02 -04:00
ducoterra 58e9d94cff phase: 100_page_width_consistency
Build and Push Containers / build-and-push-app (push) Successful in 4m3s
Build and Push Containers / build-and-push-db (push) Successful in 18s
**Phase 100 — final verification pass: all green.**

- Verified the shipped CSS contract directly: `--chat-column: 72rem` in `:root`; 0 literal `max-width: 46rem`; no `@media (min-width: 1500px)` block; exactly 4 token-capped reading columns; tuning/theme/doc-edit shells cap-free, structurally `.sources-shell`; `mock_llm.py` diff is comment-only.
- Defect found & fixed (phase-93 suite): `test_theme_semantic_completion.py::test_reset_removes_tag_byte_identical` raced theme.js's post-PUT refetch — it asserted the `#bor-theme` tag was gone right after the result line, but tag removal lands in the reconcile after the re-fetch (failed ~1 in 5 runs after `test_chat_rag`, reproduced). Fixed with an auto-waiting `expect(...).to_have_count(0)` (settled-state wait, the file's existing house pattern). 6/6 clean on the repro loop after.
- Tests/lint/coverage: `uv run pytest --cov=app --cov-report=term-missing` → 2052 passed, **99%** on `app/`; `uv run ruff check .` + `uv run pyright` → 0 errors.
- E2E in isolation (all passed): `test_wide_desktop_column.py` 3 (the phase suite — chat==tuning==theme==RAG ±4px at 1280 & 1920, ≈1152px; shared ≈1152px; standalone doc ≈1112px; modal unchanged ≈1100px; 360px overflow-free), `test_ui_customization` 4, `test_admin_theme_tab` 5, `test_document_viewer` 7, `test_save_share_ux` 5, `test_sticky_navbar` 3, `test_markdown_tables` 6, `test_responsive_polish` 7, `test_chat_rag` 3, `test_theme_semantic_completion` 8.
- Completion criteria: ① measured 72rem everywhere (≥~1200px, ±4px) + full-width below + 360px clean — **PASS** (E2E); ② zero 46rem rules / no 1500px block / four token selectors — **PASS** (grep + unit pins); ③ B4 byte-identical no-op + mobile squeeze — **PASS** (theme + responsive suites green); ④ full suite / coverage / lint — **PASS**; ⑤ atomic commit — left to the harness per executor protocol (all changes in the working tree, uncommitted).
- Deviations: none from phase decisions; only change in this pass is the race fix above (test-only, behavior unchanged).
- Next pending phase: `98_sync_summary_visibility` (numeric order in `todo/`; also pending: 99, 101–105).
2026-09-12 13:37:19 -04:00
ducoterra a2ca2f905f chore(agent): phase 105 (hidden-folders toggle) from TODO.md, clear the list 2026-09-12 11:55:03 -04:00
ducoterra ffa6bda3e5 add phases for fixing suggestion chips
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-12 10:52:51 -04:00
ducoterra 17dd3bfac1 docs(plan): remove hardcoded phase/migration references to prevent staleness
Build and Push Containers / build-and-push-app (push) Successful in 14s
Build and Push Containers / build-and-push-db (push) Successful in 16s
Refactor §12 roadmap to point to ls commands and 00_phase.md files
instead of listing specific phase numbers and descriptions.

Remove ~20 phase-number references from anchors, revision notes,
tool surface, feedback, and sync sections — keep only the one
stable reference (phase 03 convention).

Replace migration ranges with 'ls alembic/versions/'.
Replace hardcoded retry values with 'configurable (defaults: ...)'.
Status header now points to §12 instead of listing counts.
2026-09-12 10:22:45 -04:00
ducoterra 800bea769a docs(phases): author todo queue phases 98-102
98 sync summary visibility (status phases + pending markers), 99 catalog
one-line clamp + breadcrumb back-nav, 100 72rem page-width consistency,
101 tokens page overhaul, 102 extensionless filename import.
2026-09-12 10:15:22 -04:00
ducoterra ad7585d474 phase: 97_kb_tree_catalog
Build and Push Containers / build-and-push-app (push) Successful in 2m11s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All completion criteria verified — everything is green, no defects found. Final report:

## Phase 97 final verification pass — ALL GREEN

**Verified (no code changes needed):**
- `GET /api/docs/tree` (admin), `build_kb_tree` pure builder, `PATCH /api/folders/summary`, migration 0018 (`manually_edited`, head confirmed), generator skip/keep + `kept_manual` stat, RAG tree UI + edit affordance in `sources.js`/`index.html`/`styles.css`
- `tests/e2e/test_kb_tree.py`: 8 passed — top level, drill source/folder, edit round-trip, clear, manual-desc-survives-sync, reload fallback, anonymous gate
- Integration: tree shape/order/403/empty/indexed-only + PATCH update/create/root/clear/404/403/no-LLM + stat-walk equivalence (in `test_docs_api.py`); 3-field `folder_summaries=` import token preserved

**Gates (exact commands):**
- `uv run pytest --cov=app --cov-report=term-missing` → **2053 passed**, TOTAL coverage **99%** (>90% ✓)
- `uv run ruff check . && uv run pyright` → **All checks passed / 0 errors**
- `uv run pytest tests/e2e/test_kb_tree.py -v --no-cov` → **8 passed** in isolation
- 30 story/RAG-view E2E suites run **one per process**: all passed, incl. `test_ls_tree_drilldown` (agent `ls` byte-identical ✓), `test_import_documents`, `test_edit_summaries`, `test_admin_auth`, `test_kb_overview`

**Completion criteria:** tree view ✓ · edit round-trip + clear ✓ · manual persists/clear resets ✓ · `ls` unchanged ✓ · pytest/coverage/lint ✓ · E2E isolation ✓ · commit — left to harness per protocol (working tree untouched, `git add/commit` not run)

**Deviations:** none. **Next pending phase:** none — `todo/` contains only 97 (96 already committed).
2026-09-11 22:48:02 -04:00
ducoterra a49be80b8e phase: 96_oneshot_resilience
Build and Push Containers / build-and-push-app (push) Successful in 1m34s
Build and Push Containers / build-and-push-db (push) Successful in 10s
All checks complete. Final report:

**Phase 96 (one-shot resilience) — final verification pass, all green** (all 4 task files already in `complete/`; verified the working-tree implementation against the design)

- `LLMClient.chat()` empty-content retry (D1–D3) via `_chat_once` + `_EmptyContentError` (carries `finish_reason`), under `BOR_LLM_RETRIES`/`BOR_LLM_RETRY_DELAY` — verified in diff
- `missing_folder_summaries()` + `generate_folder_summaries(only_missing=…)` — verified; `folder_summary_table_empty` deleted, both sync gates switched to the gap probe
- `.env.example` comments updated (chat-turn stream + one-shot summary calls)

**Test / lint / coverage results**
- `uv run pytest --cov=app --cov-report=term-missing` → **1988 passed**, coverage **99%** (gate >90%)
- `uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov` → **2 passed** (isolated)
- Regressions, each isolated: `test_ls_tree_drilldown` 3 passed, `test_sync_button` 3 passed, `test_local_directory_sources` 3 passed, `test_llm_retry` 4 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:** retry-then-recover unit-pinned ✓ · exhaustion + `BOR_LLM_RETRIES=0` byte-identical ✓ · streaming path untouched ✓ · gap-fill both sync paths, other rows byte-identical incl. `updated_at` ✓ · no-gap zero-burn ✓ · phase E2E green ✓ · regression E2Es green ✓ · full suite + >90% + lint/types ✓ · no completed-phase behavior change (full suite green) ✓. Commit left to the harness per executor rules (working tree, 16 files).

**Deviations:** none. **Next pending phase:** `97_kb_tree_catalog`.
2026-09-11 13:16:20 -04:00
ducoterra bcaef800c5 phase: 95_read_truncation_cap
Build and Push Containers / build-and-push-app (push) Successful in 1m38s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All gates are green. Final verification is complete — no fixes were needed; the phase is fully implemented and passing.

**Phase 95 final verification report**

- Verified all 3 tasks already implemented (task files in `complete/`): read cap + notice + `ToolResultPiece` (task 01), `tool_result` SSE + live/saved/shared marker (task 02), E2E suite (task 03).
- Code checks passed: `BOR_READ_MAX_CHARS` (default 128 000, validator), pinned `READ_TRUNCATION_NOTICE` + `TRUNCATION_MARKER`, byte-identical at/under-cap read, holder `read_truncations` (no `tool_calls` impact), piece order (after `tool`, before next round), `ChatToolResultEvent`, `ToolCall` back-compat fields, `app.js` live/restore handlers, `shared.js`, `.truncated-note` CSS, `.env.example` entry, mock-LLM echo script.
- **Tests:** `uv run pytest` → 1966 passed; `uv run pytest --cov=app --cov-report=term-missing` → all green, TOTAL **99%** (>90% gate).
- **E2E:** `uv run pytest tests/e2e/test_read_truncation_cap.py -v --no-cov` → **3 passed** (frame order + live marker + LLM notice via echo; save→shared fidelity; short-read control).
- **Regression (isolated):** `test_agent_document_tools` 4 ✓, `test_chat_history` 5 ✓, `test_share_chat` 4 ✓, `test_big_read_progress` 4 ✓, `test_stop_generation` 3 ✓.
- **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings.

**Completion criteria:** ① over-cap read → first-cap-chars + marker + pinned notice — ✓ (unit-pinned: at-cap/cap+1/notice tests); ② user marker live/saved/shared — ✓ (E2E + frontend tests); ③ at/under cap byte-identical, no frame — ✓ (unit + control E2E); ④ top-2 `<documents>` retrieval untouched — ✓ (`app/rag/retriever.py` unmodified vs HEAD); ⑤ suite green, >90% coverage, ruff+pyright clean — ✓; ⑥ no completed-phase behavior change — ✓ (all gates green; commit left to harness per pass rules).

- No defects found; no changes made this pass. Next pending phase: none in `todo/` (96 is the next free number).
2026-09-11 03:42:51 -04:00
ducoterra d4943b4822 phase: 94_ls_tree_drilldown
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 25s
All green. Verification complete.

**Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)**

- Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal
- Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths
- Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met
- `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched)
- Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed
- Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol)

**Next pending phase:** `95_read_truncation_cap`
2026-09-11 00:59:35 -04:00
ducoterra 9188be259b phase: 93_theme_semantic_completion
Build and Push Containers / build-and-push-app (push) Successful in 1m56s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All verification is complete. Final report:

**Phase 93 — Theme semantic completion: FINAL VERIFICATION PASS — ALL GREEN**

- Verified full implementation in tree: migration `0016` (8 nullable semantic columns, applied at head), 17-var `BUILTIN_COLORS`/`COLOR_FIELDS`/`effective_settings`, API validation, `#view-theme` State-colors fieldset (17 pickers), `theme.js` FIELDS/PAIRS (5→8), `.page-head` surface panel (6 shell views + doc-edit + shared.html; login card / document sticky header audited as already-surfaced), mock_llm `content: None` fix
- Fixed 2 pre-existing defects (both fail identically on baseline `d4f38ad`, proven via worktree A/B): `test_nav_rename_sources` — expected nav tail missing the phase-91 "Theme" link; `test_stale_ui_copy` — now truncates `saved_chats` before/after (house `test_suggestion_chips` pattern) so the seed-chip contract is deterministic on the shared dev DB (owner's 22 saved chats triggered phase-80 last-3-questions)
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → **1868 passed, app/ 99%** (>90% ✓); `uv run ruff check .` → clean; `uv run pyright` → **0 errors**
- E2E: dedicated `uv run pytest tests/e2e/test_theme_semantic_completion.py -v --no-cov` → **8/8 in isolation** (all-gray 17-color theme: zero residual color on saved-result/Stale/Revoked/Local/tool-call elements, text labels intact, gray heads non-transparent, pre-paint tag, Reset → byte-identical no-tag); 15 theme/header/nav/responsive suites green in isolation; full 85-file combined run: only the 2 fixed pre-existing failures + 1 combined-run artifact (`test_sync_upload_progress`, green in isolation)
- Completion criteria: (1) monochrome E2E ✓ (2) default byte-identical, no `#bor-theme` tag ✓ (3) all page heads on solid surface ✓ (4) suite/coverage/lint/E2E green ✓ (5) phases 01–92 no behavior change ✓ (6) commit left to harness per protocol
- Notable: cleaned stray uvicorn leftovers from prior implementation pass (owner's `--reload` dev server untouched); no deviations from the phase design
- Next pending phase: `94_ls_tree_drilldown`
2026-09-10 16:43:08 -04:00
ducoterra d4f38ad3ce add PLAN.md
Build and Push Containers / build-and-push-app (push) Successful in 39s
Build and Push Containers / build-and-push-db (push) Successful in 25s
2026-09-10 13:08:55 -04:00
ducoterra bf308eb795 chore(agent): phase 93-95 roadmap from TODO.md — theme completion, ls tree drill-down, read truncation cap
Convert the three unchecked TODO.md items into an executable phase
roadmap (Protocol B, appended after phase 92):

- 93_theme_semantic_completion (TODO L3): the ok/err/accent state
  families become Theme-tab-controlled (B3 revised, owner permission
  2026-09-10) + surface panels behind every page head
- 94_ls_tree_drilldown (TODO L4): ls becomes a source -> folder ->
  file tree with sync-time lite-model folder summaries; controlled
  tool-calling battery as the accuracy/performance gate
- 95_read_truncation_cap (TODO L5): read capped at BOR_READ_MAX_CHARS
  (128k chars ~= 32k tokens, spec'd on the 128k-token minimum context),
  LLM-visible truncation notice pointing at grep, new tool_result SSE
  event (A15 extension) + the visible UI marker

Owner decisions (B3 / A7 scope / A15) are recorded in the phase files;
.agents/PLAN.md is being redone separately per the owner.
2026-09-10 11:37:25 -04:00
ducoterra dac4a3eec0 docs(agent): restore master plan at .agents/PLAN.md with locked decisions through phase 92
Re-lands the master plan (removed from VCS in 281f355 as .agent/PLAN.md)
at .agents/PLAN.md — the path AGENTS.md and the codebase's PLAN §…
references use. Rebuilt from the original text plus every
owner-permitted revision recorded in the completed phases: A1–A17
anchors with the full revision log (auth, SSE thinking/tool/retry
events, agent tools, history, SPA shell), new A18–A20 (docs push,
deploy caching, security headers), current data model (11 tables),
locked persona + <tools> copy, current UI/UX contracts (46rem/92rem
column, theme tab, never-stale feedback, no autoscroll), the full
per-turn log line, and the phase 01–92 roadmap with the open TODO.md
items as next up.
2026-09-10 09:28:58 -04:00
ducoterra df91c6316c phase: 92_theme_save_and_coverage
Build and Push Containers / build-and-push-app (push) Successful in 1m47s
Build and Push Containers / build-and-push-db (push) Successful in 11s
**Phase 92 final verification pass — all green.** This pass re-verified the completed tasks (all 5 task files already in `complete/`) against every completion criterion; no defects found, nothing to fix.

- Verified: 9th identity var `grid_line` end-to-end (migration `0015` at head, model/`theming.py`/schemas/API, 422 + built-in→NULL tests present); `styles.css` zero hardcoded literals outside `:root` + derived `--brand-*` vars; 9th picker in theme form; wordmark themed; `theme.js` save/reset/re-show/mount live-sync; dedicated E2E suite + phase-91 suite updated.
- `uv run pytest --cov=app --cov-report=term-missing` → **1845 passed, exit 0, TOTAL 99%** (>90%)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- `uv run pytest tests/e2e/test_theme_save_and_coverage.py -v --no-cov` → **3 passed** (save-live, reset-live, whole-site)
- `uv run pytest tests/e2e/test_admin_theme_tab.py -v --no-cov` → **5 passed**
- Criteria: (1) Save/Reset repaint open page, no nav, SPA-nav survives, pre-paint intact ✅; (2) both `rg` gates green (only `:root` + documented `#fff` Stop label; zero SVG hex attrs), grid/selection/hovers/wash/wordmark E2E-proven ✅; (3) no-op contract live-checked: row-less `/` = no tag + exact A1 CSP, grid-only row = 9-var tag in `COLOR_FIELDS` order + sha256 CSP, with-row ≡ row-less bytes ✅; (4) full suite/coverage/lint/both E2E ✅; (5) commit left to the harness per instructions.
- Deviations (previously made, probe-verified, kept): live repaint uses CSSOM `<html>` overrides because Chromium blocks `<style>` textContent mutations under the locked sha256-only CSP (tag text still mirrors the next load; `<html>` style exact-saved after Save, empty after Reset); wordmark themed via 3 `.brand-mark` CSS rules instead of inline styles (task 03's inline attrs were CSP-blocked — fixed during task 04).
- Next pending phase: none — `todo/` contains only `92_theme_save_and_coverage`.
2026-09-10 00:23:08 -04:00
ducoterra d22d260b8b phase: 91_admin_theme_tab
Build and Push Containers / build-and-push-app (push) Successful in 5m43s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification is complete — this pass needed no code changes. Final report:

**Phase 91 — Admin Theme tab: final verification pass (all 6 tasks already in `complete/`)**

- Verified pre-paint theming end-to-end: `ui_settings` store + resolver, admin `GET/PUT /api/ui-settings`, `CachingMiddleware` inline-`<style id="bor-theme">` injection before `</head>` (incl. `/shared/<token>` prefix branch, unit-pinned), CSP sha256 exemption for the inline tag, Theme tab shell + `theme.js` editor, CSS-file theming fully retired.
- No defects found; zero changes made — working tree left exactly as the task executors left it.
- Tests: `uv run pytest --cov=app` → 1841 passed, 0 failed (TOTAL coverage **99%**; theming/ui_settings/caching all 100%); `uv run pytest tests/e2e/test_admin_theme_tab.py -v --no-cov` → **5 passed** in isolation.
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings.
- Criteria: (1) unset deployment byte-identical, no `#bor-theme` anywhere — ✓ (unit no-op test + E2E reset byte-compare); `rg "BOR_THEME|themes/"` → single hit is the permitted doc-history comment in `frontend/index.html`. (2) admin-only gate + 403s for anonymous and token users — ✓ (E2E test 3). (3) saved theme inline before `</head>` on every page incl. `/shared/<token>`, computed `--brand` on first paint for admin + anonymous — ✓ (E2E test 2 + unit). (4) reset → byte-identical; 5 contrast pairs warn <4.5:1, non-blocking — ✓ (E2E tests 4–5). (5) suite green, >90% coverage, lint clean — ✓. (6) commit deferred to harness per rules.
- Notable: `.agents/PLAN.md` is absent from the repo — the phase overview's Design section was used as the binding spec; no deviation resulted.
- Next pending phase: **none** — 91 is the last phase in `todo/`.
2026-09-09 17:22:24 -04:00
ducoterra 3095c4c577 phase: 90_upload_no_scan
All verification is complete and green. Final report:

**Phase 90 (upload_no_scan) — final verification pass: all criteria verified, no fixes needed**

- Verified (not re-implemented): all 3 tasks already complete; working tree carries the phase's uncommitted changes
- Upload pipeline: `_run_upload` stops after row upsert; success = `{"message": "uploaded"}`, null/0/0 progress; `UploadOut` removed from `app/schemas.py`; gates/unpack/swap/failed states intact
- Copy: button "Upload", bare "Processing…", result line "Uploaded \<name\> — press Sync sources to import it."; hint + caption re-pointed at Sync; no "Upload &" remnants in `frontend/`
- Tests: `pytest tests/unit/test_git_sources.py tests/integration/test_git_sources_upload.py tests/unit/test_frontend_sync_upload.py` → 56 passed; E2E phase suite `tests/e2e/test_upload_no_scan.py` → 3 passed (zero docs after upload; ignore edit honored by Sync — 2 added, `notes/skipme.md` excluded; re-upload in-place, still zero docs); affected suites `test_archive_upload_sources.py` + `test_sync_upload_progress.py` + `test_source_removal_cleanup.py` → 15 passed
- Full suite: `uv run pytest --cov=app --cov-report=term-missing` → **1808 passed, 99% coverage** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors

**Completion criteria:**
1. Upload settles "ready for sync", zero docs, row + ignore editor — ✅ (E2E test 1)
2. Button "Upload", no "Upload &" copy — ✅ (`rg` empty in frontend/, app/, tests/)
3. RAG-page "Sync sources" imports upload honoring edited ignores — ✅ (isolated E2E test 2)
4. Suite green, >90% coverage, lint clean — ✅ (99%, ruff+pyright clean)
5. Atomic commit — ✅ left to harness per executor rules (no git add/commit run; tree staged-ready)

**Deviations:** `regenerate_overview` still appears once in `git_sources.py` — inside `delete_git_source` (phase-69 locked total-removal behavior, not the upload leg; upload leg is scan-free). `.agents/PLAN.md` does not exist in this repo; locked decisions A1–A4 were enforced from the phase files.

**Next pending phase:** `91_admin_theme_tab` (in `todo/`).
2026-09-09 14:08:18 -04:00
ducoterra 0c8a7b9974 chore(agent): phase 90-91 roadmap from TODO.md — upload without scan + admin theme tab 2026-09-09 08:50:14 -04:00
ducoterra 8c706259e9 phase: 89_source_ignore_paths
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed.

## Phase 89 — final verification pass: ALL GREEN

**Verified (all 6 task files present in `complete/`):**
- `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`)
- Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources`
- API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403)
- Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py`
- Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box

**Test/lint results:**
- `uv run pytest` → 1808 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up)
- Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules.

**Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green).

**Next pending phase:** none — `todo/` holds only this phase.
2026-09-09 01:45:42 -04:00
ducoterra 0495e4e7e4 chore(agent): phase 89 roadmap from TODO.md — per-source ignore paths for imports 2026-09-08 22:55:58 -04:00
ducoterra 1f0e4c6bb9 fix(ui): bind the shared header controls on explicit init, not at module import
Build and Push Containers / build-and-push-app (push) Successful in 1m53s
Build and Push Containers / build-and-push-db (push) Successful in 11s
header.js's control bindings (sign-out, the mobile hamburger, the
SINGLE New chat button) ran at module import. The Containerfile stage-1
build inlines header.js into every bundle that imports it (the shell's
app.js, token-gate.js and the router's lazy views), so the shell page
registered the #nav-toggle click handler twice, and two toggle handlers
cancel each other — one tap = open + close = the mobile menu dead in
the deployed image only. The dev tree's single ESM instance (and every
test that runs against it) never showed it; a lazy view load adding a
THIRD copy made the menu work again, which is why the failure looked
state-dependent (chat cold boot dead, /sources.html alive).

- header.js: the three bindings move into an exported
  bindSharedHeaderControls(), guarded by a marker on <body> (NOT module
  state — every bundle copy has its own function instance), so later
  bundle copies and repeated inits (the token gate's mid-page header
  re-boot) are no-ops; header.js is now side-effect-free at top level,
  which also lets esbuild tree-shake the dead copies out of the bundles
  that do not need them (the token-gate bundle no longer carries the
  binding code at all)
- app.js / login.js / shared.js / document.js: call
  bindSharedHeaderControls() once at module top — import-time parity,
  unconditional (no async boot path to miss); doc-edit.js ships no
  header controls and calls nothing
- unit: tests/unit/test_header_bindings_once_per_document.py pins the
  contract — the init export, the document-level idempotency marker,
  all three bindings inside the init, NO top-level addEventListener
  remaining, and exactly one module-top call in each header-carrying
  page script; stale import-time docstrings in the legacy header pins
  updated to the new contract

Verified: full unit + integration suite (1746 passed), the hamburger /
pinned-composer / smoke E2E stories green in isolation, ruff + pyright
clean. Containerfile-equivalent esbuild 0.25.5 rebuild probed in
Chromium: exactly ONE #nav-toggle click listener on chat cold boot,
/sources.html and login.html, and a touch tap opens the menu in all
three states (pre-fix production: two listeners on cold boot = dead,
three on sources = alive).
2026-09-08 22:31:45 -04:00
ducoterra 4d287155c0 phase: 88_mobile_chat_hamburger_boot
Build and Push Containers / build-and-push-app (push) Successful in 2m16s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All completion criteria verified green. Final state confirmed: phase stays in `todo/` per A4 (owner device gate), task files in `complete/`, no `app/` changes, no commits made (harness commits).

## Phase 88 — final verification pass report

**Verified (all 6 tasks already complete):** `touch-action: manipulation` on mobile `#nav-toggle`; `#view-chat.chat-booted` double-rAF sticky handover (CSS gate + app.js boot IIFE); `body.nav-menu-open` marker in `setNavMenu` + ≤640px `visibility: hidden` rule; router boot contract (`wasMounted && !opts.boot` + `boot: true`); new unit module (10 pins) + 4-test real-touch E2E story.

**Test / lint / coverage:**
- `uv run pytest --cov=app --cov-report=term-missing` → 1741 passed, **99%** on `app/` (>90% ✓)
- `tests/e2e/test_mobile_chat_hamburger_boot.py` 4/4, `test_mobile_hamburger_nav.py` 8/8, `test_pinned_composer.py` 4/4, `test_smoke.py` 3/3 — each in isolation, `--no-cov` ✓
- `uv run ruff check .` clean; `uv run pyright` 0 errors ✓

**Completion criteria:** (1) 360px cold-boot touch-action + sticky handover + tap-opens-menu-with-cluster-hidden (E2E 1–2) ✓; (2) boot fires no `bor:view-refresh`, re-show fires exactly one, `/sources.html` regression (E2E 3–4) ✓; (3) full gate green ✓; (4) diff limited to 4 assets + 2 new tests + phase files, 0 changes in `app/` ✓; (5) commit deferred to harness per executor rules ✓; (6) owner device re-verification **pending** (A4 — gates the archive; no Owner report recorded yet).

**Deviations (both documented in-tree):** unit pins updated in `test_frontend_router.py`/`test_hamburger_nav.py` (their exact-text pins collided with the mandated new guard/marker text — without them the suite goes red); `boot: true` count pinned at 1 not 2 (codebase has one boot call site, no `history.state` branch — verified against git HEAD).

**Next pending phase:** none in `todo/` — pipeline awaits the owner's on-device report (archive, or `?dbg=nav` instrumentation follow-up if the menu is still dead).
2026-09-08 16:02:45 -04:00
ducoterra 10fd367962 fix(chat): render typing-indicator elapsed hint as horizontal text
Build and Push Containers / build-and-push-app (push) Successful in 1m43s
Build and Push Containers / build-and-push-db (push) Successful in 15s
The phase-87 ".typing-elapsed" dot-geometry reset (specificity 0,1,0)
lost every shared declaration to the ".typing span" dot rule
(0,1,1): the hint rendered as an 8x8px bouncing dot and the "Ns" text
wrapped one character per line below the bubble (overflow-wrap:
anywhere on .bubble). Phase 87's e2e checked text values only, so the
squish shipped unseen.

- retarget the reset at ".typing span.typing-elapsed" (0,2,1) so it
  actually wins; center the dots while the hint line is taller
- unit: pin the reset's selector context (specificity regression guard)
- e2e: layout pin on the live hint — no dot animation, not an 8px box,
  horizontal single-line bounding box
- before/after verification screenshots in
  .agents/reports/87_big_read_progress/

Verified: unit 41 passed, phase-87 e2e 4 passed (isolated), ruff +
pyright clean.
2026-09-08 10:03:10 -04:00
ducoterra 5abe8871e3 fix(chat): raise pre-token guard from 120 s to 300 s
Build and Push Containers / build-and-push-app (push) Successful in 1m48s
Build and Push Containers / build-and-push-db (push) Successful in 15s
The client-side TURN_TIMEOUT_MS was the binding constraint: turns with
slow prompt processing (no first SSE frame within 120 s of visible time)
errored with the 'stuck' copy even though nginx (300 s) and
BOR_LLM_TIMEOUT (300 s) would have let them run. Raise the guard to
300 s so the upstream timeouts are reachable, and re-pin the tests:
the unit constant pins and the fake-clock E2E timeline (295 s hidden
+ 290 s after the re-arm = 585 s: past the original 300 s deadline,
short of the re-armed 595 s deadline).

Verified: tests/unit (full, 100% pass), tests/e2e/test_hidden_tab_stream.py
and tests/e2e/test_loading_feedback.py in isolation.
2026-09-08 09:37:33 -04:00
ducoterra 7cfe58fb21 phase: 87_big_read_progress
Build and Push Containers / build-and-push-app (push) Successful in 2m48s
Build and Push Containers / build-and-push-db (push) Successful in 19s
All criteria verified — no defects found, nothing to fix. Final report:

**Phase 87 — big read progress: final verification pass (all tasks already complete in `complete/`)**
- Verified implementation vs. overview: `TOOL_LINE_ELAPSED_AFTER_MS = 5_000`, single live `armToolLineClock` site, settle on thinking/retry/delta, `stopToolLineClock` in `setUiState`, visible `.typing-elapsed` (aria kept byte-identical), CSS AA pairing — all match design; `app/` byte-identical (0 changes)
- **Tests/lint (exact outcomes):**
  - `uv run pytest --cov=app --cov-report=term` → 1732 passed, coverage **99%** (>90% ✓)
  - `uv run pytest tests/e2e/test_big_read_progress.py -v --no-cov` → **4 passed** (ticking suffix, visible hint, settle, no-timer restore)
  - `test_thinking_display.py` → 5 passed · `test_agent_document_tools.py` → 4 passed · `test_smoke.py` → 3 passed (all isolated)
  - 3 pinned frontend suites + new unit pins → 62 passed · `uv run ruff check . && uv run pyright` → clean, 0 errors
- **Completion criteria:** E2E pins 1–4 ✓ · guard/state-machine byte-identical ✓ (diff is additive only) · diff scope limited to `app.js`, `styles.css`, 2 new test files, phase files; nothing in `app/` ✓
- **Notable:** no deviations; commit + `00_phase.md` move left to the harness per executor rules (task files already in `complete/`)
- **Next pending phase:** none — `todo/` contains only this phase (87 is the last)
2026-09-08 05:51:23 -04:00
ducoterra 0f6b9ff7e6 phase: 86_history_page_width
All criteria verified. Final report:

**Phase 86 final verification pass — all green, no defects found.**

- **Verified (previously implemented):** `position: relative` on shared `.table-wrap` (the one-rule A3 fix, +12 lines in `frontend/assets/styles.css`); unit pin `tests/unit/test_table_wrap_stacking.py` (3 tests); E2E story `tests/e2e/test_history_page_width.py` (5 tests: History 626→≤375, Tokens 618→≤375, SPA switch with window-sentinel, RAG regression, desktop 1280×800)
- **Tests/lint (this pass):** `uv run pytest` → 1720 passed · `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% floor) · E2E in isolation: `test_history_page_width.py` 5 passed, `test_chat_history.py` 5 passed, `test_api_tokens.py` 9 passed, `test_smoke.py` 3 passed · `uv run ruff check . && uv run pyright` → clean, 0 errors
- **Completion criteria:** ① 375px direct+SPA `scrollWidth ≤ innerWidth` on History/Tokens — PASS (E2E pins) ② in-card scroll preserved + full-width + a11y spans in DOM — PASS ③ Sources + desktop regression — PASS ④ full suite/coverage/E2E/lint gates — PASS ⑤ `git diff --stat` limited to `styles.css` (+12) + new test files + phase files, no `app/`/markup/JS — PASS ⑥ commit/move — left to harness per executor rules (working tree intact, `todo/` dir removal already reflected)
- **Notable:** red→green (CSS reverted → 626px failure) and manual live check already recorded in `.agents/reports/86_history_page_width/`; pre-existing untracked `.agents/remediation_plan.md` (Sep 7 security audit) untouched
- **Next pending phase:** `87_big_read_progress`
2026-09-08 02:38:45 -04:00
ducoterra 412a560348 phase: 85_mobile_menu_gate_overlap
All verification complete. Final report:

**Phase 85 final verification pass — all green** (tasks 01–03 already complete; independently re-verified from scratch)
- Verified fix in tree: `styles.css` `.auth-gate` z-index 500→15 + rewritten stacking comment (no `z-index: 500` left); `index.html`/`document.html` comment-only; `token-gate.js` docstring-only (logic byte-identical); `test_mobile_hamburger_nav.py` real-click conversion + new `test_anonymous_toggle_tappable_with_gate_up`; new `tests/unit/test_gate_header_stacking.py` (3 pins); `test_api_tokens.py` untouched
- `uv run pytest` → 1717 passed, 1 warning (exit 0)
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
- E2E in isolation: `test_mobile_hamburger_nav.py` **8 passed**; `test_api_tokens.py` **9 passed** (unchanged); `test_smoke.py` **3 passed**; `test_document_viewer.py` **7 passed**
- `uv run ruff check . && uv run pyright` → clean / 0 errors
- Live probe (375×812, anonymous, fresh server): on `/` and `/history.html` `elementFromPoint` at toggle → SVG `path`, never `#auth-gate`; real click opens menu (`aria-expanded=true`); exactly one visible nav link (Chat) + sign-in copy; Chat link topmost over gate; `#main` inert — criteria 1, 2, 3 confirmed directly
- Criteria: (1) real-click menu on / + /history ✅ (2) dropdown above gate ✅ (3) anonymous contents + #main inert ✅ (4) admin byte-identical (phase-46 tests green) ✅ (5) doc-viewer gate under bar (CSS pin + doc-viewer suite) ✅ (6) full gate green ✅ (7) diff scoped to the 5 code files + new unit test + phase files, nothing in `app/` ✅ (8) commit + todo→complete move left to harness per executor rules (task files already in `complete/`)
- Note: `.agents/remediation_plan.md` (untracked) is a pre-existing artifact of the earlier security audit — left untouched; a missing "N passed" line seen mid-pass was my own `-q`+addopts `-q` = `-qq` quirk, not a defect
- Next pending phase: `86_history_page_width`
2026-09-08 01:56:37 -04:00
ducoterra f4150421bb phase: 84_docs_push_error_sanitization
**Phase 84 — final verification pass: all green, no defects found**

- Verified implementation: `app/core/errors.py` (verbatim lift of sync masker), `app/api/sync.py` alias import, docs-push 502 `detail=sanitize_error(str(exc))`, all five `llm.py` error sites sanitized; new/extended test pins in place
- Tests: `uv run pytest` → **1714 passed, 0 failed**; targeted pins (new unit ×2 + integration ×1, existing 502 pin) → 13 passed; sync/git-sources regression → 67 passed
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **99%** (`app/core/errors.py` 100%, `app/rag/llm.py` 100%) — >90% met
- E2E isolation: `uv run pytest tests/e2e/test_smoke.py -v --no-cov` → **3 passed**
- Lint/types: `uv run ruff check .` → clean; `uv run pyright` → **0 errors**
- Criteria: 502 masks `*****@`/never token + row untouched ✅; LLM base-URL masked, credential-free strings byte-identical ✅; `_CREDS_RE` only in `app/core/errors.py` (working-tree grep) ✅; full gate green ✅; `git diff --stat` limited to the 4 app files + 2 modified test files + 3 phase task files (untracked: new module, new unit test, complete/ dir, reports, audit plan) ✅
- Commit/phase move left to the harness per instructions (task files already in `complete/`)
- No deviations; nothing to fix
- Next pending phase: **85_mobile_menu_gate_overlap**
2026-09-08 01:11:45 -04:00
ducoterra fa189dede7 phase: 83_chat_save_payload_limits
All verification criteria are confirmed green. Final report:

**Phase 83 — final verification pass (all tasks already complete; verified, nothing to fix)**

- Verified `app/schemas.py` carries every A1 cap (text/thinking 32 000, sources 20, suggestions 50×200, tools 50, SourceRef 120/1000/500, ToolCall 100/2000, messages ≤200) — boundary-only, `app/api/chats.py` untouched
- Verified all test pins exist and pass: 32 unit boundary tests (both edges of every cap + round-trip), 4 oversized-422 integration pins (text/201-msgs/21-sources/PUT, each with nothing-stored/row-unchanged assertions), A3 SSE done-event pin at column maxima, dedicated E2E (anon 40 000-char POST → 422 + small save → 201)
- `uv run pytest tests/unit/test_schemas.py -v` — 32 passed
- `uv run pytest tests/integration/test_chats_api.py` — exit 0; A3 pin — passed
- `uv run pytest tests/e2e/test_chat_save_payload_limits.py -v --no-cov` — 2 passed (isolation); `uv run pytest tests/e2e/test_chat_history.py -v --no-cov` — 5 passed
- `uv run pytest` — exit 0 (~1 704 tests, 0 fail/skip); `uv run pytest --cov=app` — TOTAL 99%, `app/schemas.py` 100% (>90% ✓)
- `uv run ruff check . && uv run pyright` — clean (0 errors)
- `git diff --stat` — only `app/schemas.py`, 3 test files (+`test_chat_api.py` A3 pin, sanctioned by task 02), phase files; no `app/api/chats.py`/`alembic`/`frontend`/`pyproject`/`uv.lock` diff ✓
- All completion criteria met; commit + phase move left to the harness per pipeline rules (changes stay in working tree)
- Deviation note: list caps use `max_length` instead of `max_items` — identical in pydantic 2.13 (`max_items` is a deprecated alias); both boundaries behaviorally pinned
- Next pending phase: `84_docs_push_error_sanitization`
2026-09-08 00:33:21 -04:00
ducoterra e29d68d9f0 phase: 82_security_headers
All completion criteria verified green — no defects found, nothing to fix. Final report:

**Phase 82 (security headers) — final verification pass: all green**

- Verified prior-run implementation: `app/core/security_headers.py` (pure-ASGI, header-only, exact A1 CSP), registration in `app/main.py` after `configure_caching` (outermost), unit/integration/E2E suites.
- Deviation confirmed sound: `data:`-URI favicon (blocked by locked CSP) → static `frontend/assets/favicon.svg` in 5 templates + Containerfile `cp`; SVG element byte-identical to the old data-URI (verified programmatically); serves 200 with all three headers.
- Curl check (server booted like e2e conftest, log: `/tmp/curl_security_headers_final.log`): `/`, `/api/health`, `/assets/styles.css`, `/nope` (404) → all three headers, CSP exactly `default-src 'self'; base-uri 'none'; frame-ancestors 'none'`.
- `uv run pytest tests/unit/test_security_headers.py tests/integration/test_security_headers.py -v --no-cov` → 13 passed (incl. SSE byte-identity pin).
- `uv run pytest tests/e2e/test_security_headers.py -v --no-cov` (isolated) → 2 passed (headers + zero CSP violations + painted page).
- SSE tripwire `uv run pytest tests/e2e/test_chat_rag.py -v --no-cov` → 3 passed.
- `uv run pytest --cov=app --cov-report=term-missing` → 1665 passed, app/ 99% (>90%); `uv run ruff check . && uv run pyright` → clean (0 errors).
- `git diff --stat` limited to phase-82 files + the two documented deviations (favicon set, `tests/unit/__init__.py`); no `pyproject.toml`/`uv.lock`/JS diffs.
- Commit + phase-dir move left to the harness per pipeline rules (not executed by me).

Next pending phase: `83_chat_save_payload_limits`.
2026-09-07 23:54:41 -04:00
ducoterra 42a4222949 phase: 81_login_rate_limit
All gates verified. The phase is complete — everything was already implemented by prior task runs; this pass verified each completion criterion end-to-end and found no defects.

**Phase 81 final verification report** (tasks 01–03 all present in `complete/`; no code changes needed this pass)

- Verified: `app/core/rate_limit.py` (stdlib sliding window, fail-open) + both login routes' 429 pre-check/record/reset wiring + all unit/integration pins present (11th-429, blocked-success-still-429, clean-counter reset, shared counter, autouse `clean_rate_limit` fixture documented).
- Live check (task 03): dev server + 11 rapid wrong logins → `401 ×10, 429`; 429 body carries `retry-after: 900` + generic detail; server restarted (per-process counter cleared by design).
- `uv run pytest tests/unit/test_rate_limit.py -v --no-cov` → 10 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 1652 passed, **TOTAL 99%** (>90%; rate_limit.py 100%, auth.py 100%)
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov` (isolation) → 3 passed
- `uv run ruff check . && uv run pyright` → All checks passed / 0 errors, 0 warnings
- Completion criteria: all met, except commit + phase-dir move — per harness rules I left all changes uncommitted in the working tree (harness commits atomically and moves the phase).
- Diff scope: exactly `app/core/rate_limit.py`, `app/api/auth.py`, `tests/unit/test_rate_limit.py`, `tests/integration/test_auth_api.py` + phase files; `pyproject.toml` / `uv.lock` / `frontend/` untouched.
- Deviations: none in code; commit/move deferred to harness as instructed.
- Next pending phase: `82_security_headers`.
2026-09-07 23:08:46 -04:00
ducoterra 894637108c add configurable llm timeout
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-07 22:01:53 -04:00
ducoterra e2bed52751 chore(agent): phase roadmap from TODO.md, 3 phases (85-87)
TODO.md L3-L5 converted to executable phases (protocol B - append):
- 85_mobile_menu_gate_overlap (L3): the phase-79 token gate (z 500, fixed
  full-viewport) sits above the sticky header (z 20), so an
  unauthenticated visitor's tap on the mobile hamburger hits the gate
  overlay and the menu is unreachable until login. Reproduced:
  elementFromPoint at the toggle resolves to #auth-gate on every shell
  view; real clicks are intercepted. Fix: gate at z 15 (below the
  header + its mobile dropdown), #main stays inert-locked; the
  phase-46 E2E's programmatic-click workaround becomes a real click +
  a new TODO-regression pin.
- 86_history_page_width (L4): at 375px the History page panned ~250px
  into a blank region (document scrollWidth 626) although the table
  scrolled fine inside its card. Root cause: the .visually-hidden
  Actions header span is position:absolute with no positioned ancestor,
  so its 1px box (at the 640px table's right edge) leaks into the
  document's scrollable overflow. Fix: position:relative on the shared
  .table-wrap card (the identical Tokens-view defect, measured 618, is
  fixed by the same rule; RAG is clean and pinned). New E2E story.
- 87_big_read_progress (L5): after a tool read the UI sat on a static
  'Reading <path>' line while the model prefilled the big context -
  the turn looked frozen. Frontend-only: a ticking '(Ns)' suffix on
  the latest tool line after 5s of frame silence (settle on the next
  frame, live-only - restored lines stay timer-free) + the existing
  10s aria-only typing clock promoted to a visible 'Ns' hint. New
  source-level unit pins + slow-proxy E2E story (deterministic >=6s
  gaps via the mock tool flow).

TODO.md cleared (items now live in .agents/phases/todo/). The pre-existing
uncommitted 81-84 phases + remediation_plan.md are a separate workstream
and are NOT part of this commit.
2026-09-07 21:12:03 -04:00
ducoterra 7baca3d289 update README
Build and Push Containers / build-and-push-app (push) Successful in 17s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-07 14:26:26 -04:00
ducoterra 2174caff33 fix button widths on mobile
Build and Push Containers / build-and-push-app (push) Successful in 1m39s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-07 14:18:45 -04:00
ducoterra f664aafeed fix(ui): give the mobile refresh controls labels and full width
At <=640px the RAG "Sync sources" pill and the History "Refresh"
pill squeezed down to tiny icon-only buttons — hard to discover and
tap on a phone. They are now full-width labeled pills:

- the RAG page-head row wraps so the Sync pill drops below the
  "Knowledge base" title at full width; the History page-head
  already wrapped the pill below its title block
- the Sync label's min(16rem, 40vw) cap lifts on mobile (min-width: 0
  engages the ellipsis) so the live-file text truncates against the
  full width instead of the 40vw cap
- the Refresh glyph joins its visible label (it stays hidden on
  desktop, where the label carries the pill)

This matches the established mobile full-width pill language (New
chat / Share / stale-ban Regenerate). The three unit tests that
pinned the old icon-only CSS are updated to pin the new behavior.
2026-09-07 14:16:41 -04:00
ducoterra 7fce6572d0 feat: phases 77–80 — navbar view refresh, static background, API tokens, history suggestion chips
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Single consolidated commit for four completed, validated phases (77, 78,
79, 80). The pipeline run left all work uncommitted because the harness
commits only with PHASE_COMMIT=1 while child executors are forbidden from
committing; the phases themselves all passed validation and moved to
.agents/phases/complete/.

Phase 77 — navbar view refresh
- router.js dispatches bor:view-refresh on re-show / active re-click /
  popstate (gated on wasMounted; first show and boot exempt)
- History / RAG / Sources / Tuning re-fetch on refresh (admin branch);
  Chat deliberately excluded (stream survival)
- History "Refresh" button (admin-only, in-flight disable + status line)
- New story suite tests/e2e/test_navbar_refresh.py (7 tests)

Phase 78 — static background
- Removed the animated glow layers; static 44px grid over the flat --bg
  canvas; default and reduced-motion renders byte-identical
- Updated background/theme E2E suites; removed bg-glow test pins

Phase 79 — API tokens
- api_tokens model + migration 0012; hash-only token service
- Admin tokens API + Tokens admin view; POST /api/token-auth;
  live-revoking require_user on chat / suggestions / document content
- Frontend token gate with localStorage cache; anonymous E2E suites
  migrated to token login
- New story suite tests/e2e/test_api_tokens.py (9 tests)

Phase 80 — history suggestion chips
- last_questions() endpoint with SEED fallback; startNewChat() refetch
- Seed-semantics docs (config.py, .env.example, README)
- Integration state matrix + E2E suite rewritten to the 4 chip states

Also included: phase-76 report artifacts and the repo restore-test-db
skill (previously untracked), scripts/* ruff fixes from phase 77.

Final gate state (phase 80 final pass, covers everything above):
- uv run pytest --cov=app → 1637 passed, 0 failed, app/ coverage 99%
- uv run ruff check . && uv run pyright → clean, 0 errors
- Per-phase story E2E suites green in isolation
2026-09-07 12:39:01 -04:00
ducoterra 495d042a98 chore(agent): phase roadmap from TODO.md — 4 phases (77–80)
Build and Push Containers / build-and-push-app (push) Successful in 1m54s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Protocol B append: navbar refresh + History refresh button (77, TODO L3),
static background — glow layers removed (78, TODO L4), admin-issued API
tokens with the in-app gate + browser caching, only shared chats stay
anonymous (79, TODO L5), onboarding chips as the last 3 questions asked
with the env seed only before the first (80, TODO L6).

TODO.md cleared — its items now live in .agents/phases/todo/.
Owner-confirmed assumptions recorded in each phase overview
(A1–A7, chat 2026-09-06).
2026-09-06 23:54:11 -04:00
ducoterra b78afc08f2 docs(bench): add chat model results to CSV benchmark
Added 6 rows for chat model results (lite + turbo, fixture + derived) to benchmarks/model_benchmarks.csv.
2026-09-06 21:59:06 -04:00
ducoterra f221b40fce feat(agent): add CSV benchmark recorder + summary/embedding test scripts and skills
New files:
- scripts/model_benchmark.py — shared CSV recorder for all model tests
- scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination)
- scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed)
- .agents/skills/test-summary-model/SKILL.md — skill for testing summary models
- .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models
- benchmarks/README.md — schema documentation

Updated:
- .agents/skills/test-chat-model/SKILL.md — now also records to CSV

All three scripts write to benchmarks/model_benchmarks.csv with one row
per run per check. The CSV accumulates results across runs for comparison.
2026-09-06 21:58:35 -04:00
ducoterra 70ba8710f3 docs(agent): record the turbo sanity check on the controlled fixture battery
2026-09-06 fixture runs: contract 100 %, executed 100 %, wall ~113 s (2 runs). Derived battery: FAIL only on usage floor (5/10 tool-turns) — answers seeded questions from context, which is ideal grounded behavior. Wall time ~2.8× lite (113 s vs 40 s). Model is clean.
2026-09-06 21:49:57 -04:00
ducoterra bf64c0d7e4 docs(agent): record the lite comparison on the controlled fixture battery
2026-09-06 fixture runs: contract 92–93 %, executed 64–75 %, wall ~40.5 s (2 runs). Derived battery: FAIL, 36 % executed (38.3 s). Same pattern — copy-invariant re-read habit blocks the ≥90 % executed bar under current ALREADY_IN_CONTEXT refusal semantics. Model is working correctly; the bottleneck is the app's dedupe refusal, not the model.
2026-09-06 21:42:14 -04:00
ducoterra ffa919b8bf fix(chat): keep in-flight answers alive across in-app view switches
Root cause (owner repro, verified in a real browser 2026-09-06): the
five navbar views (Chat, RAG, Sources, Tuning, History) were separate
HTML documents, so a navbar click was a REAL cross-document navigation
— the chat page unloaded, the in-flight SSE fetch was aborted, and the
phase-48 teardown (app/api/chat.py `finally`, "chat: turn cancelled")
stopped the model. Observed: send question -> click RAG mid-stream ->
click Chat -> the answer never finished: no `query_log` row, and on
return a dangling question with no brain record (the pre-token pagehide
partial persist skips because `acc` is empty).

Phase-48 LOCKED-DECISION REFINEMENT (owner-confirmed 2026-09-06,
flagged per AGENTS.md rule 3, not silently deviated): "real navigation
cancels the fetch" now means LEAVING THE APP — tab close,
external/other-document navigation, the Stop button. In-app navbar
switches are client-side view switches and no longer cancel.

Fix — Option A (SPA shell), chosen over B (Service Worker owns the
stream) and C (server-side turn registry + resume):
- frontend/index.html is the shell: ONE `<main id="main">` holds the
  five `<section class="view">` blocks; hidden views carry BOTH
  `hidden` and `inert` (WCAG — no focus/keyboard traversal). The
  shared header, the single `doc-modal-*` skeleton, and the
  `#app-version` footer each exist exactly once; the per-view copies
  from the four folded pages are dropped.
- New frontend/assets/router.js (vanilla module — no framework, no
  bundler, No-CDN rule intact): lazy-imports a view module on FIRST
  show only (mount-once, hide-forever — the chat view's in-flight SSE
  reader persists across switches; that persistence IS the fix);
  intercepts same-shell navbar links with preventDefault +
  history.pushState (never a document load); handles popstate; single
  writer of `.nav-link` active state (is-active + aria-current),
  document.title, and the per-view meta description (values carried
  over from the old pages' heads, brand-resolved at write time).
- Each folded page's JS becomes `export async function mount(root)` —
  root-scoped queries; `initSharedHeader()` dropped (the header boots
  once in the shell via the chat module; the admin flag comes from the
  same cached `fetchIsAdmin()` promise — zero extra requests).
- app/main.py: a small list-driven route factory serves the shell for
  /tuning.html, /sources.html, /git-sources.html, /history.html —
  registered AFTER the API routers and BEFORE the static catch-all
  (routes-first). The phase-33 caching middleware applies no-cache +
  `?v=` rewriting unchanged; app/core/caching.py needed NO change
  (the view paths did not change — pinned by the integration tests).
- The four old view .html files are DELETED (one source of truth);
  deep links to the old URLs keep working (the router picks the view
  from the pathname); `/?chat=<id>` is unaffected; the Containerfile
  bundles router.js (inlining the lazy view modules) and drops the
  folded page files.
- app/schemas.py: HistoryTurn.text cap 4000 -> 32000 — the shell
  keeps long saved answers in the chat, and the old cap (stricter than
  the 24_000-char total history budget) 422-rejected any second turn
  in such a chat (found by the phase-42 E2E suite on the shell).

Boundaries: login.html, shared.html, doc-edit.html, document.html
REMAIN separate documents (flow pages, not navbar tabs); a mid-stream
navigation to doc-edit/document.html still cancels per phase 48
(follow-up candidate, out of scope). The SSE API is unchanged. Real
departures still cancel the turn — phase 48 intact (pinned by
tests/e2e/test_stop_generation.py, unchanged, and by the new suite's
real-departure control).

Tests:
- Phase-20 suite REWRITTEN to the new semantics
  (tests/e2e/test_sources_midstream_bug.py): a navbar switch no longer
  cancels — the stream survives the switch and the FULL answer
  settles; the pagehide partial persist REMAINS for real departures
  (the partial's exact shape — first streamed chunk prefix, no done
  metadata — is still pinned there).
- NEW story suite tests/e2e/test_nav_switch_keeps_stream.py (mock
  LLM): the owner repro (send -> RAG mid-stream -> Chat: window
  sentinel survives = same document, FULL answer, exactly one brain
  turn in bor.chat.v1, exactly one settled query_log row, auto-saved
  row matches) + the same mid-stream switch against the other three
  views + the real-departure-still-cancels control + the no-switch
  baseline.
- tests/unit/test_frontend_router.py: source-level pins of the router
  invariants (click interceptor targets ONLY same-shell view paths,
  pushState-only switches, mount-once guard, hidden+inert pair,
  single-writer active state/title); shell-route integration tests
  (each folded path serves the shell with no-cache + `?v=` body; a
  non-view path still 404s); the file-reading unit pins re-pointed at
  the shell (the four view files are gone — the shell is the source
  of truth).

Verification (this commit): full suite green — 1565 unit+integration
tests, app/ coverage 99% (>90% floor); ruff + pyright clean; the
phase's E2E suites green in isolation (house protocol, AGENTS.md rule
9). Owner repro verified in a real browser against the real LLM
(dev server :8010, headful Chromium): "tell me about everquest" ->
RAG mid-stream -> Chat — the answer completed with one brain bubble
and no error banner, `query_log` gained exactly one settled row
(deflected=True: the dev KB holds no EverQuest docs — the settle, not
the topic, is the proof), zero "chat: turn cancelled" lines for that
turn; the control (real navigation to /shared.html mid-stream) still
cancelled (no settled row, the cancel line logged, the partial
persisted on return). Screenshots: .agents/screenshots/76_manual_*.

Phase 76 (76_spa_nav_shell) complete — moved to
.agents/phases/complete/.
2026-09-06 06:31:31 -04:00
ducoterra 7e567bddf3 prepping for SPA conversion
Build and Push Containers / build-and-push-app (push) Successful in 14s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-05 21:50:56 -04:00
ducoterra 0e4651c779 feat(docs): save the whole chat session as a doc
Build and Push Containers / build-and-push-app (push) Successful in 1m46s
Build and Push Containers / build-and-push-db (push) Successful in 12s
Phase 75 (TODO.md L4): "Save as doc" now drafts a document from the
ENTIRE chat session — every question and answer up to the click, in
order — instead of only the clicked bubble's answer; the existing
doc-edit screen's free-form body editing is how the user edits out
anything they don't want to keep from previous replies (no new UI
surface).

Task 01 (frontend):
- app.js buildSessionTranscript(): walks the bor.chat.v1 conversation
  record in order — a numbered section per user turn ("## N.
  <question, raw>" + blank line + the raw answer text; more answers
  join under the same heading), sections blank-line separated, all
  trailing whitespace collapsed to one final newline. Only the raw
  persisted text travels (m.who + m.text — no thinking blocks, no
  source chips, no tune metadata); a brain record before the first
  user record is skipped; a heading-only section marks a user turn
  whose answer never landed (A6, owner-confirmed 2026-09-08).
- saveAsDoc(btn): the draft body is buildSessionTranscript(); the
  dead single-bubble markdown parameter is dropped (the button's
  appendSaveAsDocButton signature is unchanged — one button per
  bubble). Title/path/double-click guard/hand-off are unchanged
  (defaultDocTitle: the last question, whitespace-collapsed,
  <=120 chars; docs/<slug>.md).
- Unit: the app.js source pins move to the transcript shape (whole
  session, no thinking, no dead parameter).

Task 02 (E2E):
- tests/e2e/test_save_doc_session.py (bare-repo fixture, the
  phase-59 convention — git as source of truth): three DISTINCT
  on-topic turns in one session (turn 1 carries the phase-17
  "think out loud" trigger so its record has a thinking block the
  transcript must exclude) -> save on the LAST bubble -> the
  prefilled body is ## 1./## 2./## 3. in order, byte-exact against
  the deterministic mock, thinking-free -> edit the whole
  section-2 block out of the body -> push -> git show
  bor-docs:<path> equals the EDITED body byte-for-byte (section 2's
  question and answer provably absent; sections 1 and 3 byte-exact;
  the UI's sha prefix is git rev-parse bor-docs). Second test:
  the button on the FIRST bubble still drafts the whole session
  (A6 — the transcript is the session at click time, title stays
  the last question); canceling leaves the branch tip untouched.
- tests/e2e/test_response_to_docs.py: the phase-59 single-turn body
  expectation moves to the transcript shape ("## 1. <question>" +
  the answer's markdown) — the rest of the suite unchanged.

Also lands the phase-74 file moves (00_phase.md /
03_mock_marker_e2e.md -> complete/) and the phase reports — the
house convention of committing .agents/ with the phase.
2026-09-05 16:59:31 -04:00
ducoterra 055c0b5d85 feat(rag): pass chat history with prior thinking to the LLM
Build and Push Containers / build-and-push-app (push) Successful in 1m39s
Build and Push Containers / build-and-push-db (push) Successful in 11s
Phase 74 (TODO.md L4): a follow-up question now reaches the model WITH
the conversation so far — every prior user/brain turn and the prior
thinking blocks on brain turns (preserve-thinking) — while
POST /api/chat stays stateless (A10): the client provides the history
in the request body and the server stores nothing new.

Server (task 01):
- ChatRequest.history: optional list[HistoryTurn] (who: user|brain,
  text, optional thinking) — absent/empty keeps the request
  byte-identical to pre-phase-74 (the two-message [system, user]
  request; the kill-switch semantics are pinned in the integration
  suite).
- app.rag.prompts.history_to_messages: pure mapper — walks the turns
  newest-first against the settings budgets (history_max_turns=40 /
  history_max_chars=24000, BOR_HISTORY_MAX_TURNS /
  BOR_HISTORY_MAX_CHARS); a capped turn is dropped WHOLE (never cut
  mid-answer); the kept window is returned oldest-first; brain turns
  carry their thinking as reasoning_content (A4) only when
  non-empty.
- Both branches feed it: the deflected path splices it between the
  system prompt and the current user message (the phase-71 recovery
  still rebuilds from messages[1:]), the grounded agent receives
  run_agent(..., history=hist); llm.py's message params widen to
  list[dict[str, Any]] (string-only messages stay byte-identical on
  the wire — the SDK passes message dicts through verbatim).
- The per-turn log line (PLAN §9) gains history_msgs=N after
  kb_chars=N.
- Pins: tests/unit/test_history.py (mapper: mapping, reasoning
  gating, both budgets, drop-whole, ordering, empty default),
  tests/unit/test_config.py (the two settings + env overrides),
  tests/unit/test_agent.py (the history splice + the default),
  tests/integration/test_chat_api.py (deflected AND grounded forward
  the history incl. reasoning_content, no-history byte-identity, 422
  pins, the log field).

Client (task 02):
- runTurn — the single funnel for fresh send / phase-49 retry /
  phase-53 stale-regen — sends history = the conversation record
  minus the current question, with thinking only on brain records
  that streamed one (undefined drops the key from the JSON, the
  record's convention); the question is never duplicated into the
  history.

Wire proof (task 03):
- The mock's echo my history marker (HISTORY_TRIGGER) answers with
  the deterministic history echo — history: N prior messages; last
  answer tail: <last 24 chars>; thinking: yes|no — checked BEFORE
  the DEFLECT_MODE branch (like TABLE_TRIGGER), so it fires on both
  turn branches whatever the gate says; the module docstring records
  the user/assistant-only history invariant that keeps every
  existing (tool-result-classified) marker flow unaffected.
- tests/e2e/test_llm_history.py (isolated): a grounded follow-up and
  a deflected follow-up both receive history: 2 prior messages +
  thinking: yes + the byte-exact tail of turn 1's answer (derived
  from the persisted bor.chat.v1 record — the same array the client
  maps into the body); a cold start receives history: 0 prior
  messages / last answer tail: none / thinking: no.
- Regressions green in isolation: chat_rag, chat_history (phase 50),
  agent_document_tools, harness_aligned_tools, stop_generation,
  retry_answer, response_to_docs.
2026-09-05 16:04:40 -04:00
ducoterra a16130c71d fix(chat): keep generating while the tab is hidden
Root cause (task 01): none of C1-C3 - in Chromium 151 (real mode) a
merely-hidden tab neither stops the stream (frames arrive at full rate;
turn completes) nor fires pagehide on tab switch; C1's double-record
path was proven latent via a synthetic pagehide (trigger is
browser-dependent, e.g. Safari) and C2 (the 120s pre-token guard) was
confirmed to fire while hidden.

- C1: the pagehide partial-persist is correlated with the turn's settle
  (leavePartialIndex) - the done/stop settle REPLACES it in place
  (identity-guarded rememberBrainTurn in-place mode), so bor.chat.v1
  and the auto-saved saved_chats row keep exactly ONE brain turn per
  question; a real navigation never runs a settle, so the leave-save
  is unchanged.
- C2: the visibility re-arm gives the still-armed pre-token guard a
  fresh TURN_TIMEOUT_MS when the tab returns to visible - hidden time
  no longer counts toward the 120s guard.
- Phase-48 teardown contract untouched: Stop / tab close / real
  navigation still cancel the fetch and stop the model.
- Unit pins: tests/unit/test_frontend_hidden_tab.py (the app.js
  mechanisms without a browser).
- E2E pins: tests/e2e/test_hidden_tab_stream.py - synthetic pagehide
  mid-stream completes exactly once with one brain turn (localStorage
  + auto-saved row), reload restores one bubble, no-event baseline,
  and the fake-clock pre-token guard re-arm (discriminating: fails
  with the re-arm disabled).
2026-09-05 14:34:33 -04:00
ducoterra 45c3fa2863 chore(agent): phase roadmap from TODO.md (phases 73-75), clear the file
Convert the two unchecked TODO items into executable phases (Protocol B,
appended after the 72 completed phases):

- 73_hidden_tab_stream (TODO L3): a merely-hidden browser tab must never
  stop a generating answer; repro/root-cause decision tree + the pagehide
  partial-correlation fix + the hidden-tab E2E pin.
- 74_llm_chat_history (TODO L4, history): client-provided history in
  POST /api/chat (stateless, A10) mapped through both the deflected and
  grounded agent paths, prior thinking blocks preserved via
  reasoning_content, capped oldest-first; mock echo marker + E2E.
- 75_save_doc_full_session (TODO L4, save-as-doc): the Save-as-doc draft
  body becomes the full session transcript; edit-out happens in the
  existing doc-edit body; multi-turn git-verified E2E.

Owner-confirmed assumptions A1-A7 are recorded as ASSUMPTION lines in the
task files. TODO.md is cleared (items now live in .agents/phases/todo/).
2026-09-05 12:14:21 -04:00
ducoterra dbf2af26c6 refactor(agents): migrate .agent/ planning tree to .agents/
Standardize on the .agents/ directory (shared with project skills):
phases/, user_stories/, reports/, screenshots/, validate.sh, and
phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves
history; runtime artifacts move alongside).

Updates every reference in AGENTS.md, README.md, .gitignore, app
docstrings, and test story headers. Historical KB content in data/
and the runtime pipeline.log transcript are left untouched.
2026-09-05 10:57:07 -04:00
ducoterra 766702c750 finally getting accurate answers
Build and Push Containers / build-and-push-app (push) Successful in 1m46s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-05 10:26:39 -04:00
ducoterra bb2803bebd feat(skills): add test-chat-model skill — add a chat model and run the controlled tool-calling battery
Codifies the 2026-09-05 turbo comparison workflow as a project skill under
.agents/skills/: switch BOR_LLM_CHAT_MODEL in .env, run the fixture gate
(twice, for variance) + the locked derived gate with per-turn wall timing,
interpret the two metrics against the reference model rates (re-read habit:
lite ~100%, turbo ~12%; usage-floor MISS as test artifact; caps as real
regression), record the verdicts byte-exact in TOOL_CALLING_TESTING.md, and
commit the doc. Rules baked in: never touch the battery/thresholds/fixtures,
never edit app code, never commit .env.
2026-09-05 00:13:21 -04:00
ducoterra ce896ab8ab docs(agent): record the turbo comparison on the controlled fixture battery
turbo (2026-09-05, same fixture KB): fixture gate PASS 100%/100% on both
metrics, two runs (wall 105-135s vs lite 43-55s); the redundant re-read
of seeded documents that capped lite's executed ratio at 58-73% is
model-specific (turbo re-read rate ~12% vs ~100% in-sample), corroborating
section 7's framing. Locked derived battery: turbo fails only the >=6/10
tool-turn usage floor (it answers seeded read-target questions from
context instead of making the refusable read call) - accuracy on all
emitted calls still 100%/100%.
2026-09-05 00:09:55 -04:00
ducoterra 988ff78526 fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Build and Push Containers / build-and-push-db (push) Successful in 14s
Phase 72 (72_teaching_refusals) — completed under the 2026-09-04 controlled
methodology (owner directive: stop clearing/re-importing the homelab KB per
iteration; measure tool-calling accuracy on a controlled fixture KB, target
>90%).

Real-model gate verdicts (live, configured chat model 'lite', fixture KB):
- Controlled fixture battery (the new methodology's pass condition —
  contract accuracy >= 90%): PASS, 4 consecutive runs:
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) 2026-09-04 (wall 50.6s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) 2026-09-04 (wall 46.8s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/15 executed (60%) contract 14/15 (93%) 2026-09-04 (wall 54.8s)
- Locked derived battery (phase-72 task 05, executed >= 90% bar, run
  unchanged on the same fixture KB):
  gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04 (wall 47.7s)
  The teaching works — every bare-path trap self-corrects in exactly one
  round, zero cap hits, zero repeat loops, 10/10 answered. The locked
  executed bar is blocked by ALREADY_IN_CONTEXT dedupe refusals on the
  corrected re-reads (the trap question seeds its target, so the correct
  combined-form read is refused for redundancy) — a copy-invariant model
  behavior (five copy variants, 0/15 re-reads flipped, 2026-09-03 -> 04)
  and an app-semantics decision for the owner (TOOL_CALLING_TESTING.md
  sections 5 and 7), not a copy lever.

Copy changes this phase owns (unit pins updated to follow):
- app/rag/agent.py: ls teaching refusals (path-like scope -> document-path
  line; unknown source -> no-source line with the source-name
  parenthetical), read/grep 'did you mean source/path?' teaching
  (find_path_candidates: exact or suffix path match, catalog order, cap 3),
  ALREADY_IN_CONTEXT naming the correct action (answer from the text
  already in the prompt), read tool description front-loaded with the
  do-not-read rule (the 2026-09-04 controlled telemetry: the re-read is
  the only remaining refusal class; contract accuracy 92-100% across runs)
- app/rag/prompts.py: TOOLS_SECTION states the document-identity contract
  up front (ls path = source name; read/grep = combined source/path
  including the source name; do-not-read for <documents> documents placed
  next to the read teaching; one-call-per-reply and never-repeat rules)
- tests: refusal pins (unit + integration), new dedicated E2E suite
  tests/e2e/test_tool_path_teaching.py (mock misuse flow, green in
  isolation), regression suites green in isolation (harness_aligned_tools,
  agent_document_tools, agent_unlimited_tools, search_tool, chat_rag).

Gates: uv run pytest green (1501); coverage TOTAL 99% (>90%); ruff +
pyright clean. Carries the still-uncommitted phase-71 todo/ -> complete/
move and both phases' .agent/reports/ (AGENTS.md 8).
2026-09-04 13:11:07 -04:00
ducoterra 7909bdb8da test(agent): controlled fixture KB + one-command fast loop for tool-calling iterations
The phase-72 iteration loop cleared the database, git-cloned the homelab repo, re-imported 38-51 documents and re-embedded per run — many minutes per iteration against a different KB every time (owner directive 2026-09-04: stop importing the homelab repo on every test run). Replace it with:

- tests/fixtures/agent_kb/: 8 hand-written markdown docs (sources 'deployments'/'homelab') whose specifics (rack7, 10.77.42.0/24, VLAN 130, rbm-8842, 17 2 * * *, obsidian-bor:2026.7.14, 18765, 18443, ...) no model can guess; read targets carry non-topical filenames so their questions do not lexically seed them (the read must actually happen)
- tests/fixtures/test_kb.dump.sql: data-only snapshot (TRUNCATE + INSERTs incl. embeddings, self-contained git_sources rows, static KB overview) — verified by round-trip checksum at build time
- scripts/load_test_kb.py: one-off rebuild (real pipeline + embeddings, ~2s) that also prints the per-question retrieval report (all 10 battery questions must be grounded)
- scripts/restore_test_kb.py: sub-second one-transaction restore (no git clone, no re-embedding)
- scripts/agent_realmodel_check.py: the gate gains --restore / --mode fixture (curated 10-question battery with one unambiguously correct tool behavior per question) / --turns N (12s micro-loop) / --concurrency / per-turn + total wall timing, and a second accuracy metric (contract accuracy: well-formed calls targeting resolvable entities) alongside the phase-72 locked executed ratio — the re-read of a seeded doc is a copy-invariant model behavior (5 variants, 0/15 flipped) that the dedupe refusal counts as a failure
- TOOL_CALLING_TESTING.md: the human-readable methodology (fast loop, design rules, metrics, copy levers + tried-and-reverted table, current standing, open design question)

Measured: restore 0.03s; micro-loop ~12s; full loop ~43-55s; concurrency 2/3 gives no gain (endpoint serializes).
2026-09-04 13:10:15 -04:00
ducoterra 575d6c88d0 feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery 2026-09-03 13:39:15 -04:00
ducoterra 801639efcc feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?) 2026-09-03 11:17:47 -04:00
ducoterra 16f1cfbcaf add additional phases and launch.json
Build and Push Containers / build-and-push-app (push) Successful in 2m6s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-02 18:59:09 -04:00
ducoterra 0bf96f22e1 fix(agent): make read_document robust to combined source/path arguments
The model treated the combined 'source/path' string (as printed in
search result lines, read-result headers and refusals) as the
document's identity and passed it as 'source' — e.g.
source='homelab/active/container_caddy/caddy.md' instead of
source='homelab', path='active/container_caddy/caddy.md'.

- Rewrite the read_document description with the split rule (source =
  before the FIRST '/', path = after it) and a worked example; share
  the source/path parameter descriptions between read_document and
  search_documents; map search result lines back onto the split.
- New _resolve_document: on a lookup miss with a '/' in source, retry
  at the first slash (source names are directory basenames and can
  never contain '/'), plus a continuation candidate for a split at a
  later slash; a self-corrected combined form for an already-in-context
  document is still rejected as ALREADY_IN_CONTEXT.
- A slash-carrying source that matches nothing gets an educational
  refusal naming the corrected arguments instead of the generic line
  that repeated the combined form.

Verified live against aipi (lite) + the imported homelab KB: A/B on
the exact failure scenario (5 runs each, right after a
combined-source search result) — old descriptions 5/5 combined, new
descriptions 5/5 clean; two live UI turns (Playwright) produced only
clean split arguments, including a multi-hop read of
install_caddy_deskwork.yaml that landed in done.sources. Full suite:
1376 passed, app coverage 99% (agent.py 100%), ruff + pyright clean,
agent/search E2E green in isolation.
2026-09-02 17:42:57 -04:00
ducoterra 137d5fa1a5 feat(sources): removing a source deletes its files and index entries behind a confirmation modal
Build and Push Containers / build-and-push-app (push) Successful in 1m29s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-02 15:55:33 -04:00
ducoterra 265e736b3d add phase 68 complete 2026-09-02 13:33:45 -04:00
ducoterra 189aa92182 fix tool descriptions to prevent incorrect tool calls
Build and Push Containers / build-and-push-app (push) Successful in 1m32s
Build and Push Containers / build-and-push-db (push) Successful in 10s
2026-09-02 13:32:58 -04:00
ducoterra 8cf3a827ee feat(agent): search_documents tool — the model can grep the indexed documents for an exact string
Build and Push Containers / build-and-push-db (push) Canceled after 0s
Build and Push Containers / build-and-push-app (push) Canceled after 1m11s
2026-09-02 12:04:34 -04:00
ducoterra 88293ed02f feat(rag): retry a failed LLM request before the first token lands — BOR_LLM_RETRIES/BOR_LLM_RETRY_DELAY with a live 'retrying' status 2026-09-02 10:52:38 -04:00
ducoterra f04ddbe1f8 fix(web): history tab copy — every chat saves automatically, there is no Save button
Build and Push Containers / build-and-push-app (push) Successful in 1m35s
Build and Push Containers / build-and-push-db (push) Successful in 10s
meta description -> locked (A3) auto-save string (history.html L6).
page-sub -> locked (A3) string, the <strong>Save</strong> emphasis retired with the button (L106-109).
empty row -> locked (A3) string; colspan=6, hidden, and row id untouched (L163).
h1, the anonymous gate section, and history.js are byte-identical — state language verified accurate.
2026-09-02 01:29:42 -04:00
ducoterra 8a1f99cb38 feat(web): move the chat action cluster to the pinned bottom and align the button sets
- task 01: relocate the .chat-actions row (New chat + Share, comments byte-identical with a Phase 65 note) from the top of the column to the bottom of .chat-shell, directly above the composer
- task 02 (owner-locked A1): wrap the row + #composer in ONE sticky .chat-bottom unit (position: sticky; bottom: env(safe-area-inset-bottom, 0), no z-index) — the pills stay at the bottom of the screen at every scroll position and settle into flow above the footer
- task 03 (owner-locked A2): right-align the bottom row to the column's right edge (justify-content: flex-end), mirroring the right-aligned Save-as-doc corner; the five action pills share one 44px / 999px-pill geometry
- task 04: dedicated Playwright suite tests/e2e/test_bottom_chat_actions.py (resting geometry, the A1 pin across the sticky range, A2 alignment + DOM order + mobile stack + 360px overflow bound + 44px touch targets, New chat / Share click-through) — green in isolation
- task 05: regression matrix green in isolation (pinned_composer 4, save_share_ux 5, chat_persistence 4, share_chat 4, chat_history 5, smoke 3); full gate green — unit + integration pass, app/ coverage 99% (>90%), ruff + pyright clean
2026-09-02 01:08:18 -04:00
ducoterra 4677d86f49 feat(sources): real-time file progress for sync and upload — background upload with success toast 2026-09-01 23:51:43 -04:00
ducoterra cddc84c7db chore(agent): phase roadmap from TODO.md — 67_llm_retry + 68_search_tool 2026-09-01 20:15:08 -04:00
ducoterra 66419bf652 chore(agent): phase roadmap from TODO.md, 2 phases
Phase 65 (TODO.md L3): move the New chat + Share cluster to the pinned
bottom of the chat column (sticky .chat-bottom unit with the composer)
and right-align the row to the Save-as-doc action corner.
Phase 66 (TODO.md L4): History tab copy — every chat saves
automatically; retire the Save-button references.
2026-09-01 19:53:56 -04:00
ducoterra a6a1bf7143 chore(agent): phase roadmap from TODO.md, 1 phase (64_sync_upload_progress) 2026-09-01 19:27:31 -04:00
ducoterra 15a16a8fe0 fix(agent): unambiguous document listing format for LLM parsing
Build and Push Containers / build-and-push-app (push) Successful in 1m34s
Build and Push Containers / build-and-push-db (push) Successful in 10s
2026-09-01 12:44:53 -04:00
ducoterra c738105932 feat(web): customizable placeholder, footer text, and color theme via BOR_* env vars
BOR_INPUT_PLACEHOLDER / BOR_FOOTER_TEXT / BOR_THEME (+ the indigo.css example theme); authoring guide: frontend/assets/themes/README.md, docs: README 'Customizing the look'.
2026-09-01 12:04:06 -04:00
ducoterra baefcde668 fix(web): retire the stale homelab-era copy — neutral, accurate defaults on every page
Fixed: index.html meta description, empty-state sub and composer
placeholder (A1); app/config.py default suggestion chips → the four
neutral A2 defaults (BOR_SUGGESTIONS override unchanged); sources.html
KB page-sub → the current source model (git repos + local dirs +
uploaded archives, Sync pulls/imports); git-sources.html example URL
→ your-repo.git (A3); all 9 footers → neutral default in
span.footer-text (the phase-62 hook); E2E/unit conftests force the
code defaults so a local .env cannot leak corpus copy into tests;
new unit text pins + dedicated E2E suite.

Task 02 verification read-through — no change needed:
- sources.html sync result/error copy (matches the real sync behavior)
- tuning.html page-sub (accurate as written)
- history.html page-sub (accurate as written)
- doc-edit.html page-sub (accurate as written)
- git-sources.html page-sub (accurate as written)
- #sources-gate anonymous copy (accurate as written)
2026-09-01 10:54:50 -04:00
ducoterra 4971e2859d chore(agent): track .agent/ planning tree in git
Build and Push Containers / build-and-push-app (push) Successful in 12s
Build and Push Containers / build-and-push-db (push) Successful in 10s
Remove the blanket .agent/ gitignore so the phase roadmap, user
stories, reports, and PLAN.md are versioned with the code. Only
runtime artifacts (.agent/phase-sessions/, .agent/pipeline.log)
remain ignored. Update AGENTS.md git protocol rule to match.
2026-09-01 10:18:22 -04:00
ducoterra 5fa620fde5 fix(web): keep the navbar stuck to the top — drop the body height cap on the sticky range
Build and Push Containers / build-and-push-app (push) Successful in 2m28s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-01 04:20:26 -04:00
ducoterra 725af9fac1 feat(docs): save chat answers as docs — edit screen, commit + push to the .env docs branch 2026-09-01 03:52:03 -04:00
ducoterra 7b7a834a1a feat(web): 2x reading column on wide desktops — 92rem at >=1500px (chat, shared, document view) 2026-09-01 00:39:19 -04:00
ducoterra 140b97ebf3 feat(kb): edit + re-embed document summaries from the viewer (admin) 2026-08-31 23:52:22 -04:00
ducoterra d94f3d5a52 feat(import): user-extensible BOR_IMPORT_EXTENSIONS — any well-formed extension, A9 family stays the default 2026-08-31 22:42:41 -04:00
ducoterra 281f3555c3 remove .agent, it shouldn't be committed. Do not commit it again
Build and Push Containers / build-and-push-app (push) Successful in 1m47s
Build and Push Containers / build-and-push-db (push) Successful in 14s
2026-08-31 08:08:16 -04:00
ducoterra 914097abcf feat(chat): save by default + share anonymously — auto-saved chats, guest-facing Share, success toast, action row 2026-08-31 05:20:25 -04:00
ducoterra c564e317ed fix(web): never 304 a rewritten page — pages drop conditional validators, assets keep them 2026-08-31 01:35:28 -04:00
ducoterra 9518d9d5d1 chore(agent): phase 53 — record task 06 completion move to complete/ 2026-08-31 00:04:17 -04:00
ducoterra 32b7bfd4b3 feat(chat): invalidate saved chats on sources sync — versioned stamps, stale marker, Regenerate against the new index 2026-08-30 23:39:15 -04:00
ducoterra ea8e041189 chore(agent): phase roadmap from TODO.md — 1 phase (55 save/share UX) 2026-08-30 21:13:23 -04:00
ducoterra aba8615177 fix(chat): rest the composer at the viewport bottom — sticky alone left it mid-screen
Phase 52's first pass shipped `position: sticky; bottom` on `.composer` and
called the phase done, but the owner's requirement — "the chat message-input
textarea should be at the bottom of the screen" — still failed in the browser:
on an empty/short chat the input rested just under the empty state (~57% of
the viewport) with a dead band down to the footer.

`position: sticky` can only pull a box UP toward the scrollport's bottom edge;
it can never push a box DOWN to meet it, so on a page that does not overflow
it is a no-op. The old story suite only exercised an overflowing conversation
(one test even asserted the buggy resting position as expected), which is why
the half-fix passed.

- `.messages { flex: 1 1 auto }` — absorbs a short page's free space so the
  composer's resting in-flow position is the bottom of the full-height column
  (body min-height:100dvh -> .app-main flex:1 -> .chat-shell flex:1); basis
  stays `auto`, no height cap, no overflow — the document stays the scroller
- `.composer { bottom: env(safe-area-inset-bottom, 0) }` — the explicit 0
  fallback replaces the env()-only offset, which degraded to `auto` (no pin)
  wherever env() is unsupported
- E2E: `test_empty_chat_composer_sits_in_normal_flow` ->
  `..._at_the_screen_bottom` (chrome-only band below the resting composer);
  the phone suite now checks the resting position as well as the pinned one
- Unit pins: the flex-grow half and the full-height column are pinned, so the
  fix cannot silently regress to sticky-only

Still CSS-only — no DOM change, no JS, no new scroll call site (phase 42
never-auto-scroll contract intact), no z-index.

Verified: 1019 unit/integration tests pass (app/ coverage 99%), ruff and
pyright clean; tests/e2e/test_pinned_composer.py green in isolation (4), plus
the stop/autoscroll/persistence/mobile-nav suites and 14 layout/scroll
neighbours green in isolation.
2026-08-30 16:14:18 -04:00
ducoterra 820753948e feat(chat): pin the composer to the viewport bottom — Stop is always reachable while reading 2026-08-30 14:59:40 -04:00
ducoterra 619bf2187a chore(agent): phase roadmap from TODO.md — 2 phases (52 pinned composer, 53 stale saved chats) 2026-08-30 09:33:50 -04:00
ducoterra 114b115034 feat(chat): share a chat by link — anonymous read-only /shared/<token> page, share/unshare 2026-08-30 01:34:44 -04:00
ducoterra ece93a7c8f feat(chat): save and view chat history — admin-only saved_chats, History page, open-a-chat return 2026-08-29 21:22:25 -04:00
ducoterra 6832957ab0 feat(chat): retry the last answer — redo-in-place Retry button on the latest brain bubble 2026-08-29 18:32:15 -04:00
ducoterra 1a60ecbd8b feat(chat): stop an in-flight answer — Send becomes Stop, the partial is kept and persisted, the model stream is torn down 2026-08-29 17:27:04 -04:00
2699 changed files with 263449 additions and 14431 deletions
-521
View File
@@ -1,521 +0,0 @@
# Brain of Reese — Master Plan
> **Status:** Phase 1–3 complete (scaffolded, designed, decomposed).
> **Rule:** Every agent reads this file first. Decisions marked `LOCKED` in the
> Anchors table are settled — do not re-litigate them in a phase.
> **Revisions (2026-08-21, owner permission):** A7/A8/A9 revised (multi-format
> ingestion, hybrid FTS+vector retrieval, re-tuned honesty gate); dark tech
> theme (Phase 08); clickable document viewer (Phase 10); thinking display
> (Phase 17, owner permission 2026-08-23); follow-the-bottom scroll
> (Phase 18, owner choice 2026-08-23); shared header (Phase 19);
> whole-document context (Phase 24 — A7's 24k context cap removed,
> owner permission 2026-08-24). See roadmap §12.
---
## 1. Mission
A **knowledge base chatbot** that embeds the `~/Homelab` and `~/Deployments`
projects into a Postgres vector database and lets anyone ask *Reese* (the
bot) questions about them.
**Product feel:** a chippy, upbeat assistant that is optimistic about the
user's ability ("you've got this") and **radically honest** — if retrieval
didn't surface anything relevant it says *"I haven't done anything like
that"* and offers alternatives instead of hallucinating.
### In scope (v1)
- Chat UI (mobile-friendly, well-styled, no auth, no CDN).
- RAG over text knowledge files — `md, markdown, txt, yaml, yml, json, py`
by default (A9, revised 2026-08-21) — from `~/Homelab` + `~/Deployments`
(and any future directory the importer is pointed at).
- Self-hosted models via `https://aipi.reeseapps.com/v1` — `turbo` (chat),
`embed` (embeddings, **768 dims — verified**).
- Postgres 17 + pgvector, cosine similarity, chunk→document mapping so the
LLM receives the **entire relevant document** as context.
- Idempotent import/update script, documented in the README.
- Ample server logging + explicit UI loading/progress feedback (never a
stale submit button).
### Out of scope (v1)
- Auth / multi-user (API is stateless under `/api` so it can be added later).
- Binary / non-text content, file uploads, caching layer, message persistence.
- Real-time document watching (manual re-import for now).
---
## 2. Architectural Anchors (LOCKED DECISIONS)
| # | Component | Decision | Rationale | Status |
|---|-----------|----------|-----------|--------|
| A1 | Runtime | Python 3.12+, `uv` for all package management | Fast, reproducible envs; one language for API + tooling | LOCKED |
| A2 | Web framework | FastAPI + Pydantic v2 + Uvicorn | Async, typed, SSE-friendly for LLM streaming, free OpenAPI docs | LOCKED |
| A3 | Database | **PostgreSQL 17** (`docker.io/postgres:17`, pgvector compiled in via `db/Containerfile`) with **cosine** (`<=>`) search | One system for relational + vectors; pgvector is mature; official base image kept per project standard | LOCKED |
| A4 | Orchestration | `compose.yaml`, started with **`podman compose up -d`** | Matches Reese's toolchain | LOCKED |
| A5 | LLM backend | OpenAI-compatible `https://aipi.reeseapps.com/v1`; models **`turbo`** (chat) & **`embed`** (embeddings); `openai` async client | Self-hosted, offline from cloud; no new model management | LOCKED |
| A6 | Embedding dim | **768** (verified 2026-08-21 against live endpoint via `scripts/llm_probe.py`); configured by `BOR_EMBEDDING_DIM` | User recalled 768 — probe confirmed; dimension is fixed at table creation, so mismatch must fail loudly at import time | LOCKED |
| A7 | Retrieval→context | **Hybrid:** cosine top-30 + Postgres FTS top-30 (OR tsquery, `ts_rank`) fused with **RRF (k=60)** → map to parent documents ranked by best fused chunk score → feed the **full text of top-N=2 documents** (deduped) to the LLM | Owner permission 2026-08-21: pure-cosine top-4 missed real docs (gitlab case — best chunk ranked 7th behind vendored-cache junk; score compression 0.41–0.84); the lexical signal finds name-your-tool questions; whole-document context contract preserved. A7 revised 2026-08-24 — matched documents never truncated (owner: "this should never happen"; emergency-valve variant rejected) | LOCKED (revised 2026-08-24) |
| A8 | Honesty gate | **Deflection mode** (LLM must open with a variant of *"I haven't done anything like that"* and offer 2–3 alternative questions) when best cosine < `BOR_RELEVANCE_THRESHOLD` **and** no candidate chunk FTS-matches the question; threshold re-tuned for the `embed` model's compressed score range (default **0.62**, calibrated via `scripts/eval_retrieval.py`; the E2E mock uses its own 0.30 calibration via the app fixture) | Owner permission 2026-08-21: at 0.30 the gate never discriminated (measured corpus range 0.41–0.84); the FTS-OR keeps name-your-tool questions honest-positive; deflection product behavior unchanged | LOCKED (revised 2026-08-21) |
| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-27) |
| A10 | Auth | **None in v1**; all endpoints stateless under `/api` | Per user (auth later); statelessness keeps the future migration cheap | LOCKED |
| A11 | Frontend | Vanilla HTML/CSS/JS in git; **no CDN** — everything served by FastAPI `StaticFiles`; minified by esbuild in the `Containerfile` build stage; system font stack | No external deps at runtime; tiny, auditable surface; mobile-friendly by construction | LOCKED |
| A12 | Aux services | **None in v1** (no Valkey, no SeaweedFS) | No sessions/auth (no store), no uploads (no object storage); add later only if a need appears | LOCKED |
| A13 | Migrations | Alembic + SQLAlchemy 2.0 (sync) + psycopg 3 | Standard, reversible, reviewable schema history | LOCKED |
| A14 | Debugging | `debugpy` **only when `DEBUGPY=1`** (env var read directly, not via settings); listen `0.0.0.0:5678` (override `DEBUGPY_PORT`), non-blocking, attach-on-demand; **not imported at all when off** | Zero overhead by default per project standard; attach-on-demand keeps production runs clean | LOCKED |
| A15 | Chat transport | **SSE streaming** from `POST /api/chat` (deltas + final `done` event with metadata) | Local LLM latency is 10–30s; live token stream + explicit completion event power the UI's feedback states | LOCKED |
| A16 | Testing | Per phase: unit + integration (pytest, **coverage >90%** on `app/`) + **one dedicated Playwright E2E file per user story**, run in isolation; E2E uses a deterministic mock LLM by default (`E2E_REAL_LLM=1` opts into live aipi) | One story, one phase, one E2E gate — the pipeline's core invariant | LOCKED |
| A17 | Git | Conventional Commits, **always `--no-gpg-sign`**, repo-local `commit.gpgsign=false`; one atomic commit per completed phase | Subsequent agents may lack the GPG key | LOCKED |
> **A10 revision (phase 16, owner permission 2026-08-22):** single-admin
> signed-cookie auth — public: chat / documents / suggestions / health;
> admin-only: docs catalog + steering. The row above keeps the original v1
> decision text; the public API surface stays stateless (the signed
> session cookie is the only session state) — recorded as a revision,
> not a silent deviation.
>
> **A10 UI revision (phase 19, owner permission 2026-08-23):** the
> "Sources" nav link is hidden from anonymous users on all pages — the
> soft-gate page and the API split above are unchanged.
>
> **A7 revision (phase 24, owner permission 2026-08-24):** the
> `[…truncated…]` cap on document context is removed —
> `select_documents` always returns the full top-N texts;
> `BOR_MAX_CONTEXT_CHARS` is gone. The steering section
> (`BOR_STEERING_MAX_CHARS`, phase 15) keeps its budget and the shared
> marker.
>
> **A9 revision (phase 47, owner permission 2026-08-27):** the format
> set extends with the Podman quadlet family (`container, network,
> volume, image, pod, kube, swap, os, endpoint`) and `j2` (Jinja
> templates) — plain-text chunking (`chunk_text`), owner: `TODO.md`
> L10–L11. The narrow-only `BOR_IMPORT_EXTENSIONS` rule and the
> hidden-dir/exclusion invariants are unchanged.
---
## 3. High-Level Architecture
```
┌────────────────────────────────────────────┐
│ Podman Compose │
Browser │ ┌──────────────────────────────────────┐ │
┌──────────┐ HTTP │ │ brain-of-reese/app (FastAPI) │ │
│ index.html│◄──────┼─►│ • static frontend (no CDN) │ │
│ app.js │ SSE │ │ • /api/chat /api/suggestions │ │
└──────────┘ │ │ • /api/health /api/docs │ │
│ │ • RAG pipeline (embed→retrieve→gen) │ │
│ └──────┬──────────────────┬───────────┘ │
│ │ SQL (psycopg) │ OpenAI-compat│
│ ┌──────▼──────┐ ┌───────▼────────────┐ │
│ │ db: │ └─────────┬──────────┘ │
│ │ postgres:17 │ │ │
│ │ + pgvector │ │ │
│ └─────────────┘ │ │
└──────────────────────────────┼────────────┘
▼
https://aipi.reeseapps.com/v1
(self-hosted: turbo, embed)
Offline tooling (same repo, same venv):
scripts/import_docs.py → walks A9-format dirs, chunks, embeds, upserts
scripts/eval_retrieval.py → ranks hybrid results for a question (tuning)
scripts/llm_probe.py → verifies models + embedding dim
```
### Component breakdown
| Component | Responsibility | Lives in |
|-----------|----------------|----------|
| **App (FastAPI)** | Serves frontend + `/api`; RAG pipeline; logging | `app/` |
| **RAG pipeline** | `embed` → pgvector cosine top-K → doc mapping → context assembly → `turbo` (streamed) with persona/honesty prompt | `app/rag/` (added in story phases) |
| **Importer** | Directory walk (A9 formats, hidden dirs skipped, exclusions), sha256 delta detection, format-aware chunking, batched embedding, upsert/prune | `scripts/import_docs.py` (story phase) |
| **DB** | `documents`, `chunks`, `query_log` + `vector` extension | `db/` image, `alembic/` |
| **Frontend** | Chat shell, sources view, loading/feedback states | `frontend/` |
### Chat data flow
```
user question
→ POST /api/chat {message}
→ embed(question) [aipi /v1/embeddings, model=embed]
→ cosine top-30 + FTS top-30 (OR tsquery, ts_rank) [pgvector + PG FTS]
→ RRF fuse (k=60) → docs ranked by best fused chunk score
├─ best cosine >= 0.62 OR fts_hits > 0 → top-2 documents' FULL content
│ → system prompt (persona + HONESTY rules + docs)
│ → turbo, stream=True → SSE deltas
└─ else → DEFLECT_MODE system prompt (weak hits as topics)
→ turbo, stream=True → SSE deltas (honest reply)
→ query_log row (question, score, deflected, sources, latency)
→ final SSE "done" event: {deflected, sources[], suggestions[]}
```
---
## 4. API Design
All endpoints stateless (A10). Errors: standard JSON `{detail: str}`.
| Method | Path | Purpose | Story |
|--------|------|---------|-------|
| GET | `/api/health` | Liveness + db up/down + version | 01 |
| GET | `/api/suggestions` | Onboarding suggestion strings | 01 (05 refines) |
| GET | `/api/docs` | Indexed document list (source, path, title, chunks, indexed_at) | 02 |
| GET | `/api/documents/content?source=…&path=…` | One indexed document's full content (feeds the viewer page) | 10 |
| POST | `/api/chat` | RAG chat turn → **SSE stream** | 03/04 |
### SSE contract (`POST /api/chat`)
```
data: {"type":"thinking","text":"…"}\n\n
data: {"type":"thinking","text":"…"}\n\n
data: {"type":"delta","text":"Hey! "}\n\n
data: {"type":"delta","text":"Good "}\n\n
...
data: {"type":"done","deflected":false,"sources":[{"source":"Homelab","path":"kubernetes.md","title":"Kubernetes Homelab Cluster"}],"suggestions":[]}\n\n
```
Client rules: render deltas as they arrive; render `thinking` text in a
collapsible block above the answer; auto-collapse on the first `delta`;
tolerate interleaved `thinking` events (append — never reopen once the
answer started); the `done` shape is unchanged (thinking never travels on
`done`); on `done` append source chips / suggestion chips and clear the
busy state; on HTTP/stream error show the error banner + retry (never a
stuck button).
> **SSE revision (phase 17, owner permission 2026-08-23):** the contract
> gains one event type — `{"type":"thinking","text":"…"}` — carrying the
> model's reasoning ahead of the `delta` events (the `turbo` model emits
> `delta.reasoning_content` chunks before the first content chunk, verified
> live 2026-08-23; `BOR_STREAM_THINKING=0` suppresses the frames
> server-side). `delta` and `done` shapes are unchanged — a recorded
> extension of A15, not a silent deviation.
>
> **SSE revision (phase 37, owner permission 2026-08-26):** the contract
> gains a second event type — `{"type":"tool","name":"…","argument":…}` —
> carrying the model's document tool calls on grounded turns (phase 37:
> `list_documents` / `read_document`, budgeted by `BOR_AGENT_LIST_CALLS`
> / `BOR_AGENT_READ_CALLS` (removed in phase 45 — see the revision note
> below); `argument` is `"source/path"` for
> `read_document`, null otherwise). Client rule: render each `tool` frame
> as a "calling tool" line/state (task 05); `delta` and `done` shapes are
> unchanged — the read document is reflected in `done.sources` instead
> (deduped) — a recorded extension of A15, not a silent deviation.
>
> **SSE revision (phase 45, owner permission 2026-08-27):** the phase-37
> per-turn tool budgets are **removed** (owner: "allow the LLM to make
> as many tool calls as it wants — `TODO.md` L8): `BOR_AGENT_LIST_CALLS`
> / `BOR_AGENT_READ_CALLS` no longer exist; `BOR_AGENT_MAX_ROUNDS`
> (default 10) caps the tool rounds and `0` disables the tools
> entirely (the pre-phase-37 path). The `tool` event shape and the
> `done` shape are unchanged — a recorded revision of the phase-37
> note's budget wording, not a silent deviation.
---
## 5. Data Model (PostgreSQL 17)
Created by `alembic/versions/0001_initial_schema.py` (idempotent
`CREATE EXTENSION IF NOT EXISTS vector`).
### `documents`
| Column | Type | Notes |
|--------|------|-------|
| id | `UUID` PK | |
| source | `VARCHAR(120)` | source dir basename, e.g. `Homelab` |
| path | `VARCHAR(1000)` | relative to source dir, e.g. `ansible/roles/k3s.md` |
| full_path | `VARCHAR(2000)` | absolute path at import time (diagnostics) |
| title | `VARCHAR(500)` | first markdown H1, else file stem |
| content | `TEXT` | **full markdown — the RAG context** |
| content_hash | `VARCHAR(64)` | sha256 of content — change detection |
| indexed_at | `TIMESTAMPTZ` | |
| — | `UNIQUE (source, path)` | upsert key |
### `chunks`
| Column | Type | Notes |
|--------|------|-------|
| id | `UUID` PK | |
| document_id | `UUID` FK→documents CASCADE | **embedding→document mapping** |
| position | `INT` | 0-based order within the doc |
| content | `TEXT` | chunk text (heading-aware) |
| embedding | `VECTOR(768)` | nullable until embedded (two-phase import) |
| tsv | `TSVECTOR` | **generated** `to_tsvector('english', content) STORED` + GIN index (hybrid retrieval, A7) |
> No vector index in v1: sequential scan is fine at this corpus size
> (~100–500 docs). Revisit with an HNSW index if retrieval latency grows.
### `query_log`
`id UUID PK, question TEXT, top_score FLOAT, fts_hits INT, chunk_hits INT, deflected BOOL, sources TEXT, latency_ms INT, created_at TIMESTAMPTZ`
### Document state transitions
```
unseen ──import──▶ indexed ──hash changed + re-import──▶ reindexed
│
└──file deleted + --prune──▶ removed (chunks cascade)
```
### Chunking policy (markdown-aware)
Split on `## `/`### ` headings into sections; sub-split any section longer
than `BOR_CHUNK_TARGET_CHARS` (2000) at paragraph boundaries with
`BOR_CHUNK_OVERLAP_CHARS` (200) overlap; each chunk keeps its nearest
preceding heading in the text for retrieval quality.
**Format-aware (A9, revised):** `yaml`/`yml` split on top-level keys and
`---` separators (key line kept as anchor); `json` pretty-printed, split on
top-level keys; `py` split on top-level defs/classes (stdlib `ast`);
`txt` on paragraphs; markdown unchanged. The quadlet family
(`container, network, volume, image, pod, kube, swap, os, endpoint`)
and `j2` (Jinja templates) are plain-text chunked — no format-specific
splitter (A9 revised 2026-08-27, phase 47). Every format honors the
1200-char hard cap (aipi ~1024-token request limit).
---
## 6. RAG Pipeline & Persona
### Locked system prompt (sent with every chat turn)
```
You are "Brain of Reese" — the digital brain of Reese, a self-hoster and
homelab tinkerer. Personality: chippy, upbeat, warm, and genuinely
optimistic about the user's ability to do things ("you've got this").
Rules:
1. Answer ONLY from the provided document context. Cite which document(s)
you used, by path.
2. Be concrete: names, versions, ports, hosts, schedules — the specifics in
the docs are the value.
3. HONESTY GATE: if <relevance> is "LOW", you must NOT pretend to know.
Start your answer with a variant of: "I haven't done anything like that."
Then offer 2-3 alternative questions about things you DO have notes on.
4. Never invent facts, hosts, or steps that are not in the context.
5. Keep answers tight: short paragraphs, bullets where helpful.
<relevance>{HIGH|LOW}</relevance>
```
- `HIGH` mode appends the full document text under `<documents>…</documents>`.
- `LOW` mode (deflection) appends only the **titles** of the weak hits so the
model can suggest real alternatives (marker used by the E2E mock:
`DEFLECT_MODE` appears in the system prompt).
### Retrieval (hybrid — A7/A8, revised 2026-08-21)
- Embed the question (`embed`, 768-d) → cosine top-30 candidates.
- Lexical: OR tsquery over the question's tokens → FTS top-30 by `ts_rank`.
- **Reciprocal Rank Fusion** (`Σ 1/(k+rank)`, k=60) → distinct parent docs
ranked by best chunk's fused score → top 2 → full content, concatenated
— **never truncated** (A7 revised, phase 24, owner permission
2026-08-24).
- Honesty gate: LOW only when `best cosine < BOR_RELEVANCE_THRESHOLD`
(default 0.62, calibrated against the `embed` model's measured 0.41–0.84
distribution) **and** zero FTS hits among the candidates.
---
## 7. UI/UX Strategy
### 7.1 Layout structure
- **App frame:** sticky header (64px) + `<main>` (flex-grow) + footer.
Container: `max-width: 72rem; margin-inline: auto; padding-inline: 1.25rem`.
- **Shared header (Phase 19, owner permission 2026-08-23):** the bar is a
shared contract across chat / sources / viewer — one bar per page, same
controls (brand + nav [Chat, Sources — admin only] + New Chat + Sign in
/ Sign out on chat & sources; the viewer bar = back + title + the same
actions in `.doc-header-actions`). One shared module
(`frontend/assets/header.js`) toggles the existing controls on each
page, so they can never "disappear" between pages again; the heights
stay pinned at 64px / 58px (phase 12).
- **Chat:** a *centered column capped at 46rem*. This is deliberate: chat is
a vertical conversation — a centered, capped column is the correct pattern
(NOT a layout bug). The 72rem frame + header/footer ensure the column
never reads as a hairline in a sea of whitespace.
- **Sources page:** full-width responsive **table** (min 640px, horizontal
scroll wrapper on small screens) + stat cards in
`grid-template-columns: repeat(auto-fit, minmax(170px, 1fr))`.
No skinny single-column lists anywhere: lists/tables/grids use ≥80–90% of
the container width.
- **Mobile (≤640px):** suggestion chips become a horizontally scrollable row;
composer stays reachable with `safe-area-inset-bottom`; touch targets ≥44px.
### 7.2 Accessibility (WCAG 2.1 AA)
- Semantic landmarks on every page: `<header>`, `<nav aria-label>`,
`<main>`, `<footer>`; skip-link to `#main`.
- Every control labeled: visible `<label>` or `aria-label` (icon-only
buttons always get `aria-label`); form input has a (visually-hidden) label.
- Live regions: message stream `aria-live="polite"`; typing indicator
`role="status"`; banner `role="status"`; errors `role="alert"`.
- **Dark tech theme (Phase 08, 2026-08-21)** — page `#0a0e17`, surface
`#121a2e`; ink `#e8ebf4` on surface ≈14.5:1; ink-soft `#9aa4bd` on
surface ≈6.9:1; **dark ink `#0a0e17` on brand `#6d78f2` ≈5.2:1** (white
on brand ≈3.7:1 — never used for text); brand-ink `#a5b4fc` on
brand-soft `#232b52` ≈6.9:1; deflection `#fbbf24` on `#2b2110` ≈9.5:1
(border `#f59e0b`); error `#fca5a5` on `#2d1318` ≈9.1:1. All computed,
all ≥4.5:1. `prefers-reduced-motion` also stills the Phase-08 background
layer.
- `:focus-visible` outline 3px; `prefers-reduced-motion` respected by the
typing/spinner animations.
### 7.3 No external dependencies
- System font stack only (no font files to bundle, no CDN fonts).
- Zero `<script src="https://…">` / `<link href="https://…">` — enforced by
an integration test (`tests/integration/test_api.py::test_index_html_served_locally`)
and re-checked by every UI phase's verification step.
- Markdown rendering is a ~60-line local function (escape-first, then
transform) — XSS-safe, no library.
### 7.4 Visual feedback standard (the "never stale" contract)
| State | UI |
|-------|----|
| **Idle** | Send button enabled, label "Send". |
| **Thinking (pre-token)** | 3-dot typing bubble + button disabled with spinner, label "Thinking…". |
| **Thinking (model reasoning)** | Collapsible `.thinking` block streams open (replaces the typing dots as the live indicator), auto-collapses on the first answer token, toggleable afterwards, persisted with the message (phase 14); 120s guard clears on the first `thinking` *or* `delta` event. |
| **Streaming** | Deltas append live into the brain bubble; button stays busy. |
| **Done (answer)** | Source chips under the bubble (mono, path-based); button re-enabled. |
| **Done (deflected)** | Amber-bordered bubble + "Maybe try:" suggestion chips. |
| **Error** | Red banner (`role="alert"`) with retry hint; button re-enabled. |
| **KB offline** | Amber banner at top of chat ("start Postgres…"); chat disabled with explanation. |
| **Guard** | 120s client-side timeout → error state (a button can never sit "stuck" forever). |
| **Scroll (follow-the-bottom, phase 18)** | The page auto-scrolls only while the user is pinned to the bottom (≤200px band, `NEAR_BOTTOM_PX` — the composer zone; submitting reveals the user's message through the same gate, which holds in real use); scrolling up holds the viewport for the rest of the turn (thinking and answer alike); restore lands one-shot on the latest message. |
> The **Thinking (model reasoning)** row is a phase-17 addition (owner
> permission 2026-08-23) — see the §4 SSE revision.
>
> The **Scroll** row is a phase-18 addition (owner choice 2026-08-23 —
> option 1: follow-the-bottom, no "↓ new content" pill).
### 7.5 Component inventory (ids used by tests)
`#messages` (stream), `#empty-state`, `#suggestions`, `.suggestion-chip`,
`#composer`, `#message-input`, `#send-btn` / `#send-label`, `#typing-indicator`,
`.msg.user/.msg.brain .bubble`, `.source-chip`, `.msg.brain.is-deflected`,
`#kb-banner`, `#app-version`; sources: `#stat-docs`, `#stat-chunks`,
`#stat-last`, `#docs-table`, `#docs-tbody`, `#sources-empty`; viewer
(Phase 10): `/document.html`, `#doc-title`, `#doc-meta`, `#doc-content`,
`.doc-raw`, `.format-badge`, `#doc-not-found`, `.doc-link` (Sources table
path links); thinking (phase 17, owner permission 2026-08-23):
`.thinking`, `.thinking-text` (collapsible thinking block; plain
`<summary>`, no id); auth (phase 16, owner permission 2026-08-22):
`#sign-in-link`, `#sign-out-btn`, `#sources-gate`; shared header (phase
19, owner permission 2026-08-23): `#nav-sources` (Sources nav link,
hidden for anonymous), `#new-chat-btn` + `#sign-in-link` + `#sign-out-btn`
on the sources and viewer pages (ids shared with chat),
`.doc-header-actions` (viewer).
---
## 8. Debugging (debugpy protocol)
- `DEBUGPY` unset/`0` → **`debugpy` is never imported** (verified by unit test).
- `DEBUGPY=1` → listener on `0.0.0.0:${DEBUGPY_PORT:-5678}`, **non-blocking**,
app continues; IDE attaches on demand.
- Entry point: `app/core/debugging.py::configure_debugging()` called at the top
of `app/main.py` module import — so `uv run uvicorn app.main:app`,
`python -m scripts.…`, and tests all honor it.
- VS Code: `"type": "debugpy", "request": "attach", "connect": {"host": "localhost", "port": 5678}`.
---
## 9. Observability
- **App logs:** single-line `timestamp LEVEL logger :: message` on stdout;
uvicorn access logs on. INFO by default (`BOR_LOG_LEVEL`).
- **Per-chat-turn log line (required):**
`question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N kb_chars=N threshold=… deflected=… sources=… thinking_chars=… tool_calls=N total_ms=…`
(`thinking_chars=` counts the turn's reasoning chars — phase 17, owner
permission 2026-08-23 — and is counted even when `BOR_STREAM_THINKING=0`
suppresses the frames.)
> **Log-line revision (phase 37, owner permission 2026-08-26):** the
> required per-turn line gains `tool_calls=N` after `thinking_chars=` —
> the count of agent tool executions that consumed budget on the turn
> (phase 37's `list_documents` / `read_document`; rejected calls do not
> count, and deflected turns run no tools). `summary_hits=` (phase 30)
> and `kb_chars=` (phase 31) are recorded here as well; `sources=` lists
> the retrieval docs plus any agent-read documents, deduped.
- **Importer logs:** per-file `added|updated|unchanged|pruned` + summary
(counts, embedding batches, total time).
- **`query_log` table:** durable record of every question (score, deflection,
sources, latency) for tuning the threshold and finding gaps in the docs.
---
## 10. Testing Strategy (LOCKED — A16)
| Layer | Tooling | Runs | Gate |
|-------|---------|------|------|
| Unit | pytest | `uv run pytest tests/unit` | pass |
| Integration | pytest + FastAPI TestClient | `uv run pytest tests/integration` | pass |
| Coverage | pytest-cov on `app/` | `uv run pytest --cov=app --cov-report=term-missing` | **>90%** per phase |
| E2E | Playwright (sync API), one file per story | `uv run pytest tests/e2e/test_<story>.py -v --no-cov` | passes **in isolation** |
- **E2E determinism:** `tests/e2e/mock_llm.py` serves a deterministic
OpenAI-compatible API. Embeddings are genuine L2-normalized token-overlap
vectors, so the cosine threshold behaves like production: on-topic
questions retrieve, off-topic questions deflect. `E2E_REAL_LLM=1` switches
the app fixture to live aipi (needs imported KB).
- **E2E prerequisites:** `podman compose up -d db`; Chromium installed via
`uv run playwright install chromium`.
- DB isolation: story E2E fixtures truncate `query_log` (and re-import
fixtures for import-dependent stories) per test module.
---
## 11. Import & Update Workflow (documented in README)
```
# first import (and any future refresh):
uv run python -m scripts.import_docs # defaults: ~/Homelab ~/Deployments
uv run python -m scripts.import_docs --source ~/OtherProject # extra dirs
uv run python -m scripts.import_docs --prune # drop deleted / filtered-out files
uv run python -m scripts.eval_retrieval "How did I install gitlab?"
uv run python -m scripts.llm_probe # sanity: models + dim
```
Behavior: sha256 delta per `(source, path)` — unchanged files are skipped
(no re-embedding); changed files are re-chunked + re-embedded (chunks
replaced atomically); `--prune` removes docs whose files disappeared or no
longer match the format filter. Formats per A9 (revised 2026-08-27):
`md, markdown, txt, yaml, yml, json, py`, the quadlet family
(`container, network, volume, image, pod, kube, swap, os, endpoint`), and
`j2` (plain-text chunked) (`BOR_IMPORT_EXTENSIONS`), hidden (dot)
directories skipped, exclusion list applied. `scripts/eval_retrieval.py`
ranks live hybrid results for a question (retrieval tuning).
---
## 12. Roadmap (one story → one phase → one Playwright gate)
| Phase | File | Story | Playwright gate |
|-------|------|-------|-----------------|
| 01 | `01_infrastructure.md` | — (foundation) | `tests/e2e/test_smoke.py` |
| 02 | `02_story_import_documents.md` | `import-documents.md` | `tests/e2e/test_import_documents.py` |
| 03 | `03_story_chat_rag.md` | `chat-rag-answer.md` | `tests/e2e/test_chat_rag.py` |
| 04 | `04_story_honest_deflection.md` | `honest-deflection.md` | `tests/e2e/test_honest_deflection.py` |
| 05 | `05_story_suggestion_chips.md` | `suggestion-chips.md` | `tests/e2e/test_suggestion_chips.py` |
| 06 | `06_story_loading_feedback.md` | `loading-feedback.md` | `tests/e2e/test_loading_feedback.py` |
| 07 | `07_story_responsive_polish.md` | `responsive-polish.md` | `tests/e2e/test_responsive_polish.py` |
| 08 | `08_story_dark_tech_theme.md` | `dark-tech-theme.md` | `tests/e2e/test_dark_tech_theme.py` |
| 09 | `09_story_retrieval_quality.md` | `retrieval-quality.md` | `tests/e2e/test_retrieval_quality.py` |
| 10 | `10_story_document_viewer.md` | `document-viewer.md` | `tests/e2e/test_document_viewer.py` |
| 17 | `17_thinking_display.md` | `thinking-display.md` | `tests/e2e/test_thinking_display.py` |
| 18 | `18_follow_bottom_scroll.md` | `follow-bottom-scroll.md` | `tests/e2e/test_follow_bottom_scroll.py` |
| 19 | `19_shared_header.md` | `shared-header.md` | `tests/e2e/test_shared_header.py` |
| 24 | `24_whole_document_context.md` | `whole-document-context.md` | `tests/e2e/test_whole_document_context.py` |
> Row 17 (thinking display) added 2026-08-23 with owner permission — the
> A15 SSE extension recorded in §4.
>
> Row 18 (follow-the-bottom scroll) added 2026-08-23 with owner choice —
> option 1: follow-the-bottom, no "↓ new content" pill (UI-behavior-only
> change; no anchor revised).
>
> Row 19 (shared header) added 2026-08-23 with owner permission —
> Sign in/Sign out + New Chat on every page via one shared module, and
> the phase-16 "Sources" link UX choice revised: the nav link is hidden
> for anonymous (the soft-gate page and the A10 API split are unchanged).
>
> Row 24 (whole-document context) added 2026-08-24 with owner permission
> — A7's 24k context cap removed (documents are never truncated)
Completion = unit+integration green, coverage >90%, story E2E green in
isolation, UI verification passed, **one `--no-gpg-sign` commit**.
---
## 13. Future (post-v1 hooks, deliberately not built)
- Auth (stateless API makes this a drop-in: sessions → Valkey).
- HNSW index on `chunks.embedding` at scale.
- Conversation persistence (messages tables).
- Watchdog auto-re-import (inotify) — until then the script is the truth.
- More sources: any directory of A9-format files via `--source`.
@@ -1,70 +0,0 @@
# Task 02 — E2E story suite, story file, validation, commit
**Phase:** `21_thinking_no_scroll` · **Source:** `TODO.md` L4
## Objective
The story gate: `tests/e2e/test_thinking_no_scroll.py` proves the window
can't be user-scrolled but always tracks the live tail, plus regressions,
story file, final validation, and the single atomic commit.
## Work
1. `tests/e2e/test_thinking_no_scroll.py` (new — reuse
`test_thinking_display.py`'s mock-LLM streaming scaffolding; the mock
must stream a **long** thinking body, in many chunks, so
`.thinking-text` overflow exceeds its 320px box). Tests:
1. `test_thinking_window_not_user_scrollable` — open the block, wait
until `scrollHeight > clientHeight`; focus `.thinking-text`
(`el.focus()`), dispatch mouse wheel over it
(`page.mouse.wheel(0, -200)` after moving the mouse over the
element) and press `Home`/`ArrowUp`: `scrollTop` must not decrease
(assert `scrollTop` unchanged within 1px between actions).
2. `test_thinking_window_tracks_live_tail` — while chunks stream,
after the 2nd-to-last and last chunk:
`scrollTop === scrollHeight` (within 1px) — the visible window is
the live tail; the **last** chunk's text is within the visible
rectangle (its offsetTop + scrollTop geometry check, or
`elementFromPoint` at the box's bottom).
3. `test_thinking_window_css_contract` — computed style of
`.thinking-text`: `overflow-y === "hidden"`,
`max-height === "320px"`.
4. `test_answer_bubble_still_scrollable` (regression, phase 11) — a
long answer (use the long-answer mock from
`test_long_answers.py`): the answer bubble is still
user-scrollable (scrollTop moves on wheel) and
`overflow-y` is not `hidden` there.
5. `test_restored_collapsed_thinking_unaffected` (regression,
phase 17) — a turn with stored `thinking`, reload: the collapsed
Thinking block renders with its text (existing pin from
`test_thinking_display.py` — replicate, don't duplicate the file).
2. `.agent/user_stories/thinking-no-scroll.md` (new) — story file per
the repo format: goal, the bug report verbatim from `TODO.md` L4, the
owner-confirmed A2 decisions from `00_phase.md`, E2E mapping table.
3. Run the suite **in isolation** (prereq `podman compose up -d db`):
`uv run pytest tests/e2e/test_thinking_no_scroll.py -v --no-cov`.
4. Regressions, in isolation, one command each:
- `uv run pytest tests/e2e/test_thinking_display.py -v --no-cov`
- `uv run pytest tests/e2e/test_long_answers.py -v --no-cov`
5. Final validation: `uv run pytest` green; `uv run pytest --cov=app
--cov-report=term-missing` ≥ today's number (>90% gate);
`uv run ruff check . && uv run pyright` clean.
6. **UI Structure Check** (AGENTS.md rule 5): no new surface; the block
keeps its summary chevron, focus-visible ring, aria-live/label
contract, and the reduced-motion stillness (styles.css ~line 686).
7. Write the phase report (`.agent/reports/21_thinking_no_scroll/`).
8. Commit (one atomic commit) and move the phase:
```bash
git add -A .agent/ frontend/ tests/
git commit --no-gpg-sign -m "fix(ui): thinking window no longer scrolls — live 320px view pinned to the stream tail"
mv .agent/phases/todo/21_thinking_no_scroll .agent/phases/complete/
```
## Testing & Quality
- Story suite green **in isolation**; both regression suites green in
isolation; full unit+integration suite green; `app/` coverage at or
above today's number (>90%); ruff + pyright clean.
## Completion Criteria
- [ ] `test_thinking_no_scroll.py` 5/5 in isolation.
- [ ] Regressions (thinking display, long answers) green in isolation.
- [ ] Story file + phase report exist.
- [ ] One `--no-gpg-sign` commit; phase directory in `complete/`.
@@ -1,60 +0,0 @@
# Task 01 — Diagnose and fix the animated background (styles.css)
**Phase:** `22_background_animation` · **Source:** `TODO.md` L5 —
*"Fix background animation not working, just blinking"*
## Objective
Find why the phase-08 animated background reads as "just blinking" and
fix `styles.css` so the grid drift and the glow breathe are both visibly
and smoothly alive, per the phase-08 design comments (pure CSS, zero JS).
## Work
1. **Reproduce:** `uv run uvicorn app.main:app --reload` (db up), open `/`
in a visible Chromium window (Playwright or the interactive browser),
observe ≥15s. Note exactly what moves and what doesn't.
2. **Diagnose per the `00_phase.md` checklist** (per-layer visibility
toggles, `background-position` samples on `body::before`, mask
inspection, occlusion check against `html`/`body` rules, glow
opacity-swing perception). Record findings + before-screenshot in
`.agent/reports/22_background_animation/` and
`.agent/screenshots/22_background_animation/`.
**ASSUMPTION (to verify, not assume):** the likely culprits, in
order — (a) the masked grid drift is too faint/slow to perceive,
(b) only the glow opacity swing is visible and it reads as a blink,
(c) a later rule occludes the `z-index: -1` layers. Confirm which
one actually fires before touching CSS; the fix must match the found
cause.
3. **Fix in `frontend/assets/styles.css`** (smallest change that makes
the design read):
- grid: raise line alpha and/or the mask's visible radius and/or the
drift speed as needed for a clearly visible, seamless drift
(drift delta must still equal one 44px cell for a seamless loop —
if the speed changes, keep `background-position` 0→44px and only
move the duration);
- glow: if the breathe reads as a blink, narrow the opacity delta
(e.g. 0.8↔1) and/or lengthen the period — it must read as
breathing, not pulsing;
- keep: both layers `position: fixed; inset: 0; z-index: -1;
pointer-events: none`; no `filter: blur`; no JS; palette/contrast
untouched.
4. **After-screenshot** (same viewport, two frames a few seconds apart
showing motion) into the same screenshots dir.
5. `tests/unit/test_background_animation.py` (new — repo source-pin
pattern): pin the **final** `styles.css` values — both
`@keyframes` present, `body::before` → `bg-grid-drift linear
infinite`, `body::after` → `bg-glow-breathe`, both layers
`fixed`/`z-index: -1`/`pointer-events: none`, `html` keeps
`background: var(--bg)`, `body` keeps `background: transparent`.
6. Manual re-verify: the "just blinking" perception is gone — smooth
drift + gentle breathe, no jank, no static frame.
## Testing & Quality
- `uv run pytest tests/unit/test_background_animation.py -v` green.
- `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] Root cause documented (with before/after screenshots) in the
phase report dir.
- [ ] Both layers visibly animate as the phase-08 design describes;
pure CSS, zero JS, no blur.
- [ ] Unit pins green against the final values; lint/types clean.
@@ -1,120 +0,0 @@
# Phase 23 — Containerfile: Build the Whole App Image Again
**Source:** `TODO.md` L6 — *"Fix Containerfile build not working"*
**Story:** `.agent/user_stories/containerfile-build.md` (created by task 02)
**Context:** `Containerfile` (3 stages: node:22-alpine + esbuild
0.25.5 frontend bundle → uv/python deps → slim runtime serving
`/app/static`); `frontend/` (4 pages: `index.html`, `sources.html`,
`document.html`, `login.html`; assets: `styles.css`, `markdown.js`
(classic script), `header.js`/`app.js`/`sources.js`/`document.js`/
`login.js` (ES modules)); `scripts/entrypoint.sh`.
## Verified diagnosis (2026-08-24, this conversion — not a guess)
1. **Root cause of the build failure:** phase 19 switched the page
scripts to `import … from "/assets/header.js"` (an absolute URL).
esbuild resolves that as the *filesystem* path `/assets/header.js`
and the stage-1 bundle dies:
`✘ [ERROR] Could not resolve "/assets/header.js"`
(reproduced with esbuild **0.25.5**, the exact pinned version, on a
copy of `frontend/`).
2. **Secondary gap (image would be broken even if it built):** stage 1
bundles only `app.js` + `sources.js` and copies only `index.html` +
`sources.html`. Missing from the image: `document.html` +
`login.html` (phases 10/16), `document.js` + `login.js`, and
`markdown.js` (classic script loaded by `index.html` +
`document.html`).
3. **Verified fix:** with relative imports (`from "./header.js"`) all
four page scripts bundle cleanly with esbuild 0.25.5.
4. **Latent double-evaluation trap:** all four HTML pages also load
`<script type="module" src="/assets/header.js">` directly while the
page script imports it. In dev the browser dedupes (same module
URL) — but in the image the bundled page script already contains the
header code, so shipping a raw `header.js` too would evaluate the
module **twice** (duplicate sign-out listener, double init). The
direct tags are redundant: the page script's `import` is hoisted and
guarantees `header.js` evaluates before the page script's body calls
`initSharedHeader()`, in dev and in the bundle alike.
## Objective
`podman build -f Containerfile .` succeeds, and the resulting image
serves the **whole app** — all four pages with their bundled, minified,
local-only assets (No CDN rule) — with `header.js` evaluated exactly
once per page.
## Owner-confirmed (2026-08-24, roadmap A4)
1. **Relative imports** (`./header.js`) over an esbuild alias — simpler,
verified working, dev-server behavior unchanged (files are
side-by-side).
2. **Remove the four redundant direct `header.js` script tags** (the
design above) rather than ship a raw `header.js` into the image —
single module evaluation, no duplicate listeners.
3. The image must cover **all four pages + all local assets** they
reference — the integration test (task 02) enforces this coverage so
the gap cannot silently reappear.
## Dependencies
- `19_shared_header` (complete) — introduced the absolute imports (root
cause) and the direct `header.js` tags.
- `10_story_document_viewer` / `16_admin_auth` (complete) — the pages
missing from the image.
- `08_story_dark_tech_theme` (complete) — No CDN rule the image must
honor.
## Tasks
1. `01_fix_containerfile_build.md` — relative imports, tag removal,
stage-1 asset coverage, green `podman build`, image smoke test.
2. `02_integration_test_commit.md` — `tests/integration/
test_containerfile_assets.py` (hermetic coverage pin), regression
suites, story file, final validation, the single atomic commit,
phase move to `complete/`.
## Locked decisions
- **A11 honored** — vanilla JS, no CDN, static serving from FastAPI.
**A16 honored** — integration test for the new build coverage; story
file + report; no Playwright suite required (this phase is
build/infrastructure — the phase gate is the hermetic integration
test + the real `podman build` + image smoke recorded in the report,
plus the dev-server E2E regressions). No anchor changed.
## Testing & Quality
- **Integration (new `tests/integration/test_containerfile_assets.py`,
hermetic — no podman, no network):** every `frontend/*.html` is
copied into stage 1's `/out`; every local `src`/`href` asset
referenced by the four pages is produced by a stage-1 line (esbuild
`--outfile` or `cp`); the four page module scripts are the exact set
esbuild bundles; `markdown.js` is produced; no HTML references
`/assets/header.js` directly (single-evaluation design pin); the
esbuild version stays pinned.
- **Unit:** none (no `app/` changes).
- **Coverage:** the >90% `app/` gate is unaffected, re-run to prove it.
- **Build gate (manual, recorded in the report):** `podman build
-f Containerfile .` green; image smoke (task 01 step 6) results +
log excerpt in `.agent/reports/23_containerfile_build/`.
- **Dev regressions (E2E, isolated):** `test_smoke.py`,
`test_shared_header.py`, `test_chat_persistence.py` (the HTML tag
removal touches dev page load).
- **Lint/types:** `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] Local esbuild 0.25.5 bundles all four page scripts cleanly.
- [ ] `podman build -f Containerfile .` green (log excerpt in the
report).
- [ ] Image smoke: container runs (throwaway Postgres 17 + pgvector);
`GET /`, `/sources.html`, `/document.html`, `/login.html` → 200;
`/assets/app.js` minified and contains the header code;
`/assets/markdown.js` 200; no `http(s)://` asset reference in any
served page (No CDN rule).
- [ ] Dev server unchanged in behavior: the three regression E2E suites
green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app
--cov-report=term-missing` ≥ today's number.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/containerfile-build.md` exists.
- [ ] One `--no-gpg-sign` commit (below);
`.agent/phases/todo/23_containerfile_build/` moved to
`.agent/phases/complete/`.
## Commit
```bash
git add -A .agent/ Containerfile frontend/ tests/ && git commit --no-gpg-sign -m "fix(build): Containerfile builds again — relative module imports, all four pages and shared assets in the image"
```
@@ -1,91 +0,0 @@
# Task 01 — Fix the build: relative imports, tag removal, full stage-1 asset coverage
**Phase:** `23_containerfile_build` · **Source:** `TODO.md` L6 —
*"Fix Containerfile build not working"*
## Objective
Make `podman build -f Containerfile .` succeed and ship the **complete**
frontend in the image: all four pages, all four bundled page modules,
the classic `markdown.js`, and the minified `styles.css` — with
`header.js` evaluated exactly once per page.
## Work
1. **Reproduce the failure** and record it in
`.agent/reports/23_containerfile_build/` (log excerpt):
- fast: `npx -y esbuild@0.25.5` on a copy of `frontend/` → the
`Could not resolve "/assets/header.js"` error (root cause, already
reproduced during conversion);
- authoritative: `podman build -f Containerfile .` → stage 1 fails
at the same line.
2. **`frontend/assets/{app,sources,document,login}.js`** — change the
header import from absolute URL to relative (one line each; the
specifiers are currently `from "/assets/header.js"`):
```js
import { … } from "./header.js";
```
(owner-confirmed A4 — relative over esbuild alias; dev-server
behavior is unchanged since the files are side-by-side and the
module URL resolves to the same file.)
3. **Remove the four redundant direct `header.js` tags** (owner-confirmed
A4-2 — the single-evaluation design from `00_phase.md`):
- `frontend/index.html` (~line 119) —
`<script type="module" src="/assets/header.js"></script>`;
- `frontend/sources.html` (~line 125), `frontend/document.html`
(~line 80), `frontend/login.html` (~line 67) — same tag.
- Update the surrounding HTML comments that describe the
header-before-page-script load order (e.g. index.html ~lines
115–119): the order is now guaranteed by the page script's own
`import` (hoisted, evaluated before the page script body calls
`initSharedHeader()`).
4. **`Containerfile` stage 1** — cover the whole app (keep the pinned
`esbuild@0.25.5` and the existing flags):
```dockerfile
RUN mkdir -p /out/assets \
&& esbuild ./assets/app.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/app.js \
&& esbuild ./assets/sources.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/sources.js \
&& esbuild ./assets/document.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/document.js \
&& esbuild ./assets/login.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/login.js \
&& esbuild ./assets/markdown.js --minify --outfile=/out/assets/markdown.js \
&& esbuild ./assets/styles.css --minify --outfile=/out/assets/styles.css \
&& cp ./index.html ./sources.html ./document.html ./login.html /out/
```
(`markdown.js` is a classic script — minify only, **no** `--bundle`;
it exposes globals used by the pages.)
5. **Verify locally (no podman):** with esbuild 0.25.5, all four module
bundles + the markdown minify succeed on the real `frontend/` (not a
copy — the copy was only for the diagnosis).
6. **`podman build -f Containerfile .`** → green.
7. **Image smoke test** (results + log excerpt into the report dir):
- throwaway Postgres 17 + pgvector (`podman compose up -d db` and
point the container at it, or a one-off container with the same
env as `compose.yaml`);
- run the built image (migrations run via the entrypoint);
- `GET /`, `/sources.html`, `/document.html`, `/login.html` → 200;
- `GET /assets/app.js` → 200, minified (single-line-ish), and
contains the header code (e.g. the `clearChatStorage` function
body); `GET /assets/markdown.js`, `/styles.css`, the other three
page modules → 200;
- No CDN rule: none of the four served pages contain an `http(s)://`
`src`/`href` asset reference.
- Teardown the throwaway containers when done.
8. **Dev-server regression check** (the tag removal touches dev page
load — confirm boot order still holds): `uv run uvicorn
app.main:app --reload`, load all four pages, check the sign-out
binding exists exactly once (DevTools: no duplicate listener — one
`POST /api/logout` per click) and `initSharedHeader()` ran. (The
isolated E2E regressions run in task 02.)
## Testing & Quality
- Steps 5–8 above; `uv run ruff check . && uv run pyright` clean
(no Python changes, but keep the gate green).
## Completion Criteria
- [ ] The recorded build failure is fixed at the root cause (relative
imports) — not masked by an alias/patch.
- [ ] All four direct `header.js` tags removed + comments updated; the
page scripts' `import "./header.js"` is the only header load.
- [ ] Stage 1 produces: 4 HTML pages, 4 bundled modules, minified
`markdown.js`, minified `styles.css`.
- [ ] `podman build` green; image smoke all-200 + No CDN + single
header evaluation; dev-server boot unchanged (step 8).
- [ ] Log/screenshot evidence in `.agent/reports/23_containerfile_build/`.
@@ -1,70 +0,0 @@
# Task 02 — Integration coverage test, story file, validation, commit
**Phase:** `23_containerfile_build` · **Source:** `TODO.md` L6
## Objective
Pin the stage-1 asset coverage so it can't silently rot again (a new
page/script/asset without a matching Containerfile line fails CI), plus
regressions, story file, final validation, and the single atomic commit.
## Work
1. `tests/integration/test_containerfile_assets.py` (new — **hermetic**:
parses `Containerfile` + `frontend/` as text, no podman, no network).
Tests:
1. `test_every_html_page_is_copied_into_stage1` — for each
`frontend/*.html` in the repo, a stage-1 line copies it into
`/out` (regex over the `cp` line; the set must be exactly the
four current pages — a new page added to `frontend/` fails this).
2. `test_every_local_asset_reference_is_produced` — collect every
local `src=`/`href=` under `assets/` or `/assets/` from the four
HTML files; each basename must be produced by a stage-1 line
(an `esbuild … --outfile=/out/assets/<name>` or a `cp` of it).
(This is what catches a missing `markdown.js`-style gap.)
3. `test_page_module_scripts_are_bundled` — the set of `type="module"`
page scripts referenced by the HTML (basenames) equals the set of
scripts esbuild bundles in stage 1 (`app.js`, `sources.js`,
`document.js`, `login.js`).
4. `test_header_module_is_imported_not_directly_loaded` — no HTML
file contains a `<script … src="/assets/header.js">` (or
`assets/header.js`) tag (the single-evaluation design pin,
owner-confirmed A4-2); and each of the four page scripts imports
it relatively (`from "./header.js"`).
5. `test_markdown_js_is_a_produced_classic_script` — `markdown.js`
has a stage-1 minify line **without** `--bundle` (it is a classic
global script) and no `import`/`export` statements at its top
level (source pin of that assumption).
6. `test_esbuild_stays_pinned` — the frontend stage pins a concrete
`esbuild@X.Y.Z` version (no floating version).
2. `.agent/user_stories/containerfile-build.md` (new) — story file per
the repo format: goal, the bug report verbatim from `TODO.md` L6, the
verified diagnosis (root cause + missing-asset gap + double-eval
trap), the owner-confirmed A4 decisions, and the test mapping table.
3. Regressions, in isolation, one command each (prereq
`podman compose up -d db`):
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov`
- `uv run pytest tests/e2e/test_shared_header.py -v --no-cov`
- `uv run pytest tests/e2e/test_chat_persistence.py -v --no-cov`
4. Final validation: `uv run pytest` green (includes the new
integration test); `uv run pytest --cov=app --cov-report=term-missing`
≥ today's number (>90% gate); `uv run ruff check . && uv run pyright`
clean.
5. Finish the phase report (`.agent/reports/23_containerfile_build/` —
build log excerpt, smoke results, regression results).
6. Commit (one atomic commit) and move the phase:
```bash
git add -A .agent/ Containerfile frontend/ tests/
git commit --no-gpg-sign -m "fix(build): Containerfile builds again — relative module imports, all four pages and shared assets in the image"
mv .agent/phases/todo/23_containerfile_build .agent/phases/complete/
```
## Testing & Quality
- New integration suite green within `uv run pytest`; the three
regression E2E suites green in isolation; full suite green; `app/`
coverage at or above today's number (>90%); ruff + pyright clean.
## Completion Criteria
- [ ] `test_containerfile_assets.py` 6/6 within the full suite.
- [ ] Regressions (smoke, shared header, chat persistence) green in
isolation.
- [ ] Story file + phase report (build log + smoke evidence) exist.
- [ ] One `--no-gpg-sign` commit; phase directory in `complete/`.
@@ -1,138 +0,0 @@
# Phase 24 — Whole-Document Context: a matched document is never truncated
**Source:** `TODO.md` L3–L4 — *"Documents are truncated for some reason?
This should never happen"* + *"When the LLM matches a chunk it should get
the entire document placed in its context so it can see the whole thing
before answering the question"*
**Story:** `.agent/user_stories/whole-document-context.md` (created by
task 03)
**Context:** `app/rag/retriever.py::select_documents` (the one and only
place document content is cut — the 24k budget), `app/api/chat.py::plan_turn`
(passes the budget on both HIGH and LOW paths), `app/config.py`
(`max_context_chars`), `app/rag/prompts.py` (the shared `[…truncated…]`
marker — still owned by the steering section), `tests/unit/test_retriever.py`
(pins the current cap), `tests/e2e/mock_llm.py` + `tests/e2e/test_chat_rag.py`
(E2E patterns), `.env.example` + `README.md` (document the knob).
## Verified diagnosis (2026-08-24, this conversion — not a guess)
1. **The importer stores the whole file** — `app/rag/importer.py:199–228`
reads each file into `documents.content` in full (sha256 over the whole
content). No truncation at import time.
2. **Chunking never touches the LLM context** — the 2000-char target /
1200-char hard cap only shapes `chunks` rows (retrieval + embeddings);
the chat prompt is built from `documents.content`.
3. **The viewer serves raw content** — `GET /api/documents/content`
(`app/api/docs.py`) returns `doc.content` unchanged; no truncation there
either. So the owner's "truncated for some reason" is *not* a separate
import/viewer bug.
4. **The one and only truncation point is `select_documents()`**
(`app/rag/retriever.py:252–289`): the top-2 documents' combined text is
capped at `BOR_MAX_CONTEXT_CHARS` (default **24 000**) and the
lowest-ranked overflowing document is truncated in place with
`[…truncated…]`. `plan_turn()` (`app/api/chat.py`) passes the budget on
both the HIGH (grounded) and LOW (deflected) paths. The marker the owner
saw in answers comes from here.
5. **The cap is pinned + documented** — `tests/unit/test_retriever.py`
(`test_combined_content_capped_with_truncation_marker`,
`test_single_doc_over_budget_is_truncated_to_budget`,
`test_under_budget_no_truncation`); the knob is in `.env.example`
(`BOR_MAX_CONTEXT_CHARS=24000`) and `README.md:377`.
## Objective
When hybrid retrieval matches a chunk, the LLM sees the **entire** parent
document — the 24k context budget (and `BOR_MAX_CONTEXT_CHARS`) is removed
from the document path, and a dedicated story E2E proves deterministically
that a >24k document — and the *second* document of a >24k pair (the exact
case the old budget cut) — reaches the model whole.
## Owner-confirmed (2026-08-24, roadmap D1–D5)
1. **D1 — no cap at all (revises LOCKED A7):** `select_documents` returns
the full top-N document texts, always. The `max_context_chars` setting
and `BOR_MAX_CONTEXT_CHARS` env var are removed. If a future KB ever
makes the prompt too large for the model, the existing `LLMError` → SSE
`error` path surfaces it loudly — no silent partial context. The
emergency-valve variant (raised cap + warning log) was **explicitly
rejected**.
2. **D2 — `top_n_docs = 2` unchanged** (the TODO is about truncation, not
about how many documents).
3. **D3 — no viewer/import changes** — both already serve full content
(verified diagnosis above).
4. **D4 — E2E evidence via a deterministic mock tail-echo** (repo pattern,
cf. the phase-15 tuning-note echo); the big documents are seeded
directly in the DB inside the E2E test — `tests/fixtures/docs/` must
not grow, because other suites pin `summary.added == 8`.
5. **D5 — no `query_log` schema change** (no new columns, no migration).
## Dependencies
- `03_story_chat_rag` (complete) — the RAG turn + `plan_turn` this phase
modifies.
- `09_story_retrieval_quality` (complete) — hybrid retrieval +
`select_documents` (A7) whose cap this phase revises.
- `15_steering_notes` (complete) — still owns `[…truncated…]` +
`BOR_STEERING_MAX_CHARS` (shared marker; unchanged).
## Tasks
1. `01_remove_context_cap.md` — remove the 24k budget from
`select_documents`, `plan_turn`, `config`, `.env.example`, `README`;
rewrite the unit tests to pin *no* truncation.
2. `02_whole_doc_e2e_suite.md` — mock tail-echo trigger +
`tests/e2e/test_whole_document_context.py` (whole >24k doc, whole second
doc of a >24k pair, small-doc regression).
3. `03_story_docs_plan_commit.md` — story file, PLAN.md A7/§6/§12 revision
(owner permission 2026-08-24), full validation, one `--no-gpg-sign`
commit, phase move.
## Locked decisions
- **A7 revision (owner permission 2026-08-24, D1):** A7's context clause
becomes *"feed the **full text of top-N=2 documents** (deduped)"* — the
"capped at 24k chars" clause is **removed**; matched parent documents are
**never truncated**. The revision note lands in PLAN.md via task 03
(phase-17/19 precedent — recorded, not silently deviated).
- **Steering unchanged** — `BOR_STEERING_MAX_CHARS` (8 000) still caps the
`<tuning>` section with the same `[…truncated…]` marker (phase-15
behavior byte-identical).
- **A16 honored** — dedicated Playwright story suite run in isolation;
unit + integration green; `app/` coverage >90%.
## Testing & Quality
- **Unit:** `tests/unit/test_retriever.py` — `select_documents` returns
byte-identical full content well past the old 24k budget, no marker;
ranking / dedup / n-cap tests unchanged.
- **Integration:** `uv run pytest tests/integration` green — existing
`test_chat_api.py` prompt tests exercise `plan_turn` through the new
signature (no integration test pins the cap — verified).
- **Coverage:** >90% on `app/` held
(`uv run pytest --cov=app --cov-report=term-missing`).
- **E2E (new, isolated):** `tests/e2e/test_whole_document_context.py` —
the tail sentinel of a 30k-char document (and of the *second* document of
a >24k pair) appears in the rendered answer; `[…truncated…]` never
appears; the small-document grounded path is unchanged.
- **Lint/types:** `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] `select_documents` has no budget parameter and never truncates;
`TRUNCATION_MARKER` remains for the steering section only.
- [ ] `BOR_MAX_CONTEXT_CHARS` gone from `app/config.py`, `.env.example`,
and `README.md`.
- [ ] `uv run pytest` green; `uv run pytest --cov=app
--cov-report=term-missing` ≥ today's number.
- [ ] `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`
green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/whole-document-context.md` exists; PLAN.md
carries the A7 revision + §6 bullet + §12 row 24 (owner permission
2026-08-24).
- [ ] One `--no-gpg-sign` commit (below);
`.agent/phases/todo/24_whole_document_context/` moved to
`.agent/phases/complete/`.
## Commit (task 03 — after the phase dir has moved to `complete/`)
```bash
git add -f .agent/phases/complete/24_whole_document_context/ .agent/user_stories/whole-document-context.md .agent/PLAN.md
git add -A .agent/phases/todo/24_whole_document_context/ app/ README.md .env.example tests/
git commit --no-gpg-sign -m "feat(rag): feed whole matched documents to the LLM — no context truncation (A7 revised)"
```
(The *conversion* commit — this phase dir force-added under `todo/` plus
the cleared `TODO.md` — lands separately when the roadmap is written, per
the phase-20–23 precedent, commit `824914c`.)
@@ -1,62 +0,0 @@
# Task 01 — Remove the 24k document-context budget
**Phase:** `24_whole_document_context` · **Source:** `TODO.md:3–4` — *"Documents are truncated for some reason? This should never happen"* + *"When the LLM matches a chunk it should get the entire document placed in its context so it can see the whole thing before answering the question"*
**Story:** `.agent/user_stories/whole-document-context.md` (created by task 03)
## Objective
`select_documents` returns the full text of the top-N parent documents,
always — the budget parameter, the `max_context_chars` setting, the env var,
and the documentation lines are removed, and the unit tests pin the new
no-truncation contract (owner-confirmed D1: no cap at all — the
emergency-valve variant was explicitly rejected).
## Work
1. `app/rag/retriever.py` — `select_documents(chunks, n=None)`: drop the
`max_chars` parameter and the whole `remaining`/truncation loop; return
the deduped top-N documents with their `content` byte-identical. Update
the module docstring (remove the "capped at ``BOR_MAX_CONTEXT_CHARS``"
sentence; state the A7 revision — whole documents, never truncated,
owner permission 2026-08-24) and the `select_documents` docstring.
`TRUNCATION_MARKER` **stays** exported — `app/rag/prompts.py`
(steering section, phase 15) still imports it.
2. `app/api/chat.py` — `plan_turn`: both `select_documents(…)` call sites
(HIGH and LOW paths) drop the `max_chars=settings.max_context_chars`
argument. No other turn-flow change (deflection, steering, `query_log`,
SSE all unchanged).
3. `app/config.py` — delete `max_context_chars: int = 24_000` and its
comment. (Any leftover `BOR_MAX_CONTEXT_CHARS=…` in an operator's
gitignored `.env` is silently ignored via `extra="ignore"` — no
migration, no startup check needed.)
4. `.env.example` — delete the `BOR_MAX_CONTEXT_CHARS=24000` line.
5. `README.md` — delete the settings-table row
`| BOR_MAX_CONTEXT_CHARS | 24000 | cap on total document text sent to the LLM |`
(line ~377). Leave the steering-notes `[…truncated…]` mention (~line 178)
alone — that budget still exists.
6. `tests/unit/test_retriever.py` — drop the `max_chars=…` argument from
every `select_documents` call; replace the three cap tests
(`test_combined_content_capped_with_truncation_marker`,
`test_single_doc_over_budget_is_truncated_to_budget`,
`test_under_budget_no_truncation`) with
`test_content_never_truncated_even_past_old_budget`: two documents of
20 000 + 15 000 chars (35 000 combined — past the old 24 000 cap) come
back with content **byte-identical** (assert equality against the
originals) and `TRUNCATION_MARKER` absent from both. Keep the module
docstring accurate ("ordering and dedup — no context cap, A7 revised").
## Testing & Quality
- Unit: `uv run pytest tests/unit/test_retriever.py -v` green — ranking,
dedup, n-cap, and the new no-truncation test.
- Integration: `uv run pytest tests/integration` green — the existing
`test_chat_api.py` prompt tests exercise `plan_turn` through the new
signature (no integration test pins the cap — verified during conversion).
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` ≥ today's
number (the >90% gate).
## Completion Criteria
- [ ] `rg "max_context" app/ tests/ .env.example README.md` → **no matches**
(`steering_max_chars` is a different setting and must remain).
- [ ] `uv run pytest tests/unit/test_retriever.py -v` green.
- [ ] `uv run pytest` green; `uv run pytest --cov=app
--cov-report=term-missing` ≥ today's number.
- [ ] No behavior change outside the context budget — deflection, steering,
viewer, and import behave byte-identically to before.
@@ -1,84 +0,0 @@
# Task 02 — Story E2E: whole documents reach the LLM (mock tail-echo)
**Phase:** `24_whole_document_context` · **Source:** `TODO.md:3–4` — *"Documents are truncated for some reason? This should never happen"* + *"When the LLM matches a chunk it should get the entire document placed in its context so it can see the whole thing before answering the question"* (this task makes the no-truncation contract provable end-to-end)
**Story:** `.agent/user_stories/whole-document-context.md` (created by task 03)
## Objective
`tests/e2e/test_whole_document_context.py` proves, with the deterministic
mock, that a matched document **larger than the old 24k cap** — and the
**second** document of a pair whose combined size exceeds it (the exact case
the old budget cut) — reach the LLM whole.
## Work
1. `tests/e2e/mock_llm.py` — add one user-message trigger, same pattern as
`LONG_ANSWER_TRIGGER` and the phase-15 tuning-note echo:
- `END_OF_NOTES_TRIGGER = "show the end of your notes"` — verified
2026-08-24: no existing E2E question or fixture file contains the
phrase, so every other suite is unaffected.
- In `compose_answer`, after the `DEFLECT_MODE` check and before the
generic branch: when the trigger is present in the user message, the
answer quotes the **tail of the context** — e.g.
`f"…and the very end of my notes reads: “{_context(body)[-160:]}” (Deterministic mock answer for E2E.)"`.
Byte-stable across runs, so a sentinel placed at the *end* of a
document appears in the rendered answer **iff the whole document was
in the prompt**. (The tail includes the closing `</documents>` —
harmless for `to_contain_text` sentinel assertions.)
- Keep the mock docstring's trigger list updated.
2. `tests/e2e/test_whole_document_context.py` — new story suite. Reuse the
`page`, `app_url`, `mock_llm`, `db_ready` fixtures and the
TRUNCATE-then-seed pattern from `tests/e2e/test_chat_rag.py` /
`tests/integration/test_document_content.py`. **Seed the big documents
directly via SQLAlchemy** (a `documents` row + 2–3 `chunks` rows, each
chunk with `embedding = embed_text(chunk_text)` imported from
`tests.e2e.mock_llm` — the same deterministic bag-of-words vector the
mock computes, so the question's live mock embedding genuinely overlaps).
**Do not add fixture files** — `tests/fixtures/docs/` must stay at its
current 8 files; other suites pin `summary.added == 8`.
- Content helper: a ~30 000-char document = repeated
"gitlab install playbook" body (tokens shared with the question →
hybrid hit) whose **last line is a unique sentinel**, e.g.
`WHOLE-DOC-TAIL-GITLAB-<test-unique>`.
- `test_whole_document_over_old_cap_reaches_llm` — seed one 30 000-char
document (past the old 24 000 cap); ask
`"Show the end of your notes about the gitlab install playbook, please."`;
assert the rendered brain bubble contains the tail sentinel **and**
`"Deterministic mock answer for E2E"`, contains **no**
`[…truncated…]`, the document's `.source-chip` renders, and the
`query_log` row has `deflected == False`.
- `test_second_document_of_over_cap_pair_reaches_llm` — seed two
documents of ~16 000 chars each (32 000 combined — under the old
budget the lower-ranked one was truncated in place). Doc 1's first
chunk carries the question's key tokens ("gitlab install playbook");
doc 2's first chunk carries a weaker overlap ("playbook", "notes") so
doc 1 ranks first and doc 2 is the **last block inside
`<documents>`** — i.e. the tail-echo surfaces doc 2's sentinel. Assert
doc 2's tail sentinel in the answer. If the ranking ever flips locally,
strengthen doc 1's token overlap (repeat the question phrase in its
first chunk) until two consecutive isolated runs are stable — do not
weaken the assertion.
- `test_small_document_path_unchanged` — regression: re-import the
standard fixtures via the real importer (the `test_chat_rag.py`
`_import_fixtures` pattern), ask the usual on-topic question
(`"How is my Kubernetes cluster set up?"`) → grounded answer +
`kubernetes.md` chip, no `[…truncated…]` (the under-24k path is
byte-identical to before).
- Header comment: story path + run-in-isolation command
(`uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`).
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`
green **in isolation** (DB up: `podman compose up -d db`).
- E2E regressions: `uv run pytest tests/e2e/test_chat_rag.py
tests/e2e/test_honest_deflection.py -v --no-cov` still green (the mock
change is additive — the trigger phrase appears in no existing question).
- Coverage: the >90% `app/` gate is unaffected (tests + mock only) — re-run
`uv run pytest --cov=app --cov-report=term-missing` to prove it.
- Lint/types: `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`
— all three tests green in isolation.
- [ ] `uv run pytest tests/e2e/test_chat_rag.py
tests/e2e/test_honest_deflection.py -v --no-cov` green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `tests/fixtures/docs/` unchanged (still 8 files).
@@ -1,76 +0,0 @@
# Task 03 — Story file, PLAN.md revision, validation, commit
**Phase:** `24_whole_document_context` · **Source:** `TODO.md:3–4` — *"Documents are truncated for some reason? This should never happen"* + *"When the LLM matches a chunk it should get the entire document placed in its context so it can see the whole thing before answering the question"*
**Story:** `.agent/user_stories/whole-document-context.md` (created by this task)
## Objective
Record the user story and the A7 revision (owner permission 2026-08-24),
run the full validation gate, land one atomic commit, and move the phase
to `complete/`.
## Work
1. `.agent/user_stories/whole-document-context.md` — story file in house
style (cf. `.agent/user_stories/sources-midstream.md`):
- Header: `**Phase:** 24_whole_document_context · **E2E:**
tests/e2e/test_whole_document_context.py`.
- Bug report section: the two TODO.md items (L3–L4), verbatim.
- Narrative: *Given* a question whose chunk matches a document,
*when* the turn assembles context, *then* the LLM receives the
**entire** parent document — never a `[…truncated…]`-cut version —
so the answer is grounded in the whole note.
- Owner-confirmed (2026-08-24, roadmap D1–D5) — the five decisions from
the phase overview, verbatim.
- Acceptance criteria: no budget parameter in `select_documents`;
`BOR_MAX_CONTEXT_CHARS` gone from settings/env/README; the story E2E's
three tests green in isolation; steering-note truncation (phase 15)
unchanged; unit+integration green, coverage >90%, one
`--no-gpg-sign` commit.
- Playwright Mapping Rule: the three tests of task 02, one line each.
2. `.agent/PLAN.md` revisions (phase-17/19 precedent — record the owner
permission, never deviate silently):
- §2, A7 row: replace *"feed the **full text of top-N=2 documents**
(deduped, capped at 24k chars)"* with *"feed the **full text of
top-N=2 documents** (deduped)"* and append the rationale addendum:
"A7 revised 2026-08-24 — matched documents never truncated (owner:
'this should never happen'; emergency-valve variant rejected)".
- §2, revision-note block under the anchors table: add
**A7 revision (phase 24, owner permission 2026-08-24):** the
`[…truncated…]` cap on document context is removed —
`select_documents` always returns the full top-N texts;
`BOR_MAX_CONTEXT_CHARS` is gone. The steering section
(`BOR_STEERING_MAX_CHARS`, phase 15) keeps its budget and the shared
marker.
- §6, retrieval bullet: *"top 2 → full content, concatenated, truncated
to `BOR_MAX_CONTEXT_CHARS` (24k) with a `[…truncated…]` marker"* →
*"top 2 → full content, concatenated — **never truncated** (A7
revised, phase 24, owner permission 2026-08-24)"*.
- §12 roadmap table: add row 24 —
`| 24 | 24_whole_document_context.md | whole-document-context.md | tests/e2e/test_whole_document_context.py |`
plus a footnote: "Row 24 added 2026-08-24 with owner permission —
A7's 24k context cap removed (documents are never truncated)".
3. Final validation (AGENTS.md §9 — all must be green before the commit):
- `uv run pytest` (unit + integration)
- `uv run pytest --cov=app --cov-report=term-missing` (>90% gate)
- `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`
(in isolation)
- `uv run ruff check . && uv run pyright`
4. Commit + phase move (AGENTS.md §8; use the commit block in
`00_phase.md` — it assumes the move happens first):
`mv .agent/phases/todo/24_whole_document_context
.agent/phases/complete/`, then the two `git add` lines + one
`--no-gpg-sign` commit.
## Testing & Quality
- All AGENTS.md §9 gates: unit + integration green, `app/` coverage >90%,
story E2E green in isolation, lint + types clean.
## Completion Criteria
- [ ] `.agent/user_stories/whole-document-context.md` exists (story,
D1–D5, acceptance criteria, Playwright mapping).
- [ ] `.agent/PLAN.md` carries the A7 row + revision note (§2), the §6
bullet, and the §12 row 24 + footnote — every one marked
"owner permission 2026-08-24".
- [ ] All validation commands green (work step 3).
- [ ] One `--no-gpg-sign` commit;
`.agent/phases/complete/24_whole_document_context/` exists and
`.agent/phases/todo/24_whole_document_context/` is gone.
@@ -1,128 +0,0 @@
# Task 01 — Still Background: Static Grid + Three Opacity-Only Glow Fades
**Phase:** `25_background_no_motion` · **Story:** `.agent/user_stories/background-no-motion.md`
## Objective
Redesign the background block of `frontend/assets/styles.css` to the
owner's spec (no movement; different bright spots slowly fading in and
out), and pin the new contract at source level.
## Work
0. **Capture the "before" evidence FIRST** (pre-change): with the DB up
(`podman compose up -d db`), boot the app the same way
`tests/e2e/conftest.py`'s `app_server` fixture does (uvicorn
`app.main:app` on a free port, wait for it to answer), and with a small
throwaway Playwright script screenshot the `/` page at 1280×800 →
`.agent/screenshots/25_background_no_motion/before.png`, then again
~4s later → `before_4s.png` (the pair shows the down-right jitter +
the uniform brightness pulse). Kill the server when done.
1. `frontend/assets/styles.css` — **grid layer `body::before`:** remove
the `animation: bg-grid-drift 60s linear infinite;` declaration and
delete the whole `@keyframes bg-grid-drift { … }` block. Keep the 44px
cells, the 60% `--line` alpha 1px lines, and the widened radial mask —
the grid remains as a **static texture**. Update the block's comment:
drift removed (owner 2026-08-25) — the 0.73px/s sub-pixel drift
rasterizes as a once-per-second down-right jitter; the owner wants no
movement.
2. `frontend/assets/styles.css` — **glow layers.** Replace the
whole-layer breathe with three independent spot layers:
- `body::after` — keep the phase-08 indigo spot exactly:
`background-image: radial-gradient(circle 56rem at 12% 8%,
rgb(109 120 242 / 0.14), transparent 62%);` and
`animation: bg-glow-a 26s ease-in-out infinite;`
- **new `html::before`** — the phase-08 cyan spot:
`background-image: radial-gradient(circle 60rem at 88% 92%,
rgb(34 211 238 / 0.10), transparent 62%);` and
`animation: bg-glow-b 34s ease-in-out -12s infinite;`
- **new `html::after`** — a third indigo spot:
`background-image: radial-gradient(circle 52rem at 14% 86%,
rgb(109 120 242 / 0.09), transparent 62%);` and
`animation: bg-glow-c 42s ease-in-out -23s infinite;`
- All three layers (and the grid layer) must declare:
`content: ""; position: fixed; inset: 0; z-index: -1;
pointer-events: none;` — no `transform`, no `background-position`,
no `filter` on any of them.
- Delete `@keyframes bg-glow-breathe { … }` and add the three
**opacity-only** keyframe blocks (nothing else may appear in any
`bg-*` keyframe):
```css
@keyframes bg-glow-a { 0%, 100% { opacity: 0.25; } 50% { opacity: 1; } }
@keyframes bg-glow-b { 0%, 100% { opacity: 0.20; } 50% { opacity: 1; } }
@keyframes bg-glow-c { 0%, 100% { opacity: 0.15; } 50% { opacity: 1; } }
```
3. `frontend/assets/styles.css` — **reduced motion:** in the existing
`@media (prefers-reduced-motion: reduce)` block that stills the
background (currently `body::before, body::after { animation: none; }`,
right after the spinner block), extend the selector list to all four
layers: `body::before, body::after, html::before, html::after
{ animation: none; }`. Do not touch the typing/spinner/thinking
reduced-motion blocks.
4. `frontend/assets/styles.css` — **section comment:** rewrite the
"Animated background" comment to document the new spec: no movement
(owner 2026-08-25); three independent soft spots, opacity-only fades
at 26/34/42s with negative delays → out of phase, so the total light
fluxuates smoothly and irregularly; `html::before`/`html::after` are
background layers (root stacking context: they paint above the
`var(--bg)` canvas and below the transparent, non-stacking `<body>`'s
content — the no-occlusion contract is unchanged).
5. **New `tests/unit/test_background_no_motion.py`** (repo source-pin
pattern — reuse the helpers from `tests/unit/test_background_animation.py`
(`_css`, `_css_no_comments`, `_rule_block`; note `_rule_block` matches
top-level `selector { … }`, which works for `html::before`/`html::after`
as written in step 2). Pin, at minimum:
- `body::before` carries **no `animation`** declaration;
`@keyframes bg-grid-drift` is absent from the file; the grid keeps
its static texture (`background-size: 44px 44px`, the two 60%-alpha
1px line gradients, the widened mask, both mask properties).
- `body::after` runs `bg-glow-a 26s ease-in-out infinite`;
`html::before` runs `bg-glow-b 34s ease-in-out -12s infinite`;
`html::after` runs `bg-glow-c 42s ease-in-out -23s infinite`;
each glow layer's `background-image` is exactly the single radial
gradient from step 2 (color, radius, position, 62% transparent stop).
- **No movement:** parse every `@keyframes bg-*` rule in the file —
the set of declared property names across all keyframe frames is
exactly `{opacity}` (no `transform`, `scale`, `background-position`,
…); none of the three glow layers declares `transform` or
`background-position`.
- The three glow durations are distinct and each ≥ 20s (slow).
- All four pseudo-layers: `position: fixed`, `inset: 0`,
`z-index: -1`, `pointer-events: none`, `content: ""`.
- Plumbing: `html` keeps `background: var(--bg)`; `body` keeps
`background: transparent` and declares none of `z-index`,
`transform`, `opacity`, `filter`.
- The reduced-motion block stills all four layers (all four selectors
present together with `animation: none`).
- No `filter` in any background layer block; no `blur` anywhere in
the file (comments stripped).
6. **Adapt `tests/unit/test_background_animation.py`** (the phase-22
source pins) to the new contract so the whole unit suite is green:
replace the grid-drift and breathe tests with their new-contract
equivalents (where a check is already covered by the new file, keep the
file self-contained rather than importing from it — the repo pattern is
local pins); keep the generic layer-plumbing tests
(`test_both_layers_are_fixed_zminus1_noninteractive` — extend it to
cover `html::before`/`html::after` — and
`test_html_owns_bg_and_body_stays_transparent`) and the no-blur/no-JS
anchor test; update the module docstring to describe the phase-25
design and point at `.agent/user_stories/background-no-motion.md`.
## Testing & Quality
- `uv run pytest tests/unit -v` green (new + adapted pins).
- `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ the
pre-change number (`app/` is untouched — the >90% gate holds).
- `uv run ruff check . && uv run pyright` clean.
- Coverage **>90%** on new/modified code: the functional change is CSS;
the new/modified Python is pytest source pins, exercised in full.
## Completion Criteria
- [ ] `styles.css`: no `bg-grid-drift`, no `bg-glow-breathe`, no
`animation` on `body::before`; exactly three opacity-only
`bg-glow-a/b/c` cycles on `body::after`, `html::before`,
`html::after` (26s/34s/42s, delays 0/−12s/−23s); reduced-motion
stills all four layers; comments document the owner's no-movement
spec.
- [ ] `uv run pytest tests/unit` green with
`tests/unit/test_background_no_motion.py` present and passing.
- [ ] `uv run pytest --cov=app` TOTAL ≥ pre-change; ruff + pyright clean.
- [ ] `.agent/screenshots/25_background_no_motion/before.png` and
`before_4s.png` captured **before** the CSS change.
@@ -1,44 +0,0 @@
# Phase 26 — Document Modal Viewer
**Source:** `TODO.md L4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
**Story:** `.agent/user_stories/document-modal.md`
**Context:** Phase 10 added the separate `/document.html` viewer page; phase 19 added the shared header bar that now lives on every page. The document content is served by the stateless `GET /api/documents/content` endpoint (PLAN §4).
## Objective
Stop opening cited documents in a new page/tab. Clicking a source chip or a Sources-table path link now opens the document in an **almost-fullscreen modal overlay** on the current page, fed by the same `/api/documents/content` endpoint. The existing `/document.html` page stays as the no-JS / direct-link fallback and its behaviour is unchanged.
## Dependencies
- `10_story_document_viewer` (complete) — the `/document.html` page, the `document.js` renderer, the `renderMarkdown` escape-first renderer in `markdown.js`, and the `#doc-content` / `.doc-md` / `.doc-raw` markup this phase reuses inside the modal.
- `19_shared_header` (complete) — the shared header bar the modal sits under; the modal must not disturb the header.
- `08_story_dark_tech_theme` (complete) — the Phase-08 tokens and the ≥4.5:1 contrast / `prefers-reduced-motion` contract the modal must honour.
## Tasks
1. `01_modal_css_and_html.md` — the modal CSS (overlay, backdrop, close button, scrollable content area) + inject the modal skeleton into `index.html`
2. `02_app_js_modal_intercept.md` — intercept document links in `app.js` + `sources.js`, fetch content via `/api/documents/content`, render inside the modal
3. `03_document_js_modal_mode.md` — adapt `document.js` to optionally render in modal mode (reuse the same API call) for the direct-link fallback path
4. `04_e2e_regression_suite.md` — update `test_document_viewer.py` to verify modal behaviour; the story gate, run in isolation
## Testing & Quality
- Unit/integration: none required for the modal itself (frontend-only); the `/api/documents/content` endpoint is unchanged (no `app/` change → no coverage delta).
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
- E2E: `tests/e2e/test_document_viewer.py` rewritten for the modal contract (task 4), green **in isolation** (prereq `podman compose up -d db`).
## Completion Criteria
- [ ] Clicking a source chip (chat) or a Sources-table path link opens the document in an almost-fullscreen modal on the **same page** (no new tab, no navigation).
- [ ] The modal renders the same content the `/document.html` page renders: md/markdown via the shared renderer (`.doc-md`), other formats in `<pre class="doc-raw">`, source/format/path/indexed/chunks meta.
- [ ] The modal has a visible close control, closes on Escape, closes on backdrop click, and keeps the dark theme + a11y frame (skip-link, focus trap, `:focus-visible`, aria-label).
- [ ] The existing `/document.html` page still works unchanged (direct link, back button, XSS-safe rendering, not-found state).
- [ ] No CDN tags on any touched page; every asset reference is same-origin or `data:`.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (gate >90%).
- [ ] `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5): modal content uses the standard centered column width for md; backdrop behind content; no 360px overflow.
- [ ] `.agent/user_stories/document-modal.md` exists.
- [ ] One `--no-gpg-sign` commit staging only this phase's files; `.agent/phases/todo/26_document_modal_viewer/` moved to `.agent/phases/complete/`.
## Locked decisions
- **No backend change** — the modal reuses `GET /api/documents/content` unchanged (A10 untouched: the API stays stateless).
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, zero new packages, no new assets, system font stack; the modal is pure CSS + JS.
- **No anchor revised** — this is a UI-behaviour change (PLAN §7.5 gains `#doc-modal`, `#doc-modal-backdrop`, `#doc-modal-close`, `#doc-modal-content`); the `/document.html` page and its story are unchanged.
- **A16 honoured** — one story E2E suite (rewritten) + adapted regressions.
- **A17 honoured** — one atomic `--no-gpg-sign` commit.
@@ -1,62 +0,0 @@
# Task 01 — Modal CSS + HTML skeleton
**Phase:** `26_document_modal_viewer` · **Source:** `TODO.md:4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
**Story:** `.agent/user_stories/document-modal.md`
## Objective
Add the modal markup to `index.html` and the CSS that styles an almost-fullscreen overlay (backdrop + panel + close button + scrollable content) using the Phase-08 tokens.
## Work
1. `frontend/index.html` — insert the modal skeleton just before the closing `</body>` (after the existing script tags, or before them — order doesn't matter for a static skeleton). The skeleton:
```html
<div class="doc-modal" id="doc-modal" hidden>
<div class="doc-modal-backdrop" id="doc-modal-backdrop" aria-hidden="true"></div>
<div class="doc-modal-panel" role="dialog" aria-modal="true" aria-labelledby="doc-modal-title" aria-describedby="doc-modal-desc">
<header class="doc-modal-header">
<h2 class="doc-modal-title" id="doc-modal-title">Loading…</h2>
<div class="doc-modal-actions">
<a class="doc-modal-open" id="doc-modal-open" target="_blank" rel="noopener" hidden aria-label="Open in full page">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"><path d="M18 13v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/><path d="M15 3h6v6"/><path d="M10 14 21 3"/></svg>
<span>Full page</span>
</a>
<button type="button" class="doc-modal-close" id="doc-modal-close" aria-label="Close document">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"><path d="M18 6 6 18M6 6l12 12"/></svg>
</button>
</div>
</header>
<div class="doc-modal-meta" id="doc-modal-meta" aria-live="polite"></div>
<p class="visually-hidden" id="doc-modal-desc" role="status">Document content is loading.</p>
<main class="doc-modal-content" id="doc-modal-content" tabindex="-1">
<p class="doc-modal-loading" role="status">Loading document…</p>
</main>
</div>
</div>
```
The `hidden` attribute keeps the modal off until JS opens it. The `#doc-modal-open` "Full page" link points at the same `/document.html?source=…&path=…&modal=…` URL the modal will build so a user can still open the dedicated page if JS is off.
2. `frontend/assets/styles.css` — add a `--doc-modal-*` block (Phase-08 tokens). Styling contract:
- `.doc-modal` — `position: fixed; inset: 0; z-index: 1000;` (above the shared header and every page layer, below the phase-25 background which is `z-index: -1`); the panel is flex, column; the backdrop + panel fill the viewport.
- `.doc-modal-backdrop` — `position: fixed; inset: 0; background: rgba(10,14,23,0.82);` backdrop blur is **not** used (phase-08 no-blur perf anchor); `opacity` transition 120ms.
- `.doc-modal-panel` — `display: flex; flex-direction: column; width: min(1100px, 96vw); height: 92vh; margin: auto; background: var(--surface, #121a2e); border: 1px solid var(--line, #232b52); border-radius: 12px; box-shadow: 0 24px 80px rgba(0,0,0,.55);` — "almost-fullscreen" = 96vw × 92vh, centered.
- `.doc-modal-header` — sticky top, same height/spacing as the doc header (64px / 58px pins from phase 12); title uses `--ink`; close button ≥44px target, focus-visible ring.
- `.doc-modal-content` — `flex: 1; overflow: auto;` (vertical scroll inside the panel, not the viewport); padding; the md content reuses `.doc-md` (≤46rem centered column) — the modal just provides the scroll container. For wide raw formats the `.doc-raw` pre already has `overflow-x: auto`.
- `.doc-modal-meta` — reuses the `.doc-meta` styling already defined for the viewer page (source/format/path/indexed/chunks badges); keep it compact (single row, wrap).
- `.doc-modal-close` — icon-only button, `aria-label` kept, `:focus-visible` 3px ring.
- Transitions respect `prefers-reduced-motion: reduce` (no opacity/transform animation, or `animation: none` under the reduced-motion media query — same pattern as the phase-25 background layers).
- `.doc-modal[hidden]` — `display: none` (the `hidden` IDL attribute default already hides it; add the rule to be explicit and testable).
- Ensure the modal panel does not add horizontal width at 360px (no `box-sizing` surprises; the panel is `96vw` ≤ viewport).
3. Verify the new CSS classes do not collide with any existing selector in `styles.css` (grep for `.doc-modal`, `.doc-modal-`).
## ASSUMPTIONS
- The modal panel is `96vw × 92vh` ("almost-fullscreen"). If the owner wants a different fraction, that's a follow-up.
- The "Full page" link is admin-agnostic (it just opens `/document.html`); it is shown for everyone since the viewer is public.
- The modal uses the existing `.doc-meta` badge classes already defined for the viewer page (no duplicate styling).
## Testing & Quality
- No unit/integration test for static CSS/HTML.
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
## Completion Criteria
- [ ] `index.html` contains the `.doc-modal` skeleton with the documented ids (`#doc-modal`, `#doc-modal-backdrop`, `#doc-modal-panel`, `#doc-modal-close`, `#doc-modal-title`, `#doc-modal-meta`, `#doc-modal-content`, `#doc-modal-open`).
- [ ] The modal CSS block is present, uses Phase-08 tokens, has no `filter: blur`/`backdrop-filter`, and the panel is `96vw × 92vh` centered.
- [ ] No selector collision (grep clean).
- [ ] `prefers-reduced-motion` stills any modal transition.
@@ -1,40 +0,0 @@
# Task 02 — Intercept document links → modal
**Phase:** `26_document_modal_viewer` · **Source:** `TODO.md:4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
**Story:** `.agent/user_stories/document-modal.md`
## Objective
Intercept document links on the chat page (`app.js` source chips) and the Sources page (`sources.js` table links): instead of navigating to `/document.html` in a new tab, fetch the document via `GET /api/documents/content` and render it inside the modal from task 01.
## Work
1. `frontend/assets/document.js` — extract the rendering logic into a reusable, importable function so both the standalone page (task 03) and the modal share the exact same renderer. Specifically:
- Export `renderDocument(doc, { containerEl, metaEl, titleEl })` that populates a title element, a meta element (the `.doc-meta` badges), and a content element (`.doc-md` for markdown via `renderMarkdown`, `<pre class="doc-raw">` otherwise). Keep the escape-first XSS-safety contract (`innerHTML` only through `renderMarkdown`; `textContent` for raw + badges).
- The standalone `document.html` page keeps its own `load()` that calls `renderDocument` with its page elements (task 03 wires this).
2. `frontend/assets/app.js` — add a `openDocumentModal(source, path)` helper:
- Build the modal URL: `/api/documents/content?source=…&path=…` (same encoding the chips already use).
- Show the modal: set `#doc-modal.hidden = false`, set the loading state, move focus into `#doc-modal-content` (a11y — the panel is `tabindex="-1"`).
- `fetch(contentUrl)` → on `!r.ok` render a short "document not found" line in the content area; on success call `renderDocument` with `#doc-modal-title`, `#doc-modal-meta`, `#doc-modal-content`.
- The "Full page" link (`#doc-modal-open`) is set to the `/document.html?source=…&path=…` URL on open.
- Keep the existing `documentUrl()` builder for the "Full page" link (unchanged output).
- Add modal close behaviour: `#doc-modal-close` click → `closeDocumentModal()`; backdrop click → close; `Escape` key → close; closing restores focus to the link that opened the modal (best-effort — store the triggering element).
3. `frontend/assets/app.js` — wire the source chips: replace `chip.target = "_blank"` navigation with `chip.addEventListener("click", e => { e.preventDefault(); e.stopPropagation(); openDocumentModal(s.source, s.path, chip); })`. Keep the `title`/aria-label truncation logic the chips already have. The chip keeps its `href` too (no-JS fallback would navigate to `/document.html`).
4. `frontend/assets/sources.js` — wire the table links the same way: the `.doc-link` click is intercepted, `preventDefault`, and `openDocumentModal(d.source, d.path, link)` is called. Since `openDocumentModal` lives in `app.js` (the chat page module) and `sources.js` is a separate module, **export** `openDocumentModal` from `app.js` and import it in `sources.js` — but `app.js` is loaded as a module on the chat page only. To avoid a second module instance, move the shared modal logic into a small new module `frontend/assets/document-modal.js` (task 02 step 1 refined below) and have both `app.js` and `sources.js` import it.
- **Refined split:** create `frontend/assets/document-modal.js` exporting `openDocumentModal(source, path, triggerEl)` and `closeDocumentModal()`. This module owns the modal DOM wiring (close on ESC / backdrop / button, focus management) and the `fetch` + `renderDocument` call. `app.js` and `sources.js` just call `openDocumentModal(...)` from their click handlers. This is the cleanest single-implementation approach (mirrors how `header.js` is the single owner of the shared header).
- `document.js` (standalone page) also imports `renderDocument` from itself (or a shared `document-render.js`) — keep the standalone page self-contained; it doesn't need the modal module.
## ASSUMPTIONS
- The modal module (`document-modal.js`) is a classic or module script loaded on both `index.html` and `sources.html`. It's a module (imports `renderDocument` from `document.js`), so both pages must load it via `<script type="module">`. `document.js` will export `renderDocument`.
- Close-on-`Escape` and close-on-backdrop are modal UX standards; the owner's item says "modal, not a new page", which implies standard modal affordances.
- The "Full page" link remains for users who want the dedicated viewer; it is optional and doesn't interfere with the modal.
## Testing & Quality
- No unit/integration test (frontend-only).
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
## Completion Criteria
- [ ] Clicking a source chip or a Sources-table path link opens the modal and renders the document (md via `.doc-md`, other formats via `.doc-raw`).
- [ ] The modal closes on button click, on backdrop click, and on `Escape`; focus returns to the triggering control.
- [ ] No new tab opens from either link type.
- [ ] The "Full page" link still navigates to `/document.html` (unchanged).
- [ ] XSS-safe rendering preserved (markdown escaped, raw set via `textContent`).
- [ ] Both pages load the modal module without a duplicate-module error.
@@ -1,29 +0,0 @@
# Task 03 — Standalone viewer page reuses the shared renderer
**Phase:** `26_document_modal_viewer` · **Source:** `TODO.md:4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
**Story:** `.agent/user_stories/document-modal.md`
## Objective
Keep `/document.html` working exactly as before (it is the no-JS / direct-link fallback) but refactor its `document.js` so the markdown/raw rendering lives in a shared function the modal module can reuse. No behavioural change to the standalone page.
## Work
1. `frontend/assets/document.js` — split the current inline renderer into an exported `renderDocument(doc, { titleEl, metaEl, contentEl })` function (the escape-first contract: markdown → `renderMarkdown` into a `.doc-md` div; other formats → `<pre class="doc-raw">` via `textContent`; badges via `textContent`). The page's existing `load()` IIFE now calls `renderDocument` with the page's `#doc-title`, `#doc-meta`, `#doc-content` elements. Everything else in `document.js` (query-param parsing, `back` target, not-found card, shared header wiring, New Chat button, `mainEl.focus()`) is **unchanged**.
2. `frontend/assets/document-modal.js` (new) — imports `renderDocument` from `./document.js`. Owns `openDocumentModal(source, path, triggerEl)` and `closeDocumentModal()` (see task 02). On open it fetches `/api/documents/content` and calls `renderDocument(doc, { titleEl: #doc-modal-title, metaEl: #doc-modal-meta, contentEl: #doc-modal-content })`. It also sets `#doc-modal-open.href` to the `/document.html?source=…&path=…` URL.
3. `frontend/index.html` — load `document-modal.js` as a module (add `<script type="module" src="/assets/document-modal.js"></script>` alongside the existing `app.js` module script). `index.html` already loads `markdown.js` as a classic script (needed by `renderDocument`).
4. `frontend/sources.html` — load `document-modal.js` as a module (it needs `document.js`'s `renderDocument`, so both `document.js` and `document-modal.js` must be module scripts; `markdown.js` classic script stays). The Sources page currently loads `sources.js` as a module; add the modal module script next to it.
5. Verify the no-CDN integration test (`tests/integration/test_api.py::test_index_html_served_locally`) still passes — the new module scripts are same-origin `<script src>`, so they satisfy the "local asset" rule. If the test counts script tags, update the expected count.
## ASSUMPTIONS
- `renderDocument` depends on `renderMarkdown` (from `markdown.js`), which is a classic script — so `document.js` (module) importing nothing but using the global `renderMarkdown` is fine, and `document-modal.js` (module) importing `renderDocument` from `document.js` also relies on the global `renderMarkdown` being present. Both pages load `markdown.js` before the module scripts (hoisting guarantees module scripts run after classic scripts already on the page).
- The standalone page's `document.js` no longer needs to be a module for its own rendering — but it stays a module because it imports `header.js` (shared header). Keep it a module.
## Testing & Quality
- No unit/integration test for the refactor itself.
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
## Completion Criteria
- [ ] `/document.html?source=…&path=…` still renders title/meta/content exactly as before (md in `.doc-md`, raw in `<pre.doc-raw>`).
- [ ] The not-found state, `back` target, shared header, and New Chat button on `/document.html` are unchanged.
- [ ] `document-modal.js` is loaded on `index.html` and `sources.html`; `document.js` exports `renderDocument`.
- [ ] No-CDN test still passes (new scripts are same-origin).
- [ ] No console errors on any of the three pages.
@@ -1,33 +0,0 @@
# Task 04 — E2E story suite (modal) + regressions
**Phase:** `26_document_modal_viewer` · **Source:** `TODO.md:4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
**Story:** `.agent/user_stories/document-modal.md`
## Objective
Rewrite the phase-10 E2E suite to assert the **modal** contract (open in a modal on the same page, no new tab; close on button/Escape/backdrop; dark theme; no CDN; a11y frame), and confirm the standalone `/document.html` page still works.
## Work
1. `tests/e2e/test_document_viewer.py` — rewrite for the modal contract (the seeding harness from the phase-10 file — `_import_fixtures` / `_reset_db` / `_run_in_thread` — stays identical; only the assertions change):
- `test_source_chip_opens_modal` — from the chat page, ask the QUESTION, wait for the `kubernetes.md` source chip, click it (no `target=_blank` click → `expect_popup`); assert the modal `.doc-modal` is visible, NOT hidden; `#doc-modal-title` = "Kubernetes Homelab Cluster"; `#doc-content`/`.doc-md` present; content text "Talos Linux on three nodes". Assert the page URL is unchanged (still `/`).
- `test_sources_row_opens_modal` — log in, find the `gitlab-compose.yaml` row link, click it; assert the modal is open with the yaml rendered in `<pre.doc-raw>` containing "gitlab/gitlab-ce:17.2.1-ce.0", mono font.
- `test_modal_closes_on_button_escape_and_backdrop` — open the modal, click `#doc-modal-close` → hidden; re-open, click backdrop → hidden; re-open, press Escape → hidden.
- `test_modal_focus_and_a11y` — on open, focus is inside `#doc-modal-content`; the panel has `role="dialog"` + `aria-modal="true"`; the close button has `aria-label`.
- `test_modal_xss_safe` — seed an XSS fixture doc, open via modal, assert the `<script>` shows as escaped text and no dialog fires (same as the phase-10 test but inside the modal).
- `test_standalone_page_still_works` — the phase-10 assertions for `/document.html` (title/content/format badge, not-found state, dark theme, no-CDN, a11y frame, `#doc-content .doc-md` ≤ 736px) are **kept** — the dedicated page must not regress.
- `test_modal_theme_and_no_cdn` — dark theme (document background `rgb(10,14,23)`), and the modal panel uses Phase-08 surface colour.
2. `tests/integration/test_api.py` — if the no-CDN test counts `<script>` tags on `index.html` / `sources.html`, bump the expected count to include `document-modal.js` (and confirm `document.html` count is unchanged).
3. Regressions to run green in isolation after the change: `test_document_back_navigation.py` (source chips now open a modal; verify the back-navigation story doesn't assert a new tab — if it does, adapt), `test_header_consistency.py` (new module scripts don't disturb the header), `test_smoke.py`.
4. `.agent/user_stories/document-modal.md` — write the story file mapping the modal behaviour to the E2E scenarios above.
## ASSUMPTIONS
- The phase-10 `expect_popup` calls are removed (no new tab); the modal opens in-page.
- The standalone page test is kept to guard the no-JS / direct-link fallback.
## Testing & Quality
- E2E: `tests/e2e/test_document_viewer.py` rewritten — the story gate, green **in isolation** (prereq `podman compose up -d db`).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` green in isolation.
- [ ] `test_document_back_navigation.py`, `test_header_consistency.py`, `test_smoke.py` green in isolation (adapted if they asserted a new tab).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,46 +0,0 @@
# Task 01 — Steering update endpoint
**Phase:** `27_global_tuning` · **Source:** `TODO.md:3 — "Add a way to add 'global tuning' without having a chat to reply to. Also previous tunes should be editable."`
**Story:** `.agent/user_stories/global-tuning.md`
## Objective
Add a `PUT /api/steering/{note_id}` endpoint so an existing steering note can be updated in place (the chat-page + header UI currently support create + delete only).
## Work
1. `app/schemas.py` — add `SteeringNoteUpdate(BaseModel)`:
```python
class SteeringNoteUpdate(BaseModel):
"""``PUT /api/steering/{id}`` body: a new tuning instruction."""
note: str = Field(min_length=1, max_length=2000)
@field_validator("note", mode="before")
@classmethod
def _trim_note(cls, v): return v.strip() if isinstance(v, str) else v
```
Mirror `SteeringNoteIn`'s trim-before-length-constraint behaviour so an empty/whitespace body is a 422.
2. `app/api/steering.py` — add the route:
```python
@router.put("/{note_id}", response_model=SteeringNoteOut)
def update_steering_note(note_id, payload: SteeringNoteUpdate, db):
row = db.get(SteeringNote, note_id)
if row is None:
raise HTTPException(status_code=404, detail="steering note not found")
row.note = payload.note.strip()
db.commit(); db.refresh(row)
return SteeringNoteOut(id=row.id, note=row.note, created_at=row.created_at)
```
The router already carries `dependencies=[Depends(require_admin)]` (phase 16), so anonymous callers get 403 automatically — no extra guard. Keep the existing `list`/`create`/`delete` routes unchanged.
3. Update the module docstring to name the new `PUT` route and that it reuses the router-level admin dependency (no new auth surface).
## ASSUMPTIONS
- The update reuses the same 1–2000-char, trimmed contract as create (no separate validation policy).
- `created_at` is preserved on update (editing a note doesn't redate it; the list order by newest-first is stable for edits).
- The endpoint is `PUT` (idempotent, full replacement of `note`), matching the "edit" semantics.
## Testing & Quality
- Unit (`tests/unit/test_steering.py`): `update_steering_note` via the router — 200 with updated `note`; 404 unknown id; 422 empty/whitespace/over-2000; and the router-level 403 for anonymous (via `require_admin`).
- Integration: `PUT /api/steering/{id}` → 200 returns the new note; the updated note is then read back by `load_steering_notes` (oldest-first order preserved) and appears in `build_steering_section`; anonymous → 403.
## Completion Criteria
- [ ] `PUT /api/steering/{id} {note}` → 200 with the updated note; `GET /api/steering` reflects the new text and order.
- [ ] Unknown id → 404; invalid body → 422; anonymous → 403.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,52 +0,0 @@
# Task 02 — Tuning page HTML + CSS
**Phase:** `27_global_tuning` · **Source:** `TODO.md:3 — "Add a way to add 'global tuning' without having a chat to reply to. Also previous tunes should be editable."`
**Story:** `.agent/user_stories/global-tuning.md`
## Objective
Create the `/tuning.html` page: a title, a "Add a global tuning note" form, and a list of existing notes each with an inline edit control and a delete control. Follow the Phase-08 tokens, WCAG 2.1 AA, and the standard app frame (landmarks, skip-link, sticky header reuse via `header.js`).
## Work
1. `frontend/tuning.html` (new) — standard app frame:
- `<header class="app-header">` reusing the same markup as `index.html`'s header (brand + nav + New Chat + Sign in/out) so `header.js` wires it identically. The page gains an extra admin-only link:
```html
<a href="/tuning.html" class="nav-link is-active" id="nav-tuning" aria-current="page">Tuning</a>
```
(placed after the Sources link; it is always present but only visible to admins because the whole nav is admin-scoped on non-chat pages — the Sources link is already admin-only, so this is consistent).
- `<main id="main" class="app-main" tabindex="-1">` with a centered `container` column:
- `<h1>Global Tuning</h1>` + a sub-heading explaining every note steers all future answers.
- A create form `#tune-form`: a visually-hidden `<label for="tune-note">` + `<textarea id="tune-note" maxlength="2000" rows="3" placeholder="e.g. be more concise — or: assume I'm on NixOS">` + a submit button `#tune-save` ("Add note").
- A status/announcer `<p class="visually-hidden" id="tune-announcer" role="status" aria-live="polite"></p>`.
- A list `<ul id="tune-list">` (role=list) for existing notes; each `<li class="tuning-note">` holds:
- a `.tuning-note-text` span (textContent, XSS-safe),
- an `#edit` button (`.tuning-edit`, icon + "Edit"),
- a `.tuning-delete` button (icon + "Delete").
- An empty-state `<p id="tune-empty">No tuning notes yet — add one above.</p>` toggled by `tuning.js`.
- `<footer class="app-footer">` as on other pages.
- Load `assets/markdown.js` (classic) and `assets/tuning.js` (module) + the shared header (module).
2. `frontend/assets/styles.css` — add a `--tuning-*` block mirroring the `.steering-*` / `.tune-*` tokens:
- `.tuning-note` — flex row, align-items center, gap; text flexes, actions shrink-0; border-bottom divider.
- `.tuning-note-text` — `--ink`; ellipsis overflow if very long.
- `.tuning-edit`, `.tuning-delete` — icon + label buttons, ≥44px targets, `:focus-visible` ring; edit uses brand colour, delete uses the error colour (`--err-ink`/`--err-line`) consistent with `.steering-delete`.
- `.tuning-note.is-editing .tuning-note-text` — hidden while editing (replaced by the inline form).
- Inline edit form (`.tuning-edit-form`) — a `<textarea class="tuning-edit-input" maxlength="2000">` pre-filled + Save/Cancel, styled like the existing `.tune-form` / `.tune-save` / `.tune-cancel`.
- `#tune-form` + `#tune-note` — styled like the existing composer/`#message-input`; `#tune-save` styled like `.tune-save`.
- `#tune-empty` — `--ink-soft`, centered, italic.
- Ensure the centered column matches the chat/sources width discipline (≥80–90% of container; not a skinny list).
- No `filter: blur`, no CDN, system font stack.
3. Verify no selector collision (grep `.tuning-note`, `#tune-form`, `.tuning-edit-*`).
## ASSUMPTIONS
- The page title is "Global Tuning"; the nav link label is "Tuning" (consistent with the chat header's "Tuning" panel).
- Editing is inline (swap the text for a textarea + Save/Cancel in the same list row) — the owner's "previous tunes should be editable" is satisfied without a separate editor page.
- The list is newest-first (same as the header panel) for consistency.
## Testing & Quality
- No unit/integration test for static CSS/HTML.
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
## Completion Criteria
- [ ] `/tuning.html` renders the title, the create form, the note list, and the empty state.
- [ ] All controls are labeled, ≥44px, `:focus-visible`, contrast ≥4.5:1; landmarks + skip-link present.
- [ ] No selector collision; no CDN tags; the page loads `tuning.js` + `markdown.js`.
- [ ] The admin-only "Tuning" nav link is present in the header.
@@ -1,38 +0,0 @@
# Task 03 — Tuning page CRUD logic
**Phase:** `27_global_tuning` · **Source:** `TODO.md:3 — "Add a way to add 'global tuning' without having a chat to reply to. Also previous tunes should be editable."`
**Story:** `.agent/user_stories/global-tuning.md`
## Objective
Create `frontend/assets/tuning.js` — the single owner of the tuning page's behaviour: load notes, create, edit (inline), cancel, and delete, all announced through a polite live region.
## Work
1. `frontend/assets/tuning.js` (new, module) — mirrors the structure of `app.js`'s steering section but for the standalone page:
- Cache the elements: `#tune-form`, `#tune-note`, `#tune-save`, `#tune-list`, `#tune-empty`, `#tune-announcer`.
- `announce(msg)` — sets `#tune-announcer.textContent` (role=status, aria-live=polite).
- `loadNotes()` — `GET /api/steering`; on `r.ok` render the list, else keep the last rendered list (progressive enhancement). Populates `#tune-list` (role=list) with `<li>` rows; toggles `#tune-empty` (`hidden = notes.length > 0`); each row:
- `.tuning-note-text` span (`textContent`, XSS-safe),
- `.tuning-edit` button (icon + "Edit"),
- `.tuning-delete` button (icon + "Delete", aria-label "Delete tuning note: <note>").
- Create handler (`#tune-form` submit): `POST /api/steering {note}`; 201 → clear the textarea, announce "Tuning note added. Future answers will follow it.", reload; non-2xx → inline error under the form (kept, form not cleared) with the API detail; network error → friendly message. Disable `#tune-save` during the request.
- Edit flow (`tuning-edit` click): swap the row's `.tuning-note-text` span for an inline `.tuning-edit-form` containing a `<textarea class="tuning-edit-input" maxlength="2000">` pre-filled with the note + Save/Cancel; focus the textarea. Track the note id on the form (data attribute).
- Save-edit handler: `PUT /api/steering/{id} {note}`; 200 → replace the form with a `.tuning-saved` status (role=status) + announce "Tuning note updated."; non-2xx → keep the form, show inline error; cancel → revert to the text span.
- Delete handler (`tuning-delete` click): disable the button; `DELETE /api/steering/{id}`; 204/404 → remove the `<li>` from the DOM immediately (optimistic), announce, and if 404 reload; non-2xx → re-enable the button + announce retry.
- Keep the 1–2000-char contract on the client (maxlength on the textareas); the server re-validates.
2. Wire the shared header + whoami gate: `tuning.js` imports `{ initSharedHeader, fetchIsAdmin }` from `./header.js` and at boot awaits `initSharedHeader()` then `loadNotes()` only if `fetchIsAdmin()` is true (anonymous users see the page frame but the list stays empty / the create form 403s gracefully — consistent with the Sources page gate pattern). Actually simpler: the header already hides the "Tuning" nav link for anonymous (it's a nav link like Sources); but a direct anonymous URL should still be safe — `loadNotes()` swallows non-2xx, and the create form 403s. So the page is anonymous-safe without a hard gate.
## ASSUMPTIONS
- The edit is inline in the same list row (no separate editor page) — the owner's "editable" is satisfied.
- Optimistic delete (remove the `<li>` before the server confirms) matches the chat page's `deleteSteeringNote` UX.
- The page is anonymous-safe: the list won't render for anonymous (the create/delete calls 403), and the nav link is hidden for anonymous.
## Testing & Quality
- No unit/integration test (frontend-only).
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
## Completion Criteria
- [ ] Create a note on `/tuning.html` → it appears in the list; the announcer announces the change.
- [ ] Edit a note inline → the saved text is updated in the list; the `<tuning>` prompt reflects it (verified via the integration test on the endpoint).
- [ ] Delete a note → the row is removed and the list updates.
- [ ] XSS-safe: note text rendered via `textContent`, never `innerHTML`.
- [ ] Empty state shows when there are no notes.
@@ -1,34 +0,0 @@
# Task 04 — Header "Tuning" button + E2E suite
**Phase:** `27_global_tuning` · **Source:** `TODO.md:3 — "Add a way to add 'global tuning' without having a chat to reply to. Also previous tunes should be editable."`
**Story:** `.agent/user_stories/global-tuning.md`
## Objective
Expose the tuning page via an admin-only header link and add the E2E story suite.
## Work
1. `frontend/tuning.html` header — add the admin-only "Tuning" nav link (is-active) in the `<nav class="app-nav">`, alongside the existing Sources link (also admin-only). The header markup mirrors `index.html`/`sources.html` (brand + nav + New Chat + Sign in/out) so `header.js` wires it identically. The "Tuning" link ships visible on the tuning page (it's the current page); on other pages the admin-only nav links are revealed by `header.js` whoami, consistent with Sources.
2. No change to `app.js`/`sources.html` nav needed — the tuning page is reached from its own header link. (The chat-page "Tune" button and header panel from phase 15 remain unchanged.)
3. `tests/e2e/test_global_tuning.py` (new — the story gate). Reuse the seeding harness pattern from `test_steering.py` / `test_document_viewer.py` (`_import_fixtures` / `_reset_db` / `_run_in_thread`) so the endpoint-under-test is exercised against a seeded KB. Test → mapping (Playwright Mapping Rule):
1. `test_create_note_without_chat` — log in, go to `/tuning.html`, type a note, submit; assert it appears in `#tune-list` and the announcer announced it. No chat turn was made.
2. `test_edit_note_inline` — create a note, click "Edit", change the text, Save; assert the list shows the new text and the edit form is gone.
3. `test_delete_note` — create a note, delete it; the row is removed and `#tune-empty` shows again.
4. `test_edit_note_steers_answer` — create a note via `/tuning.html`, then go to the chat, ask the QUESTION, and assert the note's marker leaks into the answer (same echo trick `test_steering.py` uses: `(tuning: <first note line>)`), proving the edited note is read into the system prompt.
5. `test_tuning_page_a11y_and_no_cdn` — landmarks, skip-link, labeled controls, ≥44px targets, `:focus-visible`; every `script[src]`/`link[href]` is same-origin or `data:`; dark theme.
6. `test_anonymous_cannot_manage` — anonymous `/tuning.html`: the list is empty, and a scripted `PUT /api/steering/{id}` returns 403 (or the create form 403s).
4. Regressions to run green in isolation: `test_steering.py` (the chat-page Tune button + panel still work — create + delete), `test_header_consistency.py` (the tuning page's header is consistent; the new nav link doesn't break the height/consistency assertions on the other pages), `test_document_viewer.py` (unrelated — modal still works), `test_smoke.py`.
5. `.agent/user_stories/global-tuning.md` — write the story file.
## ASSUMPTIONS
- The tuning page's "Tuning" nav link is admin-only (consistent with the Sources link — the catalog and tuning are admin-only).
- The chat-page "Tune" button + header panel are **not** removed (nothing regresses); the tuning page is the primary global manager, the button is a quick-add affordance.
- The edit-steers-answer test reuses the mock-LLM echo behaviour already established in `test_steering.py`.
## Testing & Quality
- E2E: `tests/e2e/test_global_tuning.py` — the story gate, green **in isolation** (prereq `podman compose up -d db`).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_global_tuning.py -v --no-cov` green in isolation.
- [ ] `test_steering.py`, `test_header_consistency.py`, `test_document_viewer.py`, `test_smoke.py` green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,37 +0,0 @@
# Task 01 — Settings: `BOR_GIT_SOURCES` + `BOR_SOURCES_DIR`
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "We shouldn't be hard-coding Homelab and Deployments. Instead, a list of git links should be specified."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Add the two new settings that drive the git-based source flow: the comma-separated list of git URLs and the dedicated local clone location.
## Work
1. `app/config.py` — in the "Import scope" section (near `import_extensions`) add:
- `git_sources: str = ""` — comma-separated git repository URLs (no default; when empty, `import_docs` falls back to `--source` / the old `DEFAULT_SOURCES`). Document: "list of git repo URLs to clone/pull into `sources_dir` before indexing (phase 28); empty means no git sources".
- `sources_dir: str = "~/bor-sources"` — the dedicated local directory the repos are cloned/pulled into (phase 28). Document: "where `import_docs` clones/pulls `git_sources` repos (expanded ~ via `Path.expanduser`)".
- Add a property `git_source_list` returning the non-empty, stripped URLs (list[str]) — used by the import script.
2. `tests/unit/test_config.py` — add tests:
- `git_sources` default is `""`; `git_source_list` returns `[]` when empty.
- `git_sources` env override parses a comma-separated list (whitespace trimmed, empties dropped).
- `sources_dir` default is `~/bor-sources` (raw string, not expanded in the setting — expansion happens in the script).
3. `.env.example` — document both under an "Import scope / git sources" comment block:
```
# --- Import sources (git; phase 28) ---
# BOR_GIT_SOURCES=https://github.com/user/homelab.git,https://github.com/user/deployments.git
# BOR_SOURCES_DIR=~/bor-sources
```
## ASSUMPTIONS
- `git_sources` is empty by default (backwards-compatible: the `import_docs` script keeps its `--source` default until the owner sets `BOR_GIT_SOURCES`).
- URLs are stored raw (no parsing of scheme/auth in the setting) — parsing happens in `git_sync.py`.
- `sources_dir` is stored as a raw string; `Path.expanduser()` is applied in the script (so the setting stays env-agnostic and testable).
## Testing & Quality
- Unit: `tests/unit/test_config.py` additions.
- Coverage: the new setting is exercised by the unit test; the >90% `app/` gate is maintained.
## Completion Criteria
- [ ] `Settings().git_source_list` returns `[]` by default and the parsed list when `BOR_GIT_SOURCES` is set.
- [ ] `sources_dir` defaults to `~/bor-sources`.
- [ ] `.env.example` documents both vars.
@@ -1,67 +0,0 @@
# Task 02 — Git clone/pull utility
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Create `scripts/git_sync.py` with a single `clone_or_pull(url, dest)` function that clones a repo (shallow, first run) or fast-forwards it (subsequent runs), returning the destination path. This is the only place `git` is invoked.
## Work
1. `scripts/git_sync.py` (new) — stdlib `subprocess` only (A11: no new packages):
```python
"""Git source sync for import_docs (phase 28).
clone_or_pull(url, dest) clones ``url`` into ``dest`` (shallow, depth 1)
the first time, or fast-forwards an existing checkout with ``git pull
--ff-only`` on subsequent runs. Auth is whatever the URL/SSH config
supplies — no credentials are stored here.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
class GitSyncError(RuntimeError): ...
def clone_or_pull(url: str, dest: Path | str) -> Path:
dest = Path(dest)
if not dest.exists() or not (dest / ".git").exists():
dest.parent.mkdir(parents=True, exist_ok=True)
_run(["git", "clone", "--depth", "1", url, str(dest)], cwd=dest.parent)
else:
_run(["git", "pull", "--ff-only"], cwd=dest)
return dest
def _run(argv, cwd):
try:
proc = subprocess.run(argv, cwd=cwd, capture_output=True, text=True)
except FileNotFoundError:
raise GitSyncError("git was not found on PATH — install git and retry")
if proc.returncode != 0:
raise GitSyncError(f"git {' '.join(argv[1:])} failed (exit {proc.returncode}): {proc.stderr.strip()}")
return proc.stdout
```
- `--depth 1` clone (fast, and the KB is re-imported incrementally anyway).
- `--ff-only` pull (refuses to merge unrelated histories — a broken checkout fails loudly rather than producing a dirty index).
- `GitSyncError` carries the `git` stderr so the caller can name the failing repo + reason.
- Auth: nothing special — a `https://…` URL uses the OS credential helper / prompts; an `git@host:repo.git` URL uses the machine's SSH key. Document this in the module docstring.
2. `tests/unit/test_git_sync.py` (new):
- `clone_or_pull` **clones** when the dest has no `.git` — verify by monkeypatching `subprocess.run` to a fake that records the argv and returns `returncode=0`; assert `["git","clone","--depth","1",url,str(dest)]` was called and `dest` returned.
- `clone_or_pull` **pulls** when `.git` exists — monkeypatch; assert `["git","pull","--ff-only"]` called.
- `clone_or_pull` raises `GitSyncError` on non-zero exit, carrying the stderr text.
- `clone_or_pull` raises `GitSyncError("git was not found…")` when `subprocess.run` raises `FileNotFoundError`.
- `dest.parent` is created before clone (assert the fake saw a parent that `mkdir` would create — or test the `mkdir` call path directly).
## ASSUMPTIONS
- `--depth 1` shallow clone is sufficient (the KB is re-imported incrementally; no need for full history).
- `--ff-only` pull is the right policy (refuse merges — a dirty/broken checkout fails loudly).
- Auth is delegated to the machine (SSH key / credential helper); no secrets are stored in code or `.env`.
- `git` CLI is assumed present (standard on homelab machines; a clear error is raised otherwise).
## Testing & Quality
- Unit: `tests/unit/test_git_sync.py` (clone vs pull dispatch, error propagation, missing-git error).
- Coverage: the new `scripts/git_sync.py` is fully covered.
## Completion Criteria
- [ ] `clone_or_pull` clones a fresh repo and pulls an existing one (verified via monkeypatched `subprocess`).
- [ ] A failing `git` call raises `GitSyncError` with the stderr; a missing `git` raises `GitSyncError` naming git.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,49 +0,0 @@
# Task 03 — `import_docs` resolves git sources → local dirs
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Rewire `scripts/import_docs.py` so that, when `BOR_GIT_SOURCES` is set, it clones/pulls each repo into `BOR_SOURCES_DIR/<name>/` and indexes the resulting directories — while keeping `--source <path>` overriding for manual local directories.
## Work
1. `scripts/import_docs.py`:
- Import `clone_or_pull` from the sibling module: `from scripts.git_sync import clone_or_pull` (or, since `main()` runs as `python -m scripts.import_docs`, a top-level `from git_sync import clone_or_pull` works — `scripts/` is on `sys.path`).
- Keep `DEFAULT_SOURCES` as the fallback for the no-`--source` + no-`BOR_GIT_SOURCES` case (backwards-compatible).
- New resolution logic in `main()`:
```python
settings = get_settings()
sources = _resolve_sources(args.source, settings)
```
where `_resolve_sources`:
- If `args.source` is given → expanduser each and return (unchanged manual behaviour; `--source` wins).
- Else if `settings.git_source_list` is non-empty → for each URL, `clone_or_pull(url, Path(settings.sources_dir).expanduser() / repo_name(url))`; collect the dest dirs; return them. Any `GitSyncError` propagates (the script exits non-zero naming the failing repo — see below).
- Else → return the old `DEFAULT_SOURCES`.
- `repo_name(url)` — derive a directory name from the URL: strip a trailing `.git`, take the basename after the last `/` (or `:` for scp-style `git@host:repo.git`). Fall back to a slug of the URL if no basename.
- Before importing, log which dirs are being imported (so the operator sees the cloned paths). The existing "source dir not found" warning still applies if a clone left an empty dir.
2. Keep the existing `--prune` / `--limit` flags and the summary print unchanged.
3. Exit code: if any `GitSyncError` is raised, let it propagate to a top-level `except` that prints `import_docs: git sync failed: <reason>` to stderr and returns 1 **before** importing anything (so a bad repo doesn't silently import partial junk). Structure:
```python
try:
sources = _resolve_sources(...)
except GitSyncError as e:
print(f"import_docs: git sync failed: {e}", file=sys.stderr)
return 1
```
## ASSUMPTIONS
- `--source` always wins over `BOR_GIT_SOURCES` (explicit CLI flag beats env).
- When `BOR_GIT_SOURCES` is set, `--source` is ignored (only one source mode at a time) — document this.
- `BOR_SOURCES_DIR` defaults to `~/bor-sources`; each repo is a subdirectory named after the repo.
- A repo that fails to clone/pull aborts the whole run (no partial import) — the operator fixes the URL and re-runs; the already-cloned repos are left on disk and will be pulled on the next run.
- The `documents.source` column will be the repo directory name (e.g. `homelab`), matching the current `source=root.name` behaviour in the importer.
## Testing & Quality
- Integration: `tests/integration/test_import_docs_git.py` (new) — monkeypatch `clone_or_pull` to a fake that creates a temp dir with a fixture `.md` file and returns it; assert `import_docs.main(["--source"])`-equivalent resolution picks the cloned dir and that `import_sources` is called with it. Also assert a `GitSyncError` from the fake → exit code 1 and no import attempt.
- Coverage: the new `main()` resolution branch is covered.
## Completion Criteria
- [ ] `BOR_GIT_SOURCES` set → `import_docs` clones/pulls each repo into `BOR_SOURCES_DIR/<name>/` and imports them.
- [ ] `--source <path>` still imports that manual directory (unchanged).
- [ ] A failing git sync → non-zero exit, message naming the repo, no partial import.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,37 +0,0 @@
# Task 04 — Integration test + docs
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Add the integration test that exercises the full `import_docs` git-resolution path with a mocked `clone_or_pull`, and finalise the docs.
## Work
1. `tests/integration/test_import_docs_git.py` (new) — drive `scripts.import_docs.main(argv)` end to end with the git flow mocked:
- Monkeypatch `scripts.import_docs.clone_or_pull` (import the name into the module's namespace after importing it) so it **creates** a temp directory containing a fixture `.md` file and returns that path (simulating a clone/pull landing real content).
- Set `settings.sources_dir` to a `tmp_path`-based dir (via `monkeypatch.setattr` on the settings or by patching `get_settings`).
- Call `main([])` with `BOR_GIT_SOURCES` patched to a single URL.
- Assert: `clone_or_pull` was called once with the URL; the returned dir was passed to `import_sources`; the summary printed includes the fixture file (added > 0).
- Negative case: monkeypatch `clone_or_pull` to raise `GitSyncError`; assert `main([])` returns `1` and prints a message naming the repo, and that `import_sources` was **not** called.
- Manual override case: `main(["--source", str(tmp_path)])` imports the manual dir and does **not** call `clone_or_pull`.
- Use the same `FakeEmbedder`-style approach the importer tests use so no real LLM is needed (the importer's two-phase embed is duck-typed; pass a fake `llm` if `main` allows injection, or let `import_docs` build the real `LLMClient` but patch `import_sources` to capture its `sources` arg and short-circuit). Simpler: monkeypatch `scripts.import_docs.import_sources` to a fake that records the `sources` list and returns a trivial `ImportSummary` — this isolates the git-resolution logic from the whole embed pipeline.
2. `README.md`:
- Section 5 (Import your knowledge base): document the two source modes — (a) git sources via `BOR_GIT_SOURCES` (clone/pull into `BOR_SOURCES_DIR`), (b) `--source <path>` for manual directories. Keep the `~/Homelab + ~/Deployments` note as the *previous* default, now replaced by `BOR_GIT_SOURCES`.
- Add a short "Git-based sources" subsection: set `BOR_GIT_SOURCES` (comma-separated URLs) + `BOR_SOURCES_DIR`; `import_docs` clones (first run) or pulls (subsequent runs) each repo and indexes them; `--source` overrides; a failed sync aborts the run.
- The "Clicking a chip opens that document in a new tab" bullet (line ~69) is now stale — update to "opens in an almost-fullscreen modal" (phase 26).
3. `.env.example` — already updated in task 01; double-check the comment block is present and correct.
4. `.agent/user_stories/git-sources.md` — write the story file.
## ASSUMPTIONS
- The integration test mocks `clone_or_pull` + `import_sources` so it runs without network, without `git`, and without a real embed endpoint — it isolates the resolution logic.
- The README keeps a migration note for owners currently relying on the hardcoded `~/Homelab`/`~/Deployments` default (set `BOR_GIT_SOURCES` to the same two repos).
## Testing & Quality
- Integration: `tests/integration/test_import_docs_git.py`.
- Coverage: the new `main()` branch covered.
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_import_docs_git.py -v --no-cov` green.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] README documents both source modes; the "opens in a new tab" bullet updated to "modal".
@@ -1,40 +0,0 @@
# Phase 29 — Global Tuning nav link
**Source:** `.agent/phases/complete/27_global_tuning` — the `/tuning.html` page and its create/edit/delete machinery shipped, but the "Tuning" header link that was supposed to *expose* the page (`27` task 04: *"Expose the tuning page via an admin-only header link"`) only exists on `tuning.html` itself. It is not present on the Chat or Sources pages, so there is no way to reach `/tuning.html` from anywhere except typing the URL.
**Story:** `.agent/user_stories/global-tuning.md`
**Context:** The shared header (phase 19) renders `[Chat, Sources — admin only]` in `<nav class="app-nav">` on the Chat and Sources pages. `frontend/assets/header.js` already contains logic to reveal an admin-only `#nav-tuning` link for admins (lines 69–70) — but that element is never in `index.html`/`sources.html` markup, so the reveal is a no-op. This phase adds the missing markup so the link the JS already handles actually exists.
## Objective
Make the Global Tuning page reachable by adding the admin-only **"Tuning"** nav link to the shared headers of the **Chat** (`index.html`) and **Sources** (`sources.html`) pages — hidden by default, revealed for admins by the existing `header.js` whoami gate (the exact same pattern as the Sources link). No backend, schema, or API change.
## Dependencies
- `19_shared_header` (complete) — the `<nav class="app-nav">` contract, the shared header markup/ids, and the 64px header-height rule on the Chat and Sources pages.
- `16_admin_auth` (complete) — the `/api/whoami` gate + `header.js` `fetchIsAdmin()` that reveals admin-only nav links.
- `27_global_tuning` (complete) — the `/tuning.html` page and `PUT/POST/DELETE /api/steering` endpoints that this link points to.
## Tasks
1. `01_tuning_links_markup.md` — add the admin-only `#nav-tuning` link to the Chat (`index.html`) and Sources (`sources.html`) headers, hidden by default.
2. `02_tuning_nav_e2e_and_regression.md` — add the story E2E suite (`test_tuning_nav_link.py`) + verify no regressions in the header/shared-header/no-CDN suites and commit.
## Testing & Quality
- Unit/integration: no new backend logic (frontend-only markup). The existing no-CDN integration test (`tests/integration/test_api.py::test_index_html_served_locally`) must still pass — the new link is same-origin `<a>`, no new tags.
- Coverage: frontend-only; the `app/` >90% gate is unaffected (unchanged).
- E2E (mandatory, A16): `tests/e2e/test_tuning_nav_link.py` — the story gate, run in isolation.
## Completion Criteria
- [ ] On the **Chat** page (`/`), a signed-in admin sees a **"Tuning"** link (next to Chat/Sources) that navigates to `/tuning.html`; an anonymous visitor does **not** see it (ships `hidden`, `header.js` reveals only for admin).
- [ ] Same on the **Sources** page (`/sources.html`).
- [ ] The link is absent from the anonymous DOM-reveal (verified via `test_anonymous` behavior); header markup stays valid and semantic.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged (no backend change).
- [ ] Existing header suites stay green in isolation: `test_header_consistency.py` (64px height + nav structure unchanged), `test_shared_header.py`, `test_smoke.py`, `test_global_tuning.py`.
- [ ] `uv run pytest tests/e2e/test_tuning_nav_link.py -v --no-cov` green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean (no Python change, but run the gate).
- [ ] UI Structure Check (AGENTS.md rule 5): the link is a labeled, focus-visible `<a>`; header landmarks/contrast unchanged; no CDN.
- [ ] One `--no-gpg-sign` commit staging only this phase's files; `.agent/phases/todo/29_tuning_nav_link/` moved to `.agent/phases/complete/`.
## Locked decisions
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages; the link is a same-origin `<a href="/tuning.html">`.
- **A10 untouched** — no new endpoint or auth surface; the link reuses the existing `#nav-sources` reveal path (`header.js` already gates `#nav-tuning` on `fetchIsAdmin()`).
- **No schema / migration** — purely a markup addition.
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
- **Scope boundary** — only the Chat and Sources pages (the two that share the standard `app-nav` with the Sources link). The viewer page (`document.html`) uses a different `.doc-header-actions` header variant and the login page is the auth gate; both stay out of scope for this fix.
@@ -1,41 +0,0 @@
# Task 01 — Tuning nav link on the Chat and Sources pages
**Phase:** `29_tuning_nav_link` · **Story:** `.agent/user_stories/global-tuning.md`
## Objective
Add the admin-only **"Tuning"** nav link to the shared headers of the **Chat** (`index.html`) and **Sources** (`sources.html`) pages, so `/tuning.html` is reachable from both.
## Work
Add one link to each page, inside the existing `<nav class="app-nav" aria-label="Primary">`, **immediately after** the Sources link, mirroring the Sources pattern exactly.
1. `frontend/index.html` (Chat page) — after the `#nav-sources` link, add:
```html
<!-- Phase 29: the Global Tuning link is admin-only (owner permission
2026-08-25) — hidden by default, header.js reveals it once
whoami says admin, exactly like the Sources link above. Points at
the standalone /tuning.html manager (phase 27). -->
<a href="/tuning.html" class="nav-link" id="nav-tuning" hidden>Tuning</a>
```
2. `frontend/sources.html` (Sources page) — same link, placed after its `#nav-sources` link:
```html
<!-- Phase 29: the Global Tuning link is admin-only (owner permission
2026-08-25) — hidden by default, header.js reveals it once
whoami says admin, exactly like the Sources link above. -->
<a href="/tuning.html" class="nav-link" id="nav-tuning" hidden>Tuning</a>
```
Rules for both edits:
- Keep `hidden` by default (anonymous-safe ship-hidden, identical to `#nav-sources`).
- Do **not** add `is-active` / `aria-current="page"` — those are "current page" visual states; this link is always hidden on the Chat/Sources pages, and `tuning.html` already sets `is-active` on its own link.
- Preserve surrounding markup and indentation so the header layout and the pinned 64px height (phases 12/19) are unchanged. The link is inline in the nav, so height is unaffected.
- **Do not modify `frontend/assets/header.js`.** It already handles this element (lines 69–70: `const navTuning = document.querySelector("#nav-tuning"); if (navTuning) navTuning.hidden = !admin;`). The link now exists for that existing reveal code to act on. If, against expectation, that reveal code were absent, add it next to the `navSources` reveal — but it is present, so leave `header.js` untouched.
## Testing & Quality
- No new unit/integration logic (frontend markup only).
- Coverage: frontend-only; `app/` coverage unaffected.
- This is a same-origin `<a>` — the existing no-CDN integration test (`tests/integration/test_api.py::test_index_html_served_locally`) still passes.
## Completion Criteria
- [ ] `index.html` and `sources.html` each contain an admin-only `#nav-tuning` `<a href="/tuning.html">` placed after `#nav-sources`, `hidden` by default, without `is-active`.
- [ ] Header structure/height unchanged (confirmed in task 02 via `test_header_consistency.py` / `test_shared_header.py`).
- [ ] Full test suite green; no behavior change in completed phases.
@@ -1,53 +0,0 @@
# Task 02 — Tuning nav-link E2E suite, regression checks, and commit
**Phase:** `29_tuning_nav_link` · **Story:** `.agent/user_stories/global-tuning.md`
## Objective
Prove the new admin-only "Tuning" link works (and is hidden for anonymous users) on the Chat and Sources pages, confirm no header/no-CDN regressions, and commit.
## Work
1. **New story E2E suite** — create `tests/e2e/test_tuning_nav_link.py`, following the harness already used in `tests/e2e/test_global_tuning.py`:
- Reuse `from e2e.auth_helpers import login` (real form login on `/login.html`; `login(page, app_url, next=...)` redirects to `next`) and the `db_ready` fixture. No KB seeding is needed for the link-visibility tests (the link is pure header markup gated by `/api/whoami`), so a light module fixture that just brings up the app + mock LLM is enough — mirror `test_global_tuning.py`'s `mock_llm`/`app_url`/`db_ready` params and its `_reset_db` if you want a clean DB.
- The visible `#nav-tuning` on an admin page is the sync point (exactly the Sources-link contract): `header.js` reveals it only after whoami says admin.
- Write these tests (Playwright `expect`, sync API):
1. `test_admin_sees_tuning_link_on_chat` — `login(page, app_url, next="/")`; on the Chat page assert `#nav-tuning` is visible and its `href` is `/tuning.html`; click it and assert the URL navigates to `/tuning.html` (and `#nav-tuning` becomes `is-active`, matching the page's own link).
2. `test_admin_sees_tuning_link_on_sources` — `login(page, app_url, next="/sources.html")`; assert `#nav-tuning` is visible on the Sources page and clicking it navigates to `/tuning.html`.
3. `test_anonymous_hides_tuning_link_on_chat_and_sources` — visit `/` and `/sources.html` **without** logging in; assert `#nav-tuning` is `hidden` on both pages, `#sign-in-link` visible, `#nav-sources` also hidden (consistent with the existing Sources gate).
4. `test_tuning_link_a11y_and_no_cdn` — on the admin Chat page, assert the link is a labeled, focus-visible `<a>` (tab through the header, focus lands on it), and assert every `script[src]`/`link[href]` on the page is same-origin or `data:` (no CDN) — reuse the same `evaluate` scan `test_global_tuning.py::test_tuning_page_a11y_and_no_cdn` uses.
2. **Regression verification** — after the new suite is green, run the header- and page-sensitive suites **in isolation** to prove no regression from the added markup/height:
```bash
uv run pytest tests/e2e/test_header_consistency.py -v --no-cov
uv run pytest tests/e2e/test_shared_header.py -v --no-cov
uv run pytest tests/e2e/test_smoke.py -v --no-cov
uv run pytest tests/e2e/test_global_tuning.py -v --no-cov
uv run pytest tests/integration/test_api.py::test_index_html_served_locally -v --no-cov
```
Fix anything the new link breaks (it should not — the link is inline in the existing nav and does not change header height).
3. **Full gate** — run the whole suite with coverage and confirm `app/` coverage TOTAL is **unchanged** (this phase adds no backend logic):
```bash
uv run pytest
uv run pytest --cov=app --cov-report=term-missing # TOTAL must not drop
```
4. **Lint / types** — `uv run ruff check . && uv run pyright` (no Python change expected, but run the gate).
5. **Commit** — one atomic `--no-gpg-sign` commit staging only this phase's files (the new E2E file + any committed edits), then move the phase directory to `.agent/phases/complete/`:
```bash
git add -f .agent/phases/todo/29_tuning_nav_link tests/e2e/test_tuning_nav_link.py
git commit --no-gpg-sign -m "feat(ui): expose Global Tuning from Chat + Sources headers (admin-only Tuning nav link)"
git mv .agent/phases/todo/29_tuning_nav_link .agent/phases/complete/29_tuning_nav_link
```
(`.agent/` is gitignored — use `git add -f` / `git mv -f` as needed.)
## Testing & Quality
- E2E (mandatory, A16): `tests/e2e/test_tuning_nav_link.py` green **in isolation**:
```bash
uv run pytest tests/e2e/test_tuning_nav_link.py -v --no-cov
```
- No new unit/integration logic (frontend markup only); `app/` coverage unchanged.
## Completion Criteria
- [ ] Admin sees a visible **"Tuning"** link on the Chat and Sources pages → clicking it opens `/tuning.html`.
- [ ] Anonymous visitors see **no** Tuning link on either page (ships `hidden`, revealed only for admin).
- [ ] Link is labeled, focus-visible; page stays CDN-free.
- [ ] `test_tuning_nav_link.py` green in isolation; `test_header_consistency.py`, `test_shared_header.py`, `test_smoke.py`, `test_global_tuning.py` green in isolation; `test_index_html_served_locally` green.
- [ ] `uv run pytest` green; `app/` coverage TOTAL unchanged; `ruff` + `pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase moved to `.agent/phases/complete/`.
@@ -1,48 +0,0 @@
# Phase 30 — Document Summaries (lite-model summaries for non-markdown documents)
**Source:** `TODO.md L3 — "One issue I'm having is bad context for the embedder which causes poor retrieval results… we need a small model to analyze non markdown documents and provide a textual summary of those documents with a pointer back to the source… if retrieval == summary, fetch documents referenced by summary… The small model available on aipi.reeseapps.com is 'lite'."`
**Story:** `.agent/user_stories/document-summaries.md`
**Context:** The importer (`app/rag/importer.py::_index_file` — chunk → embed → upsert per file, A9 scope), hybrid retrieval (`app/rag/retriever.py` — A7: cosine ∪ FTS, RRF, chunk→parent-document mapping, full-document context never truncated, phase 24), the locked persona prompts (`app/rag/prompts.py`), the aipi client (`app/rag/llm.py` — A5: `turbo` chat streaming + `embed` embeddings), and the E2E mock LLM (`tests/e2e/mock_llm.py` — deterministic, keys on system-prompt markers like `DEFLECT_MODE` / `<tuning>`).
## Objective
Give every **non-markdown** A9 document (txt, yaml, yml, json, py) a natural-language summary generated at import time by the aipi **`lite`** model. The summary is stored on the document (`documents.summary`) **and indexed as one extra embedded chunk** (`chunks.is_summary`), so hybrid search has a well-embedding natural-language target to hit instead of the badly-formatted raw text. A summary hit resolves to its parent (the source document) — the existing chunk→document mapping then feeds the **full source document** to the LLM, implementing the TODO's "if retrieval == summary, fetch the documents referenced by the summary" step. The per-turn log line records how many summary hits landed in the selected context.
## Dependencies
- `29_tuning_nav_link` (complete) — the latest finished phase (sequencing only).
- Substantively builds on: `02_story_import_documents` / `24_whole_document_context` (import pipeline + full-document context contract), `09_story_retrieval_quality` (A7 hybrid retrieval the summary chunk flows through unchanged), `01_infrastructure` (models/alembic, LLM client, E2E mock).
## Tasks
1. `01_lite_model_client.md` — `BOR_LLM_SUMMARY_MODEL` (default `lite`) + non-streaming `LLMClient.chat()` for the lite model.
2. `02_migration_summary_columns.md` — Alembic 0004: `documents.summary TEXT NULL` + `chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE`.
3. `03_summarizer_module.md` — `app/rag/summarizer.py`: `SUMMARY_MODE` prompt (capped input), lite call, output validation + deterministic `Source: <source>/<path>` pointer line.
4. `04_importer_summary_integration.md` — importer generates/stores/indexes summaries for non-md files (best-effort fail-soft) + summary counters.
5. `05_pipeline_summary_resolution.md` — `is_summary` through the retriever, `summary_hits` in `TurnPlan` + the per-turn log line; full source document on summary hit (existing mapping, asserted).
6. `06_mock_and_e2e.md` — deterministic `lite` in `mock_llm.py`, sentinel fixture, `tests/e2e/test_document_summaries.py`, story file, README, commit.
## Testing & Quality
- Unit: summarizer (prompt/cap/pointer/errors), importer (summary happy path, md exclusion, fail-soft, replacement on re-import), retriever (`is_summary` through both candidate lists + `fuse`), chat gate (`TurnPlan.summary_hits`), LLM client (`chat()`).
- Integration: migration 0004 up/down.
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`, TOTAL ≥ pre-change number).
- E2E (mandatory, A16): `tests/e2e/test_document_summaries.py` — one story, run **in isolation** (`uv run pytest tests/e2e/test_document_summaries.py -v --no-cov`); proves summary hit → full source document reaches the answer (sentinel in the raw doc, absent from the mock summary).
- All existing E2E suites stay green (new columns are defaulted; all existing chunks have `is_summary=false`).
## Completion Criteria
- [ ] After `uv run python -m scripts.import_docs`, every non-markdown fixture/doc has `documents.summary` set and exactly one `is_summary` chunk (position −1, embedded); markdown docs have neither.
- [ ] A question whose best match is a summary chunk yields an answer grounded in the **full source document** (E2E sentinel) and the per-turn log line shows `summary_hits>=1`.
- [ ] A lite-model failure during import does **not** drop the document — it is indexed without a summary, logged, and counted (`summary_errors`).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
- [ ] `uv run pytest tests/e2e/test_document_summaries.py -v --no-cov` green in isolation; existing suites (`test_chat_rag.py`, `test_retrieval_quality.py`, `test_import_documents.py`, `test_whole_document_context.py`) stay green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/document-summaries.md` exists.
- [ ] `.env.example` + README document `BOR_LLM_SUMMARY_MODEL` / `BOR_SUMMARY_MAX_CHARS` and the summary behavior.
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc`); `.agent/phases/todo/30_document_summaries/` moved to `.agent/phases/complete/`.
## Locked decisions
- **A5 extended, not revised** — the `lite` model is served by the same OpenAI-compatible endpoint (`https://aipi.reeseapps.com/v1`) via a new `BOR_LLM_SUMMARY_MODEL` setting (default `lite`); no new model management, no new package.
- **A7 untouched** — hybrid retrieval logic is unchanged; the summary is an ordinary chunk, so it flows through the existing cosine ∪ FTS ∪ RRF path and the chunk→document mapping. The "fetch the referenced document" step is the existing full-document context contract (phase 24) — never truncated.
- **A9 untouched** — "non-markdown" means every *already-imported* A9 document except `md`/`markdown`. The TODO's quadlet-file example is **out of scope**: `.quadlet` is not an A9 format and `BOR_IMPORT_EXTENSIONS` may only narrow the locked set (flagged at roadmap confirmation; importing quadlet files would require an owner-permission A9 revision).
- **A13** — migration 0004 adds two columns (`documents.summary`, `chunks.is_summary`); no table rework, both reversible.
- **Summary generation is best-effort** — a lite failure logs + counts (`summary_errors`) and the file is still indexed without a summary (same fail-soft spirit as the per-file `EmbeddingError` handling, but weaker: the doc is already committed).
- **Pointer is code-deterministic** — the `Source: <source>/<path>` line is appended by `summarizer.py`, never trusted to the model.
- **A16 honoured** — one dedicated story E2E suite; E2E stays deterministic via the mock LLM's `SUMMARY_MODE` marker.
- **A17 honoured** — one atomic `--no-gpg-sign` commit.
@@ -1,26 +0,0 @@
# Task 01 — lite model setting + non-streaming chat()
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "The small model available on aipi.reeseapps.com is 'lite'." (enabler for "we need a small model to analyze non markdown documents")`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Make the aipi **`lite`** model callable from the app: a new model setting plus a non-streaming `LLMClient.chat()` one-shot completion method that the summarizer (task 03) and the phase-31 overview generator will use.
## Work
1. `app/config.py` — add `llm_summary_model: str = "lite"` (env `BOR_LLM_SUMMARY_MODEL`), documented like the other LLM settings.
2. `app/rag/llm.py` — add to `LLMClient`:
- `async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str` — `chat.completions.create(model=model or self.settings.llm_summary_model, messages=…, temperature=0.2, max_tokens=2048, stream=False)`; returns the first choice's `message.content` stripped.
- Raise `LLMError` (wrapped, with the base URL in the message — same style as `chat_stream`) on any transport/HTTP/malformed failure, and on an empty/missing content field (a silent empty summary must never be stored).
3. `.env.example` — document `BOR_LLM_SUMMARY_MODEL` (default `lite`).
4. `README.md` — models section: add `lite` (document summaries — this phase; KB overview in phase 31) next to `turbo`/`embed`.
- ASSUMPTION: the model name is `lite` per the TODO item; single (non-streaming) completion with `temperature=0.2`, `max_tokens=2048` — summaries/outlines are short, so a fixed budget is enough (no new setting).
## Testing & Quality
- Unit: `tests/unit/test_llm_client.py` — extend the existing mock-transport pattern: `chat()` returns content (trimmed); HTTP ≥400 → `LLMError`; empty content → `LLMError`; explicit `model=` overrides the default (`llm_summary_model`).
- Coverage: **>90%** on this task's new/modified code (`app/` TOTAL ≥ pre-change).
## Completion Criteria
- [ ] `Settings().llm_summary_model == "lite"` by default; `BOR_LLM_SUMMARY_MODEL` env override works (config test).
- [ ] `chat()` unit tests green; `uv run ruff check . && uv run pyright` clean.
- [ ] No change to `embed`/`chat_stream` behavior (existing suite green).
@@ -1,23 +0,0 @@
# Task 02 — Migration 0004: summary columns
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "provide a textual summary of those documents with a pointer back to the source" (storage)`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Add the schema for storing a document's summary and for marking the extra summary chunk: `documents.summary TEXT NULL` and `chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE`.
## Work
1. `alembic/versions/0004_summary_columns.py` — new revision (down_revision = the 0003 steering-notes revision, whatever `alembic/versions/` currently heads to):
- upgrade: `op.add_column("documents", sa.Column("summary", sa.Text(), nullable=True))`; `op.add_column("chunks", sa.Column("is_summary", sa.Boolean(), nullable=False, server_default=sa.text("false")))`.
- downgrade: drop both columns.
2. `app/models.py` — `Document.summary: Mapped[str | None] = mapped_column(Text, default=None)` (comment: lite-model summary, phase 30); `Chunk.is_summary: Mapped[bool] = mapped_column(Boolean, default=False)` (comment: summary chunk, position −1, phase 30).
3. `tests/integration/test_migration_0004.py` — mirror `tests/integration/test_migration_0002.py`'s style: upgrade to head → both columns exist, `is_summary` default `false`; downgrade to 0003 → both gone; upgrade again → back (round-trip).
## Testing & Quality
- Integration: the migration test above (real Postgres, as `test_migration_0002.py` does).
- Coverage: models are exercised by the existing model tests; `app/` TOTAL ≥ pre-change.
## Completion Criteria
- [ ] `uv run alembic upgrade head` applies cleanly on the dev DB (and `alembic downgrade -1` + `upgrade head` round-trips).
- [ ] `uv run pytest` green (including all pre-existing migration/importer tests — `is_summary` default keeps old rows valid).
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,30 +0,0 @@
# Task 03 — app/rag/summarizer.py (prompt + lite call + pointer)
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "we need a small model to analyze non markdown documents and provide a textual summary of those documents with a pointer back to the source"`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Create the summarizer module: build the `lite` prompt for one document, call the model (task 01), validate the output, and return the summary text with a **code-deterministic** pointer line back to the source.
## Work
1. `app/config.py` — add `summary_max_chars: int = 12_000` (env `BOR_SUMMARY_MAX_CHARS`): the cap on document content sent to the lite model in one call.
2. `app/rag/summarizer.py` (new) —
- `SUMMARY_MODE = "SUMMARY_MODE"` — marker constant the E2E mock keys on in the system prompt (same convention as `DEFLECT_MODE`).
- `build_summary_prompt(source: str, path: str, content: str, max_chars: int | None = None) -> tuple[str, str]` → `(system, user)`:
- system: `SUMMARY_MODE` + instruction — "Write a 3–6 sentence plain-text summary of this document in natural language. Cover what it configures/defines and its most important values. Do not use markdown. Do not invent anything that is not in the document."
- user: the document content, capped at *max_chars* (default `get_settings().summary_max_chars`); on overflow cut at the cap and append the shared `TRUNCATION_MARKER` (imported from `app.rag.retriever`).
- `async def generate_summary(llm, *, source: str, path: str, content: str) -> str` — calls `llm.chat([{"role":"system",…},{"role":"user",…}], model=llm.settings.llm_summary_model)`; validates non-empty after trim (else raise `LLMError` — the client already does this, but re-assert defensively); appends the deterministic pointer line: `f"\nSource: {source}/{path}"` (the pointer is **never** model-generated).
3. `tests/unit/test_summarizer.py` (new) — fake LLM object (duck-typed `chat` + `settings`):
- prompt: system contains `SUMMARY_MODE`; user == full content when under cap; user truncated + `TRUNCATION_MARKER` when over cap (custom and default cap).
- generation: returned text = model text + pointer line `Source: <source>/<path>`; whitespace model text → `LLMError`; `LLMError` from the client propagates.
- ASSUMPTION: the pointer is the literal line `Source: <source>/<path>` appended by code (the TODO's "pointer back to the source"); the model is told what to summarize but not to write the pointer.
## Testing & Quality
- Unit: the tests in Work step 3.
- Coverage: **>90%** on `app/rag/summarizer.py`.
## Completion Criteria
- [ ] `generate_summary` returns a non-empty summary ending in the deterministic pointer line; all unit tests green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] No app endpoint change yet (pipeline integration is task 05).
@@ -1,39 +0,0 @@
# Task 04 — Importer: generate, store, index summaries (best-effort)
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "a small model to analyze non markdown documents and provide a textual summary… The similarity search will have a higher chance of hitting those summaries than the original document"`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Hook summarization into the import pipeline: every **non-markdown** file gets a lite summary stored on `documents.summary` and indexed as one extra embedded chunk (`is_summary`, position −1) — best-effort, so a lite failure never loses the document.
## Work
1. `app/rag/importer.py`:
- `Embedder` protocol — add `async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str: ...` (task 01's `LLMClient.chat` already satisfies it; the protocol is what tests duck-type).
- `ImportSummary` — new counters `summaries: int = 0` and `summary_errors: int = 0`; include both in the `log()` summary line (`… summaries=%d summary_errors=%d`).
- `_index_file` — **after** the existing doc+chunks commit (so the document is safe):
- If `full_path.suffix.lower()` is **not** in `(".md", ".markdown")`: try
`summary = await generate_summary(llm, source=source, path=rel, content=content)`;
delete any existing `is_summary` chunk of this document (re-import replacement);
add `Chunk(document_id=doc.id, position=-1, content=summary, is_summary=True)`;
`vector = (await llm.embed([summary]))[0]`; set `chunk.embedding = vector`, `doc.summary = summary`; `session.commit()`; `summary.summaries += 1`; log `import: summary source=%s path=%s chars=%d`.
- On `LLMError | EmbeddingError`: `session.rollback()`, `summary.summary_errors += 1`, log `import: summary failed source=%s path=%s — %s`, and **continue** (the document row + content chunks stay committed; `doc.summary` remains NULL).
- Markdown files: no summary, `doc.summary` stays NULL, no `is_summary` chunk.
2. `scripts/import_docs.py` — the final `print` gains `summaries=%d summary_errors=%d` from the `ImportSummary`.
3. `tests/unit/test_importer.py` — extend the existing fake embeder with a `chat` method (deterministic: returns `"Summary of " + first token of content`; raises `LLMError` when the content contains the sentinel word `SUMMARY-BLOWUP`):
- non-md file (e.g. `.yaml`) → after import: `doc.summary` set, exactly one `is_summary` chunk at position −1 with a non-NULL embedding; `summary.summaries == 1`.
- `.md` file → `doc.summary is None`, no `is_summary` chunk, `summaries == 0`.
- fail-soft: content with `SUMMARY-BLOWUP` → document fully indexed (chunks present, embedding set), `doc.summary is None`, `summary_errors == 1`, no exception.
- replacement: re-import the same file with changed content → still exactly **one** `is_summary` chunk (old one deleted), new text.
- `log()` line includes the new counters (existing log-format test updated accordingly).
- ASSUMPTION: the summary chunk sits at `position = -1` (content chunks stay 0-based in order) so chunk ordering and the viewer are undisturbed; only one summary chunk per document at a time.
- ASSUMPTION: "non-markdown" = suffix not in `(.md, .markdown)` — every other A9 format (txt, yaml, yml, json, py) gets a summary.
## Testing & Quality
- Unit: the tests in Work step 3 (reuse the existing test file's session/fake-embedder fixtures).
- Coverage: **>90%** on the modified `app/rag/importer.py`; `app/` TOTAL ≥ pre-change.
## Completion Criteria
- [ ] All new unit tests green; existing importer tests green (protocol change is additive).
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] A manual `uv run python -m scripts.import_docs` run against the dev KB logs `summaries=N` for the non-md docs (observable in the importer log line, PLAN §9).
@@ -1,31 +0,0 @@
# Task 05 — Pipeline: summary hits resolve to the full source document
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "the chat LLM can retrieve the source pointed to by the summary document (so as part of the pipeline: if retrieval == summary, fetch documents referenced by summary)"`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Make the retrieval + chat pipeline summary-aware: carry `is_summary` through both candidate lists into the fused result, and record in the per-turn log how many of the selected documents were hit via their summary chunk. The context assembly itself is **unchanged** — a summary chunk's parent *is* the source document, and `select_documents` already feeds the full document (A7 revised / phase 24); this task makes that resolution explicit, asserted, and observable.
## Work
1. `app/rag/retriever.py`:
- `RetrievedChunk` — add field `is_summary: bool = False` (documented: True for the lite-model summary chunk, position −1).
- `_vector_candidates` — select `Chunk.is_summary` and pass it into the constructed `RetrievedChunk`s.
- `_LEXICAL_SQL` — add `c.is_summary AS is_summary`; `_lexical_candidates` passes `row.is_summary`.
- `fuse` needs no change (dataclass passthrough) — but assert in tests that the flag survives fusion.
2. `app/api/chat.py`:
- `TurnPlan` — add `summary_hits: int = 0` (count of selected-document hit chunks with `is_summary`).
- `plan_turn` — after the selected-docs decision is known for each branch, compute `summary_hits = sum(1 for c in chunks if c.is_summary and c.document.id in selected_ids)` and store it on the `TurnPlan` (both HIGH and LOW branches).
- Per-turn log line — add `summary_hits=%d` after `fts_hits=%d` (PLAN §9 line extension; record it in the phase's locked decisions). Update any existing test that asserts the log line format verbatim.
- **No change** to `build_high_prompt`/`build_deflect_prompt` inputs or to `select_documents` — the full source document of a summary hit already lands in `<documents>`; task 06's E2E proves it end-to-end.
3. `tests/unit/test_retriever.py` — `is_summary` survives: vector candidates (flag set), lexical candidates (flag set), `fuse` (both a double-hit and a summary-only lexical hit keep the flag; default stays `False` for legacy chunks).
4. `tests/unit/test_chat_gate.py` — `plan_turn`: a summary chunk on a selected top document → `summary_hits == 1`; a summary chunk on a document **outside** the top-N selection → not counted; no summaries → `0` (existing cases unchanged).
## Testing & Quality
- Unit: the tests in Work steps 3–4; existing chat-gate and retriever tests stay green (new field is defaulted).
- Coverage: **>90%** on modified `app/rag/retriever.py` + `app/api/chat.py`; `app/` TOTAL ≥ pre-change.
## Completion Criteria
- [ ] A summary chunk retrieved via vector **or** lexical carries `is_summary=True` through `retrieve()` (unit).
- [ ] The per-turn log line (PLAN §9) now reads `… fts_hits=… summary_hits=… …` and existing log-format tests are updated + green.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] No prompt change: HIGH/LOW prompts byte-identical for summary-less KBs (covered by existing prompt tests).
@@ -1,35 +0,0 @@
# Task 06 — Deterministic lite mock + story E2E + docs + commit
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — (whole item: bad embedder context for non-markdown docs → lite summaries → summary hits fetch the referenced source; the aipi 'lite' model)`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Close the loop: a deterministic `lite` behavior in the E2E mock, a sentinel fixture proving that **a summary hit still delivers the full source document to the LLM**, the story E2E suite, the story file, README docs, and the phase commit.
## Work
1. `tests/e2e/mock_llm.py` — in the chat-completions handler (non-stream and stream paths), **before** the `DEFLECT_MODE` check: if the system prompt contains `SUMMARY_MODE`, return the deterministic digest
`f"This document covers {' '.join(TOKEN_RE.findall(_user(body).lower())[:24])}."`
— the first 24 tokens of the document content (the summarizer puts the content in the *user* message). Byte-stable for a given fixture.
2. E2E fixture — a new non-markdown fixture doc, e.g. `quadlet/qwen-llamacpp.yaml` under the existing E2E fixture KB (follow the import-dependent fixtures' pattern in `tests/e2e/conftest.py` / the fixture dir used by `test_whole_document_context.py`):
- The document **opens** with a header comment line dense in the question tokens (e.g. `# qwen 3.8 llama.cpp optimal parameters deployment notes`) so the mock's 24-token summary digest contains the question's words, followed by ~4–5 k of other yaml content (so the raw chunks dilute their overlap and the summary chunk ranks first — the mock's embeddings are a pure function of tokens, so the ranking is fully deterministic for a fixed fixture; iterate the fixture text until the E2E assertions hold).
- A unique sentinel `RESE-SUMMARY-SENTINEL-7f3a` on the **last line** of the document (outside the 24-token digest, unreachable from the summary).
3. `tests/e2e/test_document_summaries.py` (new, the story gate) — reuse the E2E conftest app/DB fixtures:
- Import the fixture KB (re-import pattern used by import-dependent stories).
- Ask the question (e.g. "What are the optimal parameters for qwen 3.8 on llama.cpp? show the end of your notes" — the phase-24 tail-echo trigger makes the answer quote the **last 160 chars of the document context**).
- Assert the rendered brain answer contains `RESE-SUMMARY-SENTINEL-7f3a` → the full **source** document was in the LLM context (only possible via the summary→parent-document resolution, since the summary digest cannot contain the sentinel).
- Assert the source chip shows the fixture doc's path and `deflected` is false.
- Control: a markdown fixture doc in the same KB gets **no** summary chunk — assert via the Sources table (admin, `#docs-table`) or a direct DB check in the test: markdown doc's chunk count == raw chunks only; the yaml doc has exactly one `is_summary` row.
4. `.agent/user_stories/document-summaries.md` (new) — narrative + acceptance criteria + Playwright mapping rule (story → `tests/e2e/test_document_summaries.py`), matching the style of `.agent/user_stories/git-sources.md`.
5. `README.md` — new "Document summaries" section: what gets summarized (non-markdown A9 docs), the `Source:` pointer, `BOR_LLM_SUMMARY_MODEL` / `BOR_SUMMARY_MAX_CHARS`, fail-soft behavior, and how summary hits appear in the per-turn log.
6. Commit: `git add` the phase's app/script/test/README files; `git commit --no-gpg-sign -m "feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc"`; move `.agent/phases/todo/30_document_summaries/` → `.agent/phases/complete/` (`.agent/` is gitignored by design — force-add only if the commit must record the plan change, otherwise leave the move out of git).
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_document_summaries.py -v --no-cov` green **in isolation** (Chromium installed; `podman compose up -d db` up; mock LLM — no live aipi needed).
- Regression: run `test_chat_rag.py`, `test_retrieval_quality.py`, `test_import_documents.py`, `test_whole_document_context.py` in isolation — all stay green.
- Full gate: `uv run pytest` + `uv run pytest --cov=app --cov-report=term-missing` (TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `tests/e2e/test_document_summaries.py` green in isolation; the sentinel assertion proves summary hit → full source document.
- [ ] All regression suites listed above green in isolation.
- [ ] Full test gate + lint/type gate green (per this phase's 00_phase.md).
- [ ] Story file + README + `.env.example` complete; one `--no-gpg-sign` commit made.
@@ -1,45 +0,0 @@
# Phase 31 — KB Overview in the System Prompt (lite-generated knowledge-base outline)
**Source:** `TODO.md L4 — "The system prompt should inject basic categories of everything that's been read so the agent knows roughly what its knowledge base contains before the rag retrieval returns documents. This part of the system prompt should be generated by the lite model and should be stored somewhere so it can be updated whenever we import new documents."`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
**Context:** Phase 15 steering notes (the `<tuning>` prompt section, its char budget, and the **byte-identical-when-absent** convention — `app/rag/prompts.py::build_steering_section`), phase 30 (the `lite` client method `LLMClient.chat`, and per-document summaries that make a much better overview input than raw titles), `scripts/import_docs.py` (the place "whenever we import new documents" happens), and the E2E mock's answer-echo convention (the `(tuning: …)` suffix — `tests/e2e/mock_llm.py`).
## Objective
Store a lite-generated, plain-text outline of the knowledge base's basic **categories** in a single-row `kb_overview` table, inject it into **both** chat prompts (HIGH and LOW) as a `<knowledge_base>` section so the agent knows roughly what the KB contains before retrieval, and regenerate it automatically whenever an import changes the KB.
## Dependencies
- `30_document_summaries` (todo) — `LLMClient.chat` + `BOR_LLM_SUMMARY_MODEL` (task 01) and the stored per-document summaries (task 04) that feed the overview input.
- `15_steering_notes` (complete) — the prompt-section pattern this phase mirrors (budget, marker, byte-identical-when-absent, per-turn load in `app/api/chat.py`).
- `11_long_answers` / README import workflow (complete) — `scripts/import_docs.py`'s `main()` structure, which this phase extends with the post-import regeneration.
## Tasks
1. `01_migration_kb_overview.md` — Alembic 0005: single-row `kb_overview` table + `KbOverview` model.
2. `02_overview_generator.md` — `app/rag/overview.py`: `KB_OVERVIEW_MODE` prompt builder, `load_kb_overview`, `regenerate_overview` (best-effort upsert).
3. `03_prompt_injection.md` — `<knowledge_base>` section in HIGH + LOW prompts (budgeted, byte-identical when absent); `plan_turn`/chat wire it in; `kb_chars` in the per-turn log.
4. `04_import_trigger.md` — `import_docs` regenerates the overview after a KB-changing import (shared with phase 32's sync).
5. `05_mock_and_e2e.md` — deterministic `KB_OVERVIEW_MODE` mock + `(kb: …)` echo, `tests/e2e/test_kb_overview.py`, story file, commit.
## Testing & Quality
- Unit: overview generator (prompt build/cap, load, regenerate upsert/fail-soft/zero-docs), prompts (section present/budgeted/absent → byte-identical, ordering vs `<tuning>`), chat gate (`kb_chars`, prompt carries the section).
- Integration: migration 0005 up/down; `import_docs` regeneration trigger (changed vs unchanged imports, failure isolation).
- Coverage: **>90%** on `app/` (`app/` TOTAL ≥ pre-change).
- E2E (mandatory, A16): `tests/e2e/test_kb_overview.py` — one story, run **in isolation**; the injected section is observable in the mock answer via the `(kb: …)` echo (steering precedent).
## Completion Criteria
- [ ] After a KB-changing import, `kb_overview` holds a fresh outline (log line `overview: regenerated docs=… chars=…`); an unchanged re-import does **not** call the lite model.
- [ ] Every chat turn's system prompt (HIGH and LOW) contains the `<knowledge_base>` section when a row exists; with no row, both prompts are **byte-identical** to the pre-phase text (unit-asserted).
- [ ] The per-turn log line records `kb_chars=<n>`; section overflow beyond `BOR_KB_OVERVIEW_MAX_CHARS` is capped with the shared `[…truncated…]` marker.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
- [ ] `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` green in isolation; existing prompt/steering/chat suites stay green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/kb-overview-prompt.md` exists; `.env.example` + README document `BOR_KB_OVERVIEW_MAX_CHARS` / `BOR_OVERVIEW_INPUT_MAX_CHARS` and the regeneration behavior.
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, <knowledge_base> section in HIGH+LOW prompts`); `.agent/phases/todo/31_kb_overview_prompt/` moved to `.agent/phases/complete/`.
## Locked decisions
- **A13** — migration 0005 adds one single-row table `kb_overview` (`id INTEGER PK DEFAULT 1`, `content TEXT NOT NULL DEFAULT ''`, `updated_at TIMESTAMPTZ`); no other schema change.
- **A5 extended** — the overview is generated by the same `lite` model via the same `BOR_LLM_SUMMARY_MODEL` setting and `LLMClient.chat` (phase 30); no new model or package.
- **Prompt-section convention (phase 15 precedent)** — the section is budgeted by `BOR_KB_OVERVIEW_MAX_CHARS` (default **4000**) with the shared `TRUNCATION_MARKER` overflow; **zero/empty row → prompts byte-identical** to pre-phase text. Section order: `<relevance>` → `<knowledge_base>` → `<tuning>` → mode body.
- **Regeneration is best-effort and change-gated** — runs only when an import added/updated at least one document (or no row exists yet); a lite failure logs and leaves the previous overview intact (an old outline is better than none).
- **Overview input is capped** — `BOR_OVERVIEW_INPUT_MAX_CHARS` (default **40 000**) on the document list (source/path/title/first summary line) sent to the model.
- **No per-turn LLM call** — chat turns only *read* the stored row (one indexed PK lookup); generation happens at import/sync time (phase 32's button triggers the same `regenerate_overview`).
- **A16 / A17 honoured** — one dedicated story E2E suite; one atomic `--no-gpg-sign` commit.
@@ -1,22 +0,0 @@
# Task 01 — Migration 0005: kb_overview table
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "should be stored somewhere so it can be updated whenever we import new documents"`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
## Objective
Create the storage for the knowledge-base outline: a single-row `kb_overview` table and its SQLAlchemy model.
## Work
1. `alembic/versions/0005_kb_overview.py` — new revision (down_revision = phase 30's 0004):
- upgrade: create table `kb_overview` (`id INTEGER` PK `server_default sa.text("1")`, `content TEXT NOT NULL server_default sa.text("''")`, `updated_at TIMESTAMPTZ NOT NULL server_default=sa.func.now()`).
- downgrade: drop the table.
2. `app/models.py` — `class KbOverview(Base)`: `id: Mapped[int] = mapped_column(Integer, primary_key=True, server_default="1")`, `content: Mapped[str] = mapped_column(Text, server_default="")`, `updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())`. Docstring: single row, lite-generated KB outline, phase 31.
3. `tests/integration/test_migration_0005.py` — same style as `test_migration_0002.py` / `test_migration_0004.py`: upgrade → table exists with the three columns and defaults; downgrade → gone; upgrade again → back.
## Testing & Quality
- Integration: the migration test above (real Postgres).
- Coverage: model exercised by existing model-test patterns; `app/` TOTAL ≥ pre-change.
## Completion Criteria
- [ ] `uv run alembic upgrade head` clean on the dev DB; round-trip with `alembic downgrade -1` + `upgrade head`.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,29 +0,0 @@
# Task 02 — app/rag/overview.py (generator + loader)
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "basic categories of everything that's been read… generated by the lite model… updated whenever we import new documents"`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
## Objective
Create the overview generator module: build the `lite` prompt from the document catalogue, generate the outline, store it in the single row, and expose a cheap loader for the chat path.
## Work
1. `app/config.py` — add:
- `kb_overview_max_chars: int = 4_000` (`BOR_KB_OVERVIEW_MAX_CHARS`) — prompt-section budget (task 03).
- `overview_input_max_chars: int = 40_000` (`BOR_OVERVIEW_INPUT_MAX_CHARS`) — cap on the document list sent to the model.
2. `app/rag/overview.py` (new):
- `KB_OVERVIEW_MODE = "KB_OVERVIEW_MODE"` — marker the E2E mock keys on (same convention as `SUMMARY_MODE` / `DEFLECT_MODE`).
- `build_overview_prompt(rows: Sequence[tuple[str, str, str, str | None]], max_chars: int | None = None) -> tuple[str, str]` → `(system, user)`. Each row is `(source, path, title, summary)`; system = `KB_OVERVIEW_MODE` + instruction ("From the document list below, write a compact plain-text outline of the basic categories and topics this knowledge base covers. Group by source where useful, use `-` bullet lines, at most ~1500 characters, no markdown headings, and no topics not present in the list."); user = one line per doc `source — path — title — {first line of summary or ''}` joined by newlines, capped at *max_chars* (default `overview_input_max_chars`, overflow → shared `TRUNCATION_MARKER`).
- `load_kb_overview(db: Session) -> str` — the single row's `content` (trimmed) or `""` when the row is missing/empty.
- `async def regenerate_overview(llm, session: Session | None = None) -> bool` — load all documents (`source, path, title, summary` ordered by source, path); **zero documents → leave the existing row untouched, return False**; build the prompt; `text = await llm.chat([system, user], model=llm.settings.llm_summary_model)`; upsert the single row (`id=1`, `content=text`, `updated_at=now(UTC)`); commit; log `overview: regenerated docs=%d chars=%d`; return True. On `LLMError`: log `overview: regeneration failed — %s` and return False (previous row stays — see phase locked decisions).
3. `tests/unit/test_overview.py` (new) — fake LLM (duck-typed `chat` + `settings`), in-memory/SQLite session where the existing test infra allows (else a real-DB integration test in the same style as `test_steering.py`):
- prompt: system contains `KB_OVERVIEW_MODE`; user lines carry source/path/title/first summary line; cap truncates + marker.
- `load_kb_overview`: no row → `""`; row present → content.
- `regenerate_overview`: happy path upserts (content + fresh `updated_at`, returns True); zero docs → no DB write, returns False; `LLMError` → previous row unchanged, returns False.
## Testing & Quality
- Unit/integration: the tests in Work step 3.
- Coverage: **>90%** on `app/rag/overview.py`.
## Completion Criteria
- [ ] `regenerate_overview` is idempotent (single row, always id=1) and fail-soft; all tests green.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,30 +0,0 @@
# Task 03 — `<knowledge_base>` section in both prompts + chat wiring
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "The system prompt should inject basic categories of everything that's been read so the agent knows roughly what its knowledge base contains before the rag retrieval returns documents"`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
## Objective
Inject the stored overview into **every** chat turn's system prompt (HIGH and LOW modes) as a budgeted `<knowledge_base>` section — absent row → byte-identical prompts — and record `kb_chars` in the per-turn log line.
## Work
1. `app/rag/prompts.py`:
- `build_kb_section(overview: str, max_chars: int | None = None) -> str` — empty/whitespace → `""`; otherwise `<knowledge_base>\n` + intro line ("The basic categories of everything in this knowledge base (generated at import time):") + the overview content, budgeted at *max_chars* (default `get_settings().kb_overview_max_chars`) with the shared `TRUNCATION_MARKER` for overflow (exact pattern of `build_steering_section`, including its pathological-budget handling).
- `build_high_prompt(documents, notes=None, kb_overview: str | None = None)` and `build_deflect_prompt(titles, notes=None, kb_overview: str | None = None)` — insert the section **between `<relevance>` and the `<tuning>` section** (i.e. order: `<relevance>` → `<knowledge_base>` → `<tuning>` → mode body); with an empty overview the output is byte-identical to today's text in both modes.
2. `app/api/chat.py`:
- Load per turn: `kb_overview = load_kb_overview(db)` next to the steering-notes load (one PK lookup — no LLM call).
- `plan_turn(chunks, settings, notes=None, kb_overview: str | None = None)` — pass it to both prompt builders; `TurnPlan` gains `kb_chars: int = 0` (length of the stored overview text when a non-empty row exists, else 0).
- Per-turn log line (PLAN §9): add `kb_chars=%d` after `tuning=%d`. Update any existing test asserting the log line format verbatim.
3. `tests/unit/test_prompts.py` —
- HIGH: no overview → byte-identical to the pre-phase builder output (build the expected string with `notes=None, kb_overview=None`); with overview → section present, ordered before `<tuning>` when both exist.
- LOW: same pair of assertions (deflection prompt).
- budget: overview longer than `kb_overview_max_chars` → capped + `TRUNCATION_MARKER`.
4. `tests/unit/test_chat_gate.py` — `plan_turn` with an overview: both branches' `system_prompt` contains the section; `TurnPlan.kb_chars` == len(overview); empty overview → `kb_chars == 0` and prompt unchanged.
## Testing & Quality
- Unit: Work steps 3–4; existing steering/prompt/gate tests stay green (new param is defaulted).
- Coverage: **>90%** on modified `app/rag/prompts.py` + `app/api/chat.py`; `app/` TOTAL ≥ pre-change.
## Completion Criteria
- [ ] HIGH and LOW prompts are byte-identical to pre-phase text when no overview row exists (unit-asserted against the exact strings).
- [ ] With a row, both prompts carry the budgeted `<knowledge_base>` section in the locked order; the per-turn log line shows `kb_chars=…`.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,33 +0,0 @@
# Task 04 — import_docs regenerates the overview after a KB-changing import
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "should be updated whenever we import new documents"`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
## Objective
Wire the "update whenever we import new documents" trigger into the import script: after an import that changed the KB, regenerate the stored overview (best-effort). Phase 32's admin sync button reuses the exact same `regenerate_overview` call.
## Work
1. `scripts/import_docs.py`:
- Restructure `main()`'s single `asyncio.run(import_sources(…))` into one `async def _run()` that (a) runs `import_sources(sources, llm, prune=args.prune, limit=args.limit)` and (b) — when the summary has `added + updated > 0` **or** no overview row exists yet — awaits `regenerate_overview(llm)` (import it from `app.rag.overview`). One event loop, same `LLMClient` instance.
- `--limit` debug runs skip the regeneration (an incomplete walk must not rewrite the outline — mirrors the existing `--prune`-with-`--limit` guard).
- The final `print` gains `overview=updated|skipped|failed` (failed = `regenerate_overview` returned False via LLMError; the import's own exit code is **unchanged** — a failed outline must not fail the import).
- `Limit` guard: when `limit` is set, `added + updated > 0` does *not* trigger regeneration (log `overview: skipped (--limit)`).
2. `tests/integration/test_import_docs_overview.py` (new) — with the git sync mocked out (reuse `test_import_docs_git.py`'s mocking style) and a fake LLM whose `chat` records calls:
- import with changed files → `kb_overview` row written; `chat` called once; print shows `overview=updated`.
- unchanged re-import (same hashes) → `chat` **not** called; print shows `overview=skipped`.
- `chat` raising `LLMError` → exit code still `0` (no import errors), print shows `overview=failed`, previous row untouched.
- `--limit` run with changes → `overview=skipped`.
- first-ever import (no row) with zero *changed* docs is not possible (new docs are "added") — but an empty-source run with no row → no row created, `overview=skipped`.
3. `.env.example` — `BOR_KB_OVERVIEW_MAX_CHARS` (default 4000), `BOR_OVERVIEW_INPUT_MAX_CHARS` (default 40000).
4. `README.md` — import workflow section: the import now refreshes the KB overview after a KB-changing run (fail-soft, `overview=` token in the summary line).
- ASSUMPTION: "whenever we import new documents" = whenever an import **added or updated** at least one document (or the row doesn't exist yet); unchanged re-imports and `--limit` debug runs do not burn a lite call.
## Testing & Quality
- Integration: Work step 2 (real Postgres, mocked git + fake LLM — no live aipi).
- Coverage: `app/` TOTAL ≥ pre-change (the script change is covered by the integration tests; the script itself is outside the `app/` gate).
## Completion Criteria
- [ ] All new integration tests green; `tests/integration/test_import_docs_git.py` stays green.
- [ ] A manual run (`uv run python -m scripts.import_docs`) against the dev KB logs `overview: regenerated docs=… chars=…` after a KB-changing import and `overview=skipped` otherwise.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,31 +0,0 @@
# Task 05 — Deterministic KB_OVERVIEW_MODE mock + story E2E + commit
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — (whole item: system prompt injects basic categories of everything read, lite-generated, stored, updated on import)`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
## Objective
Make the overview observable end-to-end in a deterministic E2E: the mock generates a `KB_OVERVIEW_MODE` outline and echoes the injected section into its answer (the `(tuning: …)` precedent), plus the story suite, story file, and the phase commit.
## Work
1. `tests/e2e/mock_llm.py`:
- Generation: in the chat-completions handler, if the system prompt contains `KB_OVERVIEW_MODE` → return the deterministic outline `f"Knowledge base outline:\n- {first 8 tokens of _user(body), space-joined}"` (the user message carries the document list).
- Echo: in `compose_answer` (all answer paths), if the system prompt contains a `<knowledge_base>` section, append `(kb: <first bullet line of the section>)` — parse with a regex in the `first_tuning_note` style (skip the intro line, take the first `-` line, strip the dash). This mirrors the steering echo exactly.
2. `tests/e2e/test_kb_overview.py` (new, the story gate):
- Seed the `kb_overview` row directly in the DB (the E2E test has DB access via the conftest fixtures — content with a recognizable first bullet, e.g. `- Kubernetes cluster and node maintenance notes`), so the test exercises the **injection** path deterministically (the CLI trigger path is covered by task 04's integration tests).
- Ask a normal on-topic question (the fixture KB is already imported by the conftest pattern) → assert the rendered brain answer ends with `(kb: Kubernetes cluster and node maintenance notes)`.
- Deflection control: ask an off-topic question (deflection path) → the answer still carries the `(kb: …)` echo (the section is in the LOW prompt too).
- Absence control: delete the row → a fresh question's answer has **no** `(kb: …)` suffix (byte-identical prompt behavior is unit-asserted in task 03; this proves it end-to-end).
3. Integration test (same task): extend `tests/integration/test_chat_api.py` (or a new `test_kb_overview_api.py`) with the capturing-fake-LLM pattern — no row: the system prompt sent to the model equals the pre-phase construction (assert the exact string via the existing `build_high_prompt`/`build_deflect_prompt` with `kb_overview=None`); row present: it contains the section in both HIGH and LOW turns.
4. `.agent/user_stories/kb-overview-prompt.md` (new) — narrative + acceptance criteria + Playwright mapping rule, styled like the other story files.
5. Commit: `git commit --no-gpg-sign -m "feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, <knowledge_base> section in HIGH+LOW prompts"`; move `.agent/phases/todo/31_kb_overview_prompt/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` green **in isolation**.
- Regression: `test_steering.py`, `test_chat_rag.py`, `test_honest_deflection.py` stay green in isolation (prompt change is additive and defaulted).
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `tests/e2e/test_kb_overview.py` green in isolation (injection, deflection, and absence all asserted).
- [ ] Integration prompt-capture tests green; existing steering/chat suites green.
- [ ] Full test + lint/type gates green (per this phase's 00_phase.md).
- [ ] Story file + `.env.example` + README complete; one `--no-gpg-sign` commit made.
@@ -1,45 +0,0 @@
# Phase 32 — Admin Sync Button (one-click doc import sync)
**Source:** `TODO.md L5 — "Need a button that only the admin can see that triggers a doc import sync by cloning the relevant repos and then running import doc script"`
**Story:** `.agent/user_stories/admin-sync-button.md`
**Context:** Phase 28 (`scripts/git_sync.py::clone_or_pull` — shallow clone / `--ff-only` pull; `BOR_GIT_SOURCES` + `BOR_SOURCES_DIR`; `repo_name` in `scripts/import_docs.py`), phase 31 (`regenerate_overview` — the sync refreshes the KB outline), phase 16 (`require_admin` dependency + the `header.js` `fetchIsAdmin()` reveal gate for admin-only UI like `#nav-sources` / `#nav-tuning`), PLAN §7.4 "never stale" feedback contract (the UI can never sit on a stale button state).
## Objective
Give the admin a **"Sync sources"** button (Sources page, visible to the admin only) that triggers the full document sync in-process — clone/pull every `BOR_GIT_SOURCES` repo, re-import (with prune) so the KB mirrors the repos, and refresh the KB overview — with live, non-stale UI feedback driven by a polled sync-status endpoint.
## Dependencies
- `31_kb_overview_prompt` (todo) — `regenerate_overview(llm)` is the sync's final step; `LLMClient.chat` for it.
- `28_git_based_sources` (complete) — `clone_or_pull` / `GitSyncError` / `BOR_GIT_SOURCES` / `repo_name` (the sync reuses them, does not re-implement git).
- `16_admin_auth` (complete) — `require_admin` for the new endpoints; the `header.js` whoami gate for the button.
- `19_shared_header` / `29_tuning_nav_link` (complete) — the Sources page header actions area where the button lives.
## Tasks
1. `01_sync_api.md` — in-process sync runner: `POST /api/sync` (admin, 409 when running) + `GET /api/sync/status` (admin).
2. `02_ui_button.md` — the admin-only button on Sources with §7.4 feedback states (polling, last-result, error banner) + frontend unit assertions.
3. `03_e2e_and_docs.md` — `tests/e2e/test_sync_button.py` (real `file://` git fixture), README, story file, commit.
## Testing & Quality
- Integration: sync API — anonymous 403s, admin idle/running/success/failed transitions, 409 double-trigger, GitSyncError → `failed` with the repo named (git + import + overview mocked, as `test_import_docs_git.py` does).
- Unit (frontend-assertion style, cf. `tests/unit/test_shared_header.py`): button markup hidden-by-default + labeled; `header.js` reveal; `sources.js` polling/terminal-state logic.
- Coverage: **>90%** on `app/` (`app/api/sync.py` fully covered); `app/` TOTAL ≥ pre-change.
- E2E (mandatory, A16): `tests/e2e/test_sync_button.py` — one story, run **in isolation**; uses a **real** local `file://` git repo fixture (deterministic, no network) with the mock LLM for embeddings.
- UI Structure Check (AGENTS.md rule 5): labeled button, focus-visible, contrast ≥4.5:1, `aria-live` result region, no CDN.
## Completion Criteria
- [ ] Anonymous: the button is not revealed (stays `hidden`) and both endpoints return 403.
- [ ] Admin: clicking "Sync sources" starts the sync (202), the button goes disabled with "Syncing…" while polling `GET /api/sync/status` every 2 s, and on completion shows the last result (`Synced HH:MM` + `N added · M updated`); a failed sync re-enables the button with an error banner (`role="alert"`) naming the failure.
- [ ] A double trigger while running returns 409 and the UI never starts a second poll loop.
- [ ] After a successful sync against the `file://` fixture repo, the newly committed fixture doc appears in the Sources table and the `kb_overview` row is fresh (phase-31 trigger).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
- [ ] `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` green in isolation; `test_admin_auth.py`, `test_shared_header.py`, `test_import_documents.py` stay green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/admin-sync-button.md` exists; README documents the button (behavior, states, prerequisites).
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status`); `.agent/phases/todo/32_admin_sync_button/` moved to `.agent/phases/complete/`.
## Locked decisions
- **A10 extended (recorded, not a revision)** — two new **admin-only** endpoints (`POST /api/sync`, `GET /api/sync/status`) behind the existing `require_admin`; the public API surface stays stateless, the signed cookie remains the only session state (same pattern as `/api/steering`).
- **A12 untouched** — the sync runs **in-process** (one `asyncio` background task + a module-level status object in `app/api/sync.py`). The app is a single instance on the homelab; no Valkey/queue. Status is in memory — a restart mid-sync loses the running state (accepted: the next click re-syncs idempotently).
- **Sync semantics** — the button targets `BOR_GIT_SOURCES` only (manual `--source` dirs have no repo to clone; an unset/empty `BOR_GIT_SOURCES` → the sync fails loudly with "no git sources configured"); the import runs with **`prune=True`** so files deleted upstream leave the index (the button is the canonical "mirror the repos" action — the CLI default of no-prune is unchanged); phase-31's `regenerate_overview` runs after the import when docs changed.
- **Concurrency** — one sync at a time: `POST /api/sync` while running → `409 {"detail": "a sync is already running"}`; the UI reflects the in-flight run (re-attaches on page load while a sync is running).
- **§7.4 adaptation (recorded)** — the 120 s client guard applies to LLM turns; a sync can legitimately run for minutes (clone + embed), so the button has **no client-side hard timeout** — the 2 s status poll is the feedback loop and the server state is authoritative. The button is disabled until the run reaches a terminal state, so it can never be stale *or* stuck: a failed run re-enables it, a running run always shows "Syncing…".
- **A16 / A17 honoured** — one dedicated story E2E suite (real `file://` git fixture — git is a documented environment prerequisite, as in phase 28); one atomic `--no-gpg-sign` commit.
@@ -1,37 +0,0 @@
# Task 01 — Sync API: in-process runner + status
**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — "triggers a doc import sync by cloning the relevant repos and then running import doc script"`
**Story:** `.agent/user_stories/admin-sync-button.md`
## Objective
The backend of the sync button: an admin-only `POST /api/sync` that starts the clone → import → overview pipeline as one in-process background task, and `GET /api/sync/status` for the UI's polling loop.
## Work
1. `app/api/sync.py` (new):
- `@dataclass SyncStatus` — `state: Literal["idle", "running", "success", "failed"] = "idle"`, `started_at: datetime | None`, `finished_at: datetime | None`, `detail: dict[str, Any] = field(default_factory=dict)`, `error: str | None`; module-level `_status` + `_task: asyncio.Task | None`.
- `GET /api/sync/status` (`Depends(require_admin)`) → JSON `{state, started_at, finished_at, detail, error}` (datetimes ISO-8601 or null).
- `POST /api/sync` (`Depends(require_admin)`) — if `_task` is not done → `409 {"detail": "a sync is already running"}`; else `_task = asyncio.create_task(_run_sync())` → `202 {"detail": "sync started"}`.
- `async def _run_sync()`:
1. `_status.state = "running"`, `started_at = now(UTC)`.
2. Resolve repos from `settings.git_source_list` — empty → fail with `"no git sources configured (BOR_GIT_SOURCES)"`.
3. For each URL: `clone_or_pull(url, Path(settings.sources_dir).expanduser() / repo_name(url))` (imported from `scripts.git_sync` / `scripts.import_docs` — no git re-implementation; `GitSyncError` carries git's stderr).
4. `summary = await import_sources(sources, LLMClient(), prune=True)` (prune per phase locked decision).
5. If `summary.added + summary.updated > 0`: `await regenerate_overview(llm)`.
6. `_status.state = "success"`, `finished_at`, `detail = {files, added, updated, unchanged, pruned, errors, chunks, summaries, summary_errors, overview: bool}`; log `sync: done detail=…`.
7. Any `GitSyncError | EmbeddingError | Exception` → `_status.state = "failed"`, `finished_at`, `error = str(e)` (sanitized: no secrets; git's stderr is fine), `logger.exception("sync: failed")`.
2. `app/main.py` — `from app.api.sync import router as sync_router` + `app.include_router(sync_router, prefix="/api")` (next to the other routers).
3. `tests/integration/test_sync_api.py` (new) — sign in via the existing auth test helper (`tests/integration/test_auth_api.py` pattern):
- anonymous: `GET /api/sync/status` → 403; `POST /api/sync` → 403.
- admin: idle state initially; `BOR_GIT_SOURCES` set to one `file://` URL with `clone_or_pull`, `import_sources`, `regenerate_overview` **monkeypatched** in `app.api.sync` (the mock import returns a canned `ImportSummary`; the mock overview returns True) → `POST` → 202; poll status → `success` with the canned detail (all ImportSummary fields + `overview: true`).
- 409: mock runner sleeps briefly (asyncio.sleep) → second `POST` while running → 409.
- failure: mock `clone_or_pull` raises `GitSyncError("git clone failed …")` → status `failed`, `error` names the failure; import is **not** called.
- empty `BOR_GIT_SOURCES` → `POST` 202 → status `failed` with the "no git sources configured" message.
- prune: assert the monkeypatched `import_sources` received `prune=True`.
## Testing & Quality
- Integration: Work step 3 (real Postgres not required for the runner logic beyond none — keep DB-free; if the session needs Postgres for nothing, use the app fixture without DB).
- Coverage: **>90%** on `app/api/sync.py` (all states/branches hit).
## Completion Criteria
- [ ] All integration tests green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] SSE/API routes untouched — `test_chat_api.py` green (no middleware or router precedence change).
@@ -1,40 +0,0 @@
# Task 02 — The admin-only Sync button on Sources (§7.4 feedback)
**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — "a button that only the admin can see that triggers a doc import sync"`
**Story:** `.agent/user_stories/admin-sync-button.md`
## Objective
The UI: a **"Sync sources"** button in the Sources page header — hidden by default, revealed only for the signed-in admin (the existing `header.js` whoami gate) — with the full "never stale" feedback lifecycle: idle → "Syncing…" (disabled, spinner, 2 s status polling) → last-result label or error banner.
## Work
1. `frontend/sources.html` — in the header actions area (next to `.new-chat-btn`, inside the same `.header-inner` container the phase-19 shared header uses on this page):
- `<button type="button" class="sync-btn" id="sync-btn" hidden aria-label="Sync sources">` — a small refresh-cycle `<svg aria-hidden="true">` icon (spin it via CSS in the running state) + `<span class="sync-label" id="sync-label">Sync sources</span>`.
- `<span class="sync-result" id="sync-result" role="status" aria-live="polite"></span>` right after the button (announces last-result / counts to screen readers).
- Update the `.page-sub` copy: it still says "Re-run the import to refresh" — extend it to mention the button (e.g. "…or hit **Sync sources** in the header to clone the repos and re-import.").
2. `frontend/assets/header.js` — add `#sync-btn` to the **admin reveal** that already handles `#nav-sources` / `#nav-tuning` after `fetchIsAdmin()` (one fetch, no extra whoami call); anonymous users never see it (stays `hidden`).
3. `frontend/assets/sources.js` — the sync state machine (new, isolated section):
- On load (admin only — `header.js` exposes the whoami result or a shared `isAdmin` flag; reuse whatever mechanism it already provides for the nav reveals): `GET /api/sync/status` →
- `running` → enter the running state and start polling (the user may have reloaded mid-sync).
- `success` / `failed` → render the last result (below) but keep the button ready for a fresh sync.
- Click → `POST /api/sync` → `202` → running state: button `disabled` + `aria-busy="true"`, icon spinning, label **"Syncing…"**, start polling `GET /api/sync/status` every **2000 ms**.
- Terminal state (stop polling):
- `success` → enabled, icon reset, label **"Synced HH:MM"** (local time of `finished_at`), `#sync-result` = `"{added} added · {updated} updated · {pruned} pruned"` (omit zero terms) — announced via `aria-live`.
- `failed` → enabled, label **"Sync sources"** (retry-ready), and show the page error banner (the existing `role="alert"` pattern used elsewhere in `sources.js`, or the chat error-banner markup style) with the `error` text; `#sync-result` cleared.
- `409` on POST (a run started elsewhere) → just enter running state + polling (adopt the in-flight run); `403` → treat as not-admin (hide the button — defense in depth).
- **No client-side hard timeout** (phase locked decision — the poll is the feedback loop; the server state is authoritative).
4. `frontend/assets/styles.css` — `.sync-btn` styled like `.new-chat-btn`/`.auth-link` (dark tech theme tokens; text contrast ≥4.5:1 — use the dark-ink-on-brand pairing per PLAN §7.2 if the button is filled, else soft-ink on surface), `.sync-btn[disabled]` state, `.sync-btn .sync-icon.is-spinning { animation: spin 1s linear infinite }` with the existing `prefers-reduced-motion` opt-out, `:focus-visible` 3px outline, ≥44 px touch target on mobile.
5. `tests/unit/test_sync_button.py` (new, frontend-assertion style of `test_shared_header.py` / `test_frontend_feedback.py`):
- `sources.html` contains `#sync-btn` with `hidden`, `aria-label="Sync sources"`, and `#sync-result` with `role="status"` + `aria-live="polite"`.
- `header.js` reveals `#sync-btn` in the admin branch (assert the element id appears in the reveal logic, same as `#nav-sources`).
- `sources.js` references `/api/sync` (POST + status GET), the 2000 ms poll, `409` adoption, `403` hide, and the terminal labels (`Syncing…` / `Synced` / failure banner).
- no-CDN integration test stays green (no new external references).
## Testing & Quality
- Unit: Work step 5 (frontend-assertion tests); no-CDN integration test green.
- Coverage: `app/` TOTAL unchanged (frontend-only task); the UI is gated end-to-end by task 03's E2E.
## Completion Criteria
- [ ] Anonymous: `#sync-btn` never leaves `hidden` in the DOM; admin: it is revealed without a page reload round-trip beyond the existing whoami fetch.
- [ ] Full lifecycle works against the dev server (manual check): click → "Syncing…" (disabled) → "Synced HH:MM" + counts, or error banner with retry; reload mid-sync re-enters the running state.
- [ ] UI Structure Check (AGENTS.md rule 5): labeled + focus-visible + contrast ≥4.5:1 + `aria-live` result; reduced-motion respected; no CDN.
- [ ] `uv run pytest` green (including the new frontend-assertion tests); `uv run ruff check . && uv run pyright` clean.
@@ -1,34 +0,0 @@
# Task 03 — Story E2E (real file:// git fixture) + README + commit
**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — (whole item: admin-only button → clone the relevant repos → run the import script)"`
**Story:** `.agent/user_stories/admin-sync-button.md`
## Objective
The story gate: a deterministic E2E that runs the **real** sync path end-to-end (real `git clone` of a local `file://` fixture repo, real import against the mock LLM, real overview regeneration) and verifies both the admin-only visibility and the full button lifecycle, plus README docs, story file, and the phase commit.
## Work
1. `tests/e2e/test_sync_button.py` (new) — module fixtures:
- Build a temp git repo in a `tmp_path` via `subprocess` (`git init -q`, write `notes/sync-fixture.md` containing a unique sentinel `RESE-SYNC-SENTINEL-9b2c`, `git add -A && git -c user.email=e@x -c user.name=t commit -qm one`); set the app's env for this module: `BOR_GIT_SOURCES=file://<repo>`, `BOR_SOURCES_DIR=<tmp_path>/checkouts` (follow the E2E conftest pattern for per-module app env; git is a documented environment prerequisite — phase 28).
- Truncate `query_log` (and the KB tables the fixture needs) per the existing E2E isolation pattern so the run starts clean.
2. Tests (in isolation):
- **`test_anonymous_sees_no_button`** — load `/sources.html` logged out: `#sync-btn` is `hidden` (or absent from the revealed DOM); `POST /api/sync` via `page.request` → 403.
- **`test_admin_sync_lifecycle`** — sign in (reuse `tests/e2e/auth_helpers.py`):
- `#sync-btn` visible with label "Sync sources".
- Click → button disabled, label "Syncing…".
- Wait (poll with Playwright, generous timeout ~60 s — real git + embed against the mock LLM): label becomes `Synced …`, `#sync-result` shows `1 added` (the fixture doc).
- The Sources table (`#docs-tbody`) now contains the fixture path `notes/sync-fixture.md`; the `kb_overview` row is non-empty (phase-31 regeneration ran — DB check in the test).
- Re-click → a second run completes with `0 added · 1 unchanged` (idempotent pull + hash-skip).
- **`test_double_trigger_409`** (integration-level, may live in `tests/integration/test_sync_api.py` if E2E timing is too flaky — the integration test already covers this; include here only if deterministic): start a sync, immediately `POST /api/sync` again → 409.
3. `README.md` — new "Sync from the UI" subsection under the import workflow: what the button does (clone/pull → import `--prune`-equivalent → KB overview refresh), the states (Syncing…/Synced/error), the 409 behavior, and prerequisites (`BOR_GIT_SOURCES` set; git available).
4. `.agent/user_stories/admin-sync-button.md` (new) — narrative + acceptance criteria + Playwright mapping rule.
5. Commit: `git commit --no-gpg-sign -m "feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status"`; move `.agent/phases/todo/32_admin_sync_button/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` green **in isolation** (Chromium + `podman compose up -d db` + git on PATH; mock LLM — no live aipi).
- Regression in isolation: `test_admin_auth.py`, `test_shared_header.py`, `test_import_documents.py`, `test_global_tuning.py`.
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] All E2E tests green in isolation, including the idempotent second run and the `kb_overview` freshness check.
- [ ] Regression suites green; full test + lint/type gates green (per this phase's 00_phase.md).
- [ ] README + story file complete; one `--no-gpg-sign` commit made.
@@ -1,40 +0,0 @@
# Phase 33 — Cache Busting (un-stick the pages)
**Source:** `TODO.md L6 — "We need better cache busting, the pages are too sticky"`
**Story:** `.agent/user_stories/cache-busting.md`
**Context:** `app/main.py` serves the whole `frontend/` directory through one `StaticFiles(html=True)` catch-all mount; the five HTML pages reference assets **without any version** (`href="/assets/styles.css"`, `src="assets/markdown.js"`, `src="/assets/app.js"`, …), so browsers happily keep stale CSS/JS/HTML after a deploy — the "too sticky" report. The no-CDN integration test (`tests/integration/test_api.py::test_html_pages_served_locally_no_cdn`) asserts no `https://` references — appending `?v=` keeps every reference same-origin, so it stays green. SSE/API live under `/api/*` and must be untouched (SSE already ships `Cache-Control: no-cache` itself).
## Objective
A deploy must be visible without a hard refresh: HTML pages are **always revalidated** (`Cache-Control: no-cache`) and reference their assets with a version token (`?v=<token>`); assets are served **immutable for 1 year** (the token in the URL identifies the content, so long caching is safe). Zero new services, zero build-step changes, no CDN.
## Dependencies
- `32_admin_sync_button` (todo) — sequencing only; no shared code (this phase is transport-layer and independent of the RAG work).
## Tasks
1. `01_asset_version_token.md` — `app/core/caching.py::asset_version()`: git short SHA (homelab checkouts have a `.git`), stable mtime+size content-hash fallback, computed once per process.
2. `02_caching_middleware.md` — the response middleware (HTML `no-cache` + `?v=` rewrite; `/assets/*` immutable) wired into `create_app` + integration tests.
3. `03_e2e_and_docs.md` — `tests/e2e/test_cache_busting.py` (Playwright header assertions), README, story file, commit.
## Testing & Quality
- Unit: version token (git path, fallback path, failure path), the asset-reference rewrite (both `href`/`src` and leading-slash-less `assets/…` refs, no double-`?v=`).
- Integration: page headers + rewritten references; asset headers; no-CDN test green; SSE endpoint responses untouched (existing chat SSE tests green).
- Coverage: **>90%** on `app/` (the new `app/core/caching.py` fully covered); TOTAL ≥ pre-change.
- E2E (mandatory, A16): `tests/e2e/test_cache_busting.py` — one story, run **in isolation**; real Chromium asserting the headers and the versioned request URLs the browser actually makes.
## Completion Criteria
- [ ] Every HTML page (`/`, `/sources.html`, `/document.html`, `/login.html`, `/tuning.html`) is served with `Cache-Control: no-cache` and its asset references carry `?v=<token>` (token non-empty, stable across requests, changes when the frontend content changes).
- [ ] `/assets/*` responses carry `Cache-Control: public, max-age=31536000, immutable`.
- [ ] `/api/*` (incl. the SSE chat stream) responses are byte-for-byte header-wise unaffected beyond what they already send; no-CDN integration test green.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
- [ ] `uv run pytest tests/e2e/test_cache_busting.py -v --no-cov` green in isolation; `test_smoke.py` + one RAG E2E stay green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/cache-busting.md` exists; README documents the caching behavior + how the token changes on deploy.
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `perf(ui): cache busting — HTML no-cache + versioned asset URLs (?v=) with immutable 1y asset caching`); `.agent/phases/todo/33_cache_busting/` moved to `.agent/phases/complete/`.
## Locked decisions
- **Version token** — `asset_version()`: if the project checkout has a `.git` (the homelab reality), the token is `git rev-parse --short HEAD` (a commit = a deploy, so the token flips on every deploy); otherwise a stable hash of the frontend tree (sorted `relpath + mtime_ns + size`, first 12 hex chars) so dev checkouts still bust. Computed **once per process** (`lru_cache`) — zero per-request git/file cost.
- **Rewrite scope** — only the five known HTML pages are rewritten (a small regex over `href="…assets/…"` / `src="…assets/…"` appending `?v=` when absent). No templating layer, no build step, no changes to the static files themselves (the `Containerfile` esbuild stage is untouched).
- **Asset caching** — `/assets/*` are cached `immutable` for 1 year **because** the URL carries the token; the unversioned path keeps working (StaticFiles ignores the query string), so old tabs and tests referencing `/assets/x.js` directly still resolve.
- **Middleware boundary** — the middleware touches exactly two shapes: the five page paths (body rewrite + `no-cache`) and `/assets/*` (header only). Everything else — all `/api/*` including SSE — passes through byte-identical (SSE keeps its own `no-cache`). Implemented as a Starlette middleware that only rewrites `text/html` responses under the page paths; if `Response.body()` turns out to misbehave on the `FileResponse` streaming path, the fallback is five explicit FastAPI routes that read + rewrite the files (identical observable behavior — the executor picks whichever passes the tests).
- **A11 untouched** — no CDN, no new packages, no new services (A12 untouched).
- **A16 / A17 honoured** — one dedicated story E2E suite; one atomic `--no-gpg-sign` commit.
@@ -1,28 +0,0 @@
# Task 01 — app/core/caching.py: the asset version token
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — "We need better cache busting, the pages are too sticky"`
**Story:** `.agent/user_stories/cache-busting.md`
## Objective
One source of truth for the asset version token: git short SHA when the checkout is a repo (a commit = a deploy), a stable content-hash fallback otherwise — computed once per process.
## Work
1. `app/core/caching.py` (new) —
- `def asset_version(static_dir: str | None = None) -> str`:
- `static_dir` defaults to `get_settings().static_dir` (resolves to `frontend`); the git repo root is its parent.
- **Git path:** if `(static_dir parent / ".git")` exists → `subprocess.run(["git", "-C", str(root), "rev-parse", "--short", "HEAD"], capture_output=True, text=True, timeout=5)` → the short SHA (e.g. `3841bd5`).
- **Fallback / failure path** (no `.git`, git missing, non-zero exit, timeout, unreadable): `hashlib.sha256` over the sorted list of `f"{relpath}:{mtime_ns}:{size}"` for every regular file under `static_dir`, first **12 hex chars**. A missing/empty `static_dir` → `"dev"`.
- `@functools.lru_cache(maxsize=None)` on the resolved-argument wrapper (settings are process-stable; the token must not be recomputed per request). Document that a process restart or new commit changes the token.
2. `tests/unit/test_caching.py` (new):
- git path: a `tmp_path` repo (`git init -q` + a commit of a dummy file, with a `frontend/` subdir inside) → token == `git rev-parse --short HEAD` of that repo; second call returns the same value (cache).
- fallback: a plain `tmp_path/frontend` with two files → 12-hex token; unchanged tree → same token; touch/modify a file (mtime or size change) + `asset_version.cache_clear()` → different token.
- failure: `static_dir` with a `.git` present but `git` removed from PATH (monkeypatch `subprocess.run` to raise `FileNotFoundError`) → falls back to the content hash, no exception.
- empty/missing dir → `"dev"`.
## Testing & Quality
- Unit: Work step 2.
- Coverage: **>90%** on `app/core/caching.py` (all branches).
## Completion Criteria
- [ ] All unit tests green (git, fallback, failure, empty — with `cache_clear()` between parametrized cases).
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,38 +0,0 @@
# Task 02 — Caching middleware + wiring + integration tests
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — "the pages are too sticky" (HTML revalidation + versioned asset URLs + immutable assets)`
**Story:** `.agent/user_stories/cache-busting.md`
## Objective
Apply the caching behavior at the transport layer: HTML pages `no-cache` with `?v=<token>` on every local asset reference; `/assets/*` immutable 1-year; everything else (all `/api/*`, including SSE) untouched.
## Work
1. `app/core/caching.py` (extend task 01's module):
- `HTML_PAGES: tuple[str, ...] = ("/", "/index.html", "/sources.html", "/document.html", "/login.html", "/tuning.html")`.
- `_ASSET_REF_RE = re.compile(r'((?:href|src)="(?:/)?assets/[^"?#]+)(")')` — matches `<link rel="stylesheet" href="/assets/styles.css">`, `<script src="assets/markdown.js"></script>` (no leading slash!), and `<script type="module" src="/assets/app.js">`; the rewrite appends `?v=<token>` before the closing quote, only when the reference has no query/hash yet (idempotent — never a double `?v=`).
- `def rewrite_asset_refs(html: str, token: str) -> str` — the pure, unit-testable rewrite.
- `def configure_caching(app: FastAPI) -> None` — one `@app.middleware("http")` (or equivalent Starlette middleware) that, **after** the response is produced:
- `path.startswith("/assets/")` → `response.headers["Cache-Control"] = "public, max-age=31536000, immutable"` (header only — never touch the body).
- `request.url.path` in `HTML_PAGES` **and** response content-type is `text/html` → `Cache-Control: no-cache` + `response.body = rewrite_asset_refs(body, asset_version())` (body via `await response.body()` — works for the buffered StaticFiles/FileResponse HTML responses; on any error or non-`text/html`, fall through to the unmodified response with only `no-cache`).
- everything else → completely untouched (no header, no body work). The `/api/*` SSE stream in particular must not be read or rewritten.
2. `app/main.py` — `from app.core.caching import configure_caching`; call `configure_caching(app)` inside `create_app()` after the routers (order: middleware wraps the whole app — call it before `return app`).
3. `tests/unit/test_caching.py` (extend) — `rewrite_asset_refs`:
- versioned: `href="/assets/styles.css"` → `href="/assets/styles.css?v=abc123"`; `src="assets/markdown.js"` (no slash) → versioned; `src="/assets/app.js"` (module script) → versioned.
- idempotent: an already-`?v=`-tagged reference is not double-tagged; a `#fragment` or existing `?query` reference is left alone.
- non-asset references untouched (`href="/sources.html"`, `href="data:…"`, `href="/login.html?next=…"`).
4. `tests/integration/test_api.py` (extend):
- `GET /` → 200, `cache-control: no-cache`; body contains `href="/assets/styles.css?v=<token>"` with a non-empty token matching `asset_version()`; the unversioned string `href="/assets/styles.css">` is **gone** from the body.
- each of the other four pages (`/sources.html`, `/document.html`, `/login.html`, `/tuning.html`) → `no-cache` + at least one versioned asset reference.
- `GET /assets/styles.css` → 200, `cache-control` contains `immutable` and `max-age=31536000`.
- `GET /api/health` → response has **no** `cache-control` injected (baseline: FastAPI's default) — assert equality with the pre-middleware behavior; the SSE chat endpoint (`tests/integration/test_chat_api.py`) stays green unmodified.
- the existing no-CDN test (`test_html_pages_served_locally_no_cdn`) stays green — the rewritten references are still same-origin.
5. `tests/unit/test_main.py` — app creation still succeeds with the middleware wired (existing creation tests stay green; add an assertion that `create_app()`'s middleware stack includes the caching middleware by name).
## Testing & Quality
- Unit: Work steps 3 + 5; Integration: Work step 4.
- Coverage: **>90%** on `app/core/caching.py`; `app/` TOTAL ≥ pre-change.
## Completion Criteria
- [ ] All new unit + integration tests green; the full suite green (especially `test_chat_api.py` — SSE unaffected).
- [ ] Manual check: `curl -si localhost:8000/ | grep -i cache-control` → `no-cache`; the HTML body shows `?v=…` asset refs; `curl -si localhost:8000/assets/styles.css | grep -i cache-control` → immutable.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,32 +0,0 @@
# Task 03 — Story E2E (Playwright header assertions) + README + commit
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — (whole item: better cache busting — the pages are too sticky)"`
**Story:** `.agent/user_stories/cache-busting.md`
## Objective
The story gate: a real-browser E2E that asserts what the browser actually receives (HTML `no-cache`, versioned asset request URLs, immutable asset headers), plus README docs, story file, and the phase commit.
## Work
1. `tests/e2e/test_cache_busting.py` (new) — collect responses with `page.on("response")`:
- `test_html_pages_are_no_cache_and_versioned` — navigate to `/`:
- the document response's `cache-control` header is `no-cache`;
- the `styles.css` request URL contains `?v=` and the response's `cache-control` contains `immutable` + `max-age=31536000`;
- the `app.js` request URL contains the **same** token value as the CSS one (single token per process);
- the served HTML (`page.content()`) contains no unversioned `/assets/styles.css"` reference.
- `test_other_pages_share_the_token` — navigate to `/sources.html` then `/login.html`: each document response is `no-cache`; both pages' CSS requests carry the same token.
- `test_api_responses_unaffected` — `page.request.get("/api/health")` → no `cache-control: no-cache`/immutable injection (the endpoint's baseline headers only); a chat SSE POST still streams (reuse the minimal chat-request helper from an existing E2E — the stream must complete with `done`).
2. `README.md` — new short "Caching / deploys" section: HTML is always revalidated; assets are cached 1 year immutable and carry `?v=<token>`; the token is the git short SHA (falls back to a content hash in non-git checkouts) and flips on every commit/deploy — no hard refresh needed anymore; API/SSE caching is unchanged.
3. `.agent/user_stories/cache-busting.md` (new) — narrative + acceptance criteria + Playwright mapping rule.
4. Commit: `git commit --no-gpg-sign -m "perf(ui): cache busting — HTML no-cache + versioned asset URLs (?v=) with immutable 1y asset caching"`; move `.agent/phases/todo/33_cache_busting/` → `.agent/phases/complete/`.
5. **Deploy note (post-commit, owner action)**: after this phase lands, the *first* deploy also requires browsers to see the new HTML once (revalidation) — one normal navigation; thereafter every commit is picked up automatically.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_cache_busting.py -v --no-cov` green **in isolation** (Chromium + `podman compose up -d db` for the app boot; the mock LLM keeps the SSE check deterministic).
- Regression in isolation: `test_smoke.py`, `test_chat_rag.py`.
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] All E2E tests green in isolation (headers + token consistency + SSE unaffected).
- [ ] Regression suites green; full test + lint/type gates green (per this phase's 00_phase.md).
- [ ] README + story file complete; one `--no-gpg-sign` commit made.
- [ ] `podman compose up -d` (full app) smoke: a fresh browser profile loads the site and every asset request is versioned (manual confirmation recorded in the commit message or phase notes).
@@ -1,47 +0,0 @@
# Phase 35 — Admin Page to Add / Remove Git Sources
**Source:** `TODO.md` L4 — "I need a page only the admin can access where I can add and remove git sources for docs"
**Story:** `.agent/user_stories/git-sources-admin.md`
**Context:** Phase 28 introduced git-based sources (`BOR_GIT_SOURCES` env var + `scripts/git_sync.clone_or_pull`) and phase 32 the one-click Sync button (`POST /api/sync`) — but the *list itself* can only be changed by editing `.env` and restarting. This phase makes the list admin-managed: a Postgres-backed table, an admin-only CRUD API, and a dedicated admin page, with the sync pipeline and `import_docs` resolving the effective list from the DB (env var demoted to an empty-table fallback).
## Objective
Deliver a page **only the admin can access** (`/git-sources.html`, soft-gated like Sources) to **add and remove git sources**, stored in a new `git_sources` table; the Sync button (phase 32) and `import_docs` (phase 28) use the stored list, `BOR_GIT_SOURCES` remains a fallback while the table is empty, and phase 32's fail-loud "no git sources configured" is preserved when both are empty.
## Dependencies
- `28_git_based_sources` (complete) — `scripts/git_sync.clone_or_pull`, `repo_name`, the `BOR_GIT_SOURCES` settings + `git_source_list`, the import resolution order (`--source` wins).
- `32_admin_sync_button` (complete) — the `POST /api/sync` / `GET /api/sync/status` pipeline this phase re-points at the DB list; the Sync button the page's hint refers to.
- `16_admin_auth` (complete) — `require_admin` (the router-level pattern from `app/api/sync.py`), the soft-gate page pattern (`sources.html`), the `fetchIsAdmin()` frontend gate.
- `34_consistent_navbar` (todo) — the identical five-page header this phase's admin-only "Git sources" nav link plugs into (phase 29 pattern).
- `29_tuning_nav_link` (complete) — the admin-only ship-hidden nav-link pattern to copy.
## Tasks
1. `01_model_and_migration.md` — `GitSource` model + migration `0006_git_sources.py` (reversible).
2. `02_git_sources_api.md` — admin-only `GET/POST /api/git-sources` + `DELETE /api/git-sources/{id}` with validation, the env-fallback listing, and the integration suite.
3. `03_sync_and_importer_use_db.md` — `effective_git_sources()` shared by `app/api/sync.py` and `scripts/import_docs.py` (DB wins, env fallback, fail-loud unchanged) + test updates.
4. `04_admin_page.md` — `/git-sources.html` + `git-sources.js` (soft-gated, list / add / remove, env note, sync hint) + styles.
5. `05_nav_link.md` — the admin-only "Git sources" nav link on all five pages + the `header.js` reveal.
6. `06_e2e_and_docs.md` — the story E2E suite `test_git_sources_admin.py`, `test_nav_consistency.py` nav-inventory update, README/`.env.example` notes, regressions, commit.
## Testing & Quality
- Unit/integration: `tests/unit/` for `effective_git_sources` (DB-wins / env-fallback / both-empty); `tests/integration/test_git_sources_api.py` for the CRUD contract (403/201/409/422/404, env fallback); the migration up/down test following the 0004/0005 pattern; the existing `test_sync_api.py` + `test_import_docs_git.py` suites stay green with the resolution indirection.
- Coverage: **>90%** on `app/` for the new module + API.
- E2E (mandatory, A16): `tests/e2e/test_git_sources_admin.py` — the story gate, run in isolation.
## Completion Criteria
- [ ] Migration 0006 applied (`uv run alembic upgrade head`); `git_sources` table exists with `url` unique.
- [ ] `GET /api/git-sources` (admin) lists DB rows; while the table is empty it returns the env list with `from_env: true`; anonymous gets 403 on all three routes.
- [ ] `POST` creates (201, trimmed, shape-validated, 409 duplicate without echoing the URL); `DELETE` removes (204/404).
- [ ] `POST /api/sync` and `import_docs` resolve the list via `effective_git_sources` (origin logged `db|env`); both-empty still fails loudly; `--source` override unchanged.
- [ ] `/git-sources.html`: anonymous sees the sign-in gate; the admin sees list + add + remove with a never-stale button and inline errors; the admin-only "Git sources" nav link is visible on all five pages for the admin and hidden for anonymous.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov` green in isolation; regressions (task 06 list) green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
## Locked decisions
- **A3 / A13 honoured** — the list lives in Postgres via Alembic (no JSON file, no new store).
- **A10 extended per the phase-16 pattern** — a new admin-only router behind `require_admin`; the public API surface stays stateless; no new auth mechanism.
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages.
- **Env var demoted, not removed** — `BOR_GIT_SOURCES` keeps working exactly as today while the table is empty (the fallback); once the table has rows it is ignored (the UI is the source of truth). Phase 32's fail-loud empty-config behavior is preserved.
- **Scope boundary** — adding/removing a repo does NOT immediately clone, import, or prune: the existing Sync button performs that (removal prunes on the next sync, `prune=True`). The page's hint says so.
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
@@ -1,46 +0,0 @@
# Task 01 — GitSource model + migration 0006
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
Add the `git_sources` table (one row per admin-managed repo URL) via the model + a reversible Alembic migration, following the exact conventions of migrations 0003–0005.
## Work
1. `app/models.py` — add the model (next to `SteeringNote`, docstring citing this phase + the A13 convention):
```python
class GitSource(Base):
"""One admin-managed git source (phase 35).
The UI-maintained list of repo URLs the Sync button (phase 32) and
import_docs (phase 28) clone/pull. DB rows win over the
BOR_GIT_SOURCES env var, which is a fallback while this table is
empty (see app.rag.git_sources.effective_git_sources).
"""
__tablename__ = "git_sources"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
)
url: Mapped[str] = mapped_column(Text, unique=True, nullable=False)
added_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now()
)
```
(Import `Text` — already imported in the file; verify.)
2. `alembic/versions/0006_git_sources.py` — new migration:
- Read `alembic/versions/0005_kb_overview.py` first and chain from its actual `revision` id (the filenames are not the revision ids).
- `upgrade()`: `CREATE TABLE git_sources (id UUID PRIMARY KEY, url TEXT NOT NULL, added_at TIMESTAMP WITH TIME ZONE DEFAULT now() NOT NULL)` + `CREATE UNIQUE INDEX uq_git_sources_url ON git_sources (url)` (use `sa.Uuid` / the same column types the other migrations use — mirror their style, including `op.create_table` kwargs and the `UniqueConstraint`-vs-index choice 0003/0004 made).
- `downgrade()`: drop the index + table.
3. Apply it to the dev database: `podman compose up -d db` (if needed) then `uv run alembic upgrade head`.
4. Migration test — follow the existing pattern (see how 0004/0005 are integration-tested — `tests/integration/` migration suite): assert 0006 up creates the table + unique constraint and down drops it (round-trip on the test DB).
## Testing & Quality
- Unit/integration: the migration up/down test above; `uv run pytest` green overall.
- Coverage: model-only for now — the `app/` gate stays >90% (models are thin).
## Completion Criteria
- [ ] `alembic/versions/0006_git_sources.py` exists, chains off 0005's real revision id, and is reversible.
- [ ] `uv run alembic upgrade head` applies cleanly on the dev DB; `git_sources` visible (`\d git_sources` equivalent).
- [ ] The 0006 up/down integration test passes; full `uv run pytest` green.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,39 +0,0 @@
# Task 02 — Admin-only git sources CRUD API
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
The admin CRUD contract for the stored list: `GET /api/git-sources` (DB rows, or the env fallback while the table is empty), `POST /api/git-sources` (validated create), `DELETE /api/git-sources/{id}` — all behind `require_admin`, exactly like `app/api/sync.py`.
## Work
1. `app/schemas.py` — add (mirroring the steering schemas' style):
- `GitSourceIn` — `url: str = Field(min_length=1, max_length=500)` + a `mode="before"` trim validator (whitespace-only → 422, same trick as `SteeringNoteIn`).
- `GitSourceOut` — `id: uuid.UUID | None`, `url: str`, `added_at: datetime | None` (both nullable: env-fallback rows carry neither).
- `GitSourceList` — `sources: list[GitSourceOut]`, `from_env: bool` (`True` only when the table is empty and the list comes from `BOR_GIT_SOURCES`).
2. `app/api/git_sources.py` (NEW) — `router = APIRouter(prefix="/git-sources", tags=["git-sources"], dependencies=[Depends(require_admin)])` (copy the sync.py pattern + docstring style):
- `GET ""` → `GitSourceList`: DB rows ordered by `(added_at, id)`; if the table is empty → the `get_settings().git_source_list` env URLs as rows with `id=None, added_at=None` and `from_env=True`; `from_env=False` whenever DB rows exist (the env var is then ignored — the phase's locked decision).
- `POST ""` (201) → create:
- Shape validation (module-level `URL_RE = re.compile(r"^(https?://|ssh://|git@)")` with a docstring): the trimmed URL must match — covers the phase-28 real URLs (HTTPS + `git@` SSH); scp-style `host:repo` is deliberately rejected (422).
- ASSUMPTION: the accepted shapes are exactly `http://`, `https://`, `ssh://`, `git@…`; the 422 detail is generic ("not a valid git URL (expected https://, ssh:// or git@…)") and never echoes the input.
- Duplicate (same trimmed URL already stored) → 409 with a generic detail ("a git source with this URL already exists") — **never echo the URL** (URLs may embed `user:pass@` credentials; phase 32's masking discipline).
- Success → insert, commit, return the created `GitSourceOut`.
- `DELETE "/{source_id}"` → 204; unknown id → 404 `git source not found`.
3. `app/main.py` — register the router alongside the existing `include_router` calls (check how `sync` is included and mirror it).
4. `tests/integration/test_git_sources_api.py` (NEW) — follow `tests/integration/test_sync_api.py`'s auth pattern (`_login` via `POST /api/login` with the fixture admin password, admin client as context manager):
- Anonymous → 403 `{"detail": "admin only"}` on GET, POST, and DELETE.
- Admin + empty table + env set (monkeypatch the settings `git_sources`) → GET returns the env rows, `from_env=True`, null ids.
- Admin + empty table + env empty → GET returns `sources=[]`, `from_env=True`.
- POST: valid `https://…` → 201 + the row appears in GET with `from_env` now `False`; a duplicate → 409 and the detail contains no URL; an invalid shape (`not a url`, `host:repo`) → 422; whitespace-only / >500 chars → 422; a `git@github.com:…` URL → 201 (accepted).
- DB rows win over env: seed a row AND set the env → GET returns only the DB rows, `from_env=False`.
- DELETE: known id → 204 + gone from GET (back to env fallback if the table is now empty); unknown id → 404.
## Testing & Quality
- Integration: the suite above — the full CRUD contract, auth split, fallback semantics.
- Coverage: **>90%** on the new module (`app/api/git_sources.py` + schemas).
## Completion Criteria
- [ ] All three routes exist under `/api/git-sources`, admin-only (403 anonymous), registered in `app/main.py`.
- [ ] `tests/integration/test_git_sources_api.py` green; `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
- [ ] No credential-echo path: 409/422 details never contain the submitted URL (a test asserts this).
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,38 +0,0 @@
# Task 03 — Sync + import_docs resolve the effective list (DB wins, env fallback)
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
One shared resolver — DB rows win, `BOR_GIT_SOURCES` is a fallback only while the table is empty, fail-loud unchanged when both are empty — used by **both** the in-app sync pipeline (`app/api/sync.py::_run_sync`) and the CLI (`scripts/import_docs.py`), so the admin page's list is what actually gets cloned and indexed.
## Work
1. `app/rag/git_sources.py` (NEW) — the single resolver (importable by both the app and the CLI — `scripts` already imports `app.rag.*`):
```python
def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]:
"""(urls, origin) — DB rows in (added_at, id) order win; while the
table is empty the BOR_GIT_SOURCES env list is the fallback; both
empty → ([], "env")."""
```
- DB rows: `select(GitSource).order_by(GitSource.added_at, GitSource.id)`.
- Fallback: `get_settings().git_source_list` (the phase-28 CSV parse — reuse, don't re-implement).
2. `app/api/sync.py` — `_run_sync` replaces `settings.git_source_list` with the resolver:
- Open a `SessionLocal()` (close in `finally`) around the resolution — the background task has no request session.
- Log the origin: the existing `sync: started repos=N` line gains `origin=db|env`.
- Both-empty: keep the fail-loud `GitSyncError("no git sources configured …")` (extend the message to mention both origins — e.g. `(git_sources table empty and BOR_GIT_SOURCES unset)`; if `tests/integration/test_sync_api.py::test_no_git_sources_configured_fails_loudly` asserts the old text, update that expectation — it is this phase's file to update).
3. `scripts/import_docs.py` — the git-URL resolution branch (today `settings.git_source_list`) resolves via the same function: open a short `SessionLocal()` at resolution time (the import needs the DB anyway — no DB-down fallback to design). The `--source` override still wins (manual mode), the log line records the origin (`git sources: N repo(s) origin=db|env` before the clone loop).
4. **Test updates:**
- `tests/unit/` — NEW unit tests for `effective_git_sources`: DB rows win (seeded table + env set → DB list, origin `db`); env fallback (empty table + env set); both empty → `([], "env")`. (Use a test DB session or a stubbed session following the existing unit-test conventions.)
- `tests/integration/test_sync_api.py` — where it stubs `settings.git_source_list` to drive sync scenarios, keep those scenarios working through the new indirection: either seed the `git_sources` table or monkeypatch `effective_git_sources` (whichever the file's existing fixture style favors); add one scenario asserting a DB row is used over the env when both are set (clone/import mocked — the file already mocks them).
- `tests/integration/test_import_docs_git.py` — same treatment for the CLI path (`--source` override scenario untouched); add the DB-over-env scenario at the CLI level.
## Testing & Quality
- Unit: `effective_git_sources` — all three branches.
- Integration: sync + import_docs suites green with the resolver in the path, incl. the new DB-over-env scenarios and the updated fail-loud expectation.
- Coverage: **>90%** on the new module + the modified call sites.
## Completion Criteria
- [ ] `effective_git_sources` exists in `app/rag/git_sources.py` and is the ONLY place (besides the API's GET fallback, which may call it too) that combines DB + env.
- [ ] `_run_sync` and `import_docs` both resolve through it; origin visible in their logs.
- [ ] Both-empty still raises the fail-loud error (sync) / the CLI's existing no-sources behavior (import_docs) — assertions kept/updated.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
@@ -1,41 +0,0 @@
# Task 04 — The /git-sources.html admin page
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
The page itself: a soft-gated (anonymous → sign-in gate, exactly like `sources.html`) full-width manager with an add form, a sources list with per-row Remove, an env-fallback note, and a hint pointing at the Sync button. Never-stale buttons, inline errors, WCAG 2.1 AA basics, dark tech theme.
## Work
1. `frontend/git-sources.html` (NEW) — modeled on `frontend/sources.html` (same frame, same gate pattern, the standard full header from phase 34):
- Head: same meta/favicon/stylesheet pattern; `<title>Git sources · Brain of Reese</title>`; skip-link; the identical header block (nav [Chat, Sources, Tuning] + Tuning toggle + `#sync-btn` + New chat + auth pair — the "Git sources" nav link itself arrives in task 05, so this file lands without it for now).
- `<main id="main">` — `#steering-panel` first (phase 34 contract), then:
- **Gate** `#git-sources-gate` — the `#sources-gate` soft-gate markup pattern (sign-in card + link `/login.html?next=/git-sources.html`), visible for anonymous, hidden for admin.
- **Content** `#git-sources-content` (hidden until admin):
- `.page-head` — `<h1>Git sources</h1>` + sub: "The repositories the Sync button clones and indexes. Add or remove them here — no `.env`, no restart."
- **Env note** `#git-sources-env-note` (hidden by default; shown when the API returns `from_env: true`): "These sources currently come from `BOR_GIT_SOURCES` in `.env` — adding or removing one here switches management to the database."
- **Add form** `#git-source-form` — visible label (or visually-hidden label per the tuning-page pattern — use a visible `<label for="git-source-url">Add a git source</label>`), input `#git-source-url` (type text, `maxlength="500"`, `autocomplete="off"`, placeholder `https://github.com/you/homelab.git`, `required`), submit button `#git-source-add` ("Add source"), error line `#git-source-error` (`role="alert"`, hidden) — §7.4 never-stale: the button disables + label changes while the POST is in flight, re-enables on success/failure (the form is kept on failure, same as the tuning forms).
- **List** `#git-sources-list` (a full-width table or list per §7.1 — **no skinny single-column list**: use the Sources-page table pattern — columns: URL (mono `<code>`), Added, actions) + empty state `#git-sources-empty` ("No git sources stored yet." — and, with `from_env`, the env note already explains where the active list comes from).
- **Hint box** (`role="note"`): "Use the **Sync sources** button in the header (or on the Sources page) to clone the repos and refresh the index — removing a repository prunes its documents from the index on the next sync."
2. `frontend/assets/git-sources.js` (NEW) — the page module (loaded `type="module"`, imports `./header.js` like its siblings):
- Boot: `const admin = await initSharedHeader()` (one cached whoami) — anonymous → show the gate, stop; admin → hide the gate, `loadSources()`.
- `loadSources()` — `GET /api/git-sources` → render the table rows (`textContent` only — URLs may contain credentials; never innerHTML the URL), the added date (localized, `—` for null), the per-row Remove button (`.git-source-remove`, `aria-label="Remove git source: <url>"`), the env note's `hidden` on `from_env`, the empty state. Non-2xx → the content area shows a `role="alert"` error state with a retry (never a stuck page).
- Add submit — client-side non-empty check; disable `#git-source-add` (label "Adding…"); `POST /api/git-sources` with `{url}`; success → clear the input, re-enable (label "Add source"), `loadSources()`, focus the new row (a11y); failure → `#git-source-error` with the server detail (422 shape-aware like the tuning forms), re-enable, input kept.
- Remove click — `window.confirm("Remove this git source from the list? Its documents stay indexed until the next sync prunes them.")` — cancel → nothing; ok → disable the row button, `DELETE /api/git-sources/{id}`, `loadSources()`; failure → row error state + re-enable.
- Focus management + keyboard: all controls focus-visible (theme CSS covers it), the list rows' buttons are real `<button>`s.
3. `frontend/assets/styles.css` — the `.git-source-*` rules + gate reuse:
- The table: full-width in the 72rem container (the Sources-page table styles are a good starting point — reuse classes where they fit), mono URL cells with horizontal scroll on overflow (long URLs with credentials), rows ≥44px touch targets, `:focus-visible` 3px outline.
- Env note: an info chip in the theme palette (brand-soft `#232b52` surface, brand-ink `#a5b4fc` text ≈6.9:1); hint box: the page-sub styling family; error/alert states reuse the existing `#fca5a5`/`#2d1318` error treatment.
- Gate: reuse the `#sources-gate` styles (the page is the same shape as Sources — one gate visual language).
4. `frontend/assets/header.js` — no change (the page's controls are the standard shared ones; `initSharedHeader` already handles everything that ships in the header).
## Testing & Quality
- Frontend-only — no Python change this task; the no-CDN integration test must still pass (same-origin markup; the E2E in task 06 exercises the page).
- Coverage: `app/` gate unaffected.
- Manual smoke (dev server, signed in): add a real-looking URL → row appears; remove → confirm → row gone; invalid URL → inline 422 error, button re-enabled; signed out → gate only.
## Completion Criteria
- [ ] `/git-sources.html` served at the route; anonymous sees only the gate (list/form absent or inert); admin sees list + form + env note + hint.
- [ ] Add / remove round-trip works against the task-02 API; every in-flight state disables its control and re-enables on resolution (never stale); errors are inline `role="alert"`.
- [ ] UI Structure Check (AGENTS.md rule 5): landmarks, labeled controls, contrast ≥4.5:1, focus-visible; full-width table (no skinny list); no CDN (rule 6).
- [ ] `uv run pytest` green (no-CDN test); `uv run ruff check . && uv run pyright` clean.
@@ -1,32 +0,0 @@
# Task 05 — Admin-only "Git sources" nav link on all five pages
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
Make the new page reachable from everywhere: the admin-only **"Git sources"** nav link in all five identical headers (phase 34's contract), revealed for the admin by `header.js` — the exact phase-29 pattern.
## Work
1. The five page headers — `frontend/index.html`, `frontend/sources.html`, `frontend/document.html`, `frontend/tuning.html`, `frontend/git-sources.html` — inside `<nav class="app-nav" aria-label="Primary">`, **immediately after** the `#nav-sources` link, add (mirroring the `#nav-tuning` markup, comment citing this phase + owner permission 2026-08-26):
```html
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Git sources</a>
```
- Rules (phase-29 contract): `hidden` by default on every page (anonymous-safe ship-hidden); NO `is-active` / `aria-current` on the four pages that aren't the Git sources page.
- **Exception:** on `frontend/git-sources.html` the link carries `class="nav-link is-active"` + `aria-current="page"` (the current page, like Tuning on `tuning.html`).
- Nav order on every page becomes: Chat, Sources, **Git sources**, Tuning.
2. `frontend/assets/header.js` — next to the `navTuning` reveal block, add the same ship-hidden/reveal-for-admin contract:
```js
const navGitSources = document.querySelector("#nav-git-sources");
if (navGitSources) navGitSources.hidden = !admin;
```
Update the file-header comment (the admin-only link list now includes Git sources).
## Testing & Quality
- Frontend-only — no Python change; the no-CDN integration test is unaffected (same-origin `<a>`).
- Coverage: `app/` gate unaffected.
- Manual smoke: admin sees "Git sources" on all five pages → each navigates to `/git-sources.html` (with `is-active` there); anonymous never sees it (ships hidden, no flash).
## Completion Criteria
- [ ] All five headers contain `#nav-git-sources` after `#nav-sources`, `hidden` by default; `git-sources.html`'s carries `is-active` + `aria-current="page"`.
- [ ] `header.js` reveals it for the admin on the cached whoami (no extra request) and hides it for anonymous on every page.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,37 +0,0 @@
# Task 06 — Story E2E + nav-inventory update + docs + commit
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
Prove the story end-to-end with its dedicated Playwright suite, keep phase 34's nav-consistency contract in sync with the new link, document the env-var demotion, and close the phase with the full gate + one commit.
## Work
1. `tests/e2e/test_git_sources_admin.py` (NEW — the story gate, run in isolation). Fixtures: the standard E2E app + DB (`tests/e2e/conftest.py`); admin session via `tests/e2e/auth_helpers.py`; **no git, no network** — this suite is UI + API only (the clone/import path is mocked at the integration level, task 03).
- **Anonymous:** `/git-sources.html` shows the sign-in gate (list + add form absent/inert); `#nav-git-sources` hidden on the five pages; the API 403s (assert via the page context's `fetch` or `context.request` — follow `test_admin_auth.py`'s pattern for anonymous API assertions).
- **Admin — nav:** `#nav-git-sources` visible on all five pages; clicking it from `/` lands on `/git-sources.html` with the link `is-active`.
- **Admin — list:** seed two rows via the API (or `SessionLocal`) before load → both rows render (mono URL text, added date); the env note is hidden (DB rows exist).
- **Admin — add:** submit `https://example.com/reese/new-repo.git` → the row appears, the input clears, the button re-enables (never stale); submit a duplicate → inline `role="alert"` error, no new row, button re-enabled; submit `not a valid url` → inline 422 error, button re-enabled.
- **Admin — remove:** click a row's Remove → accept the confirm dialog (Playwright `page.on("dialog")`) → the row disappears; cancel a second removal → the row stays.
- **Admin — env fallback:** truncate `git_sources`, set the E2E app's `BOR_GIT_SOURCES` (follow how `test_sync_button.py` controls the env on the app fixture), reload → the env rows render + `#git-sources-env-note` visible.
- The sync-origin behavior (DB over env in the pipeline) is integration-level (task 03) — do not trigger a real sync in this suite.
2. `tests/e2e/test_nav_consistency.py` (phase 34 — UPDATE): the admin nav inventory now includes **"Git sources"** (four links, order Chat, Sources, Git sources, Tuning); the anonymous hidden set gains `#nav-git-sources`; the per-page inventory comparison stays order-sensitive.
3. **Docs:**
- `.env.example` — the `BOR_GIT_SOURCES` comment: now the **empty-table fallback**; the primary management UI is the admin Git sources page (phase 35).
- `README.md` — the import/update workflow section: the git-sources list is managed on the admin page (stored in Postgres); `BOR_GIT_SOURCES` only applies while that list is empty; `--source` still overrides for manual runs.
4. **Regression pass — each in isolation** (`uv run pytest tests/e2e/<file>.py -v --no-cov`): `test_git_sources_admin.py` (new), `test_nav_consistency.py` (updated), `test_sync_button.py`, `test_shared_header.py`, `test_header_consistency.py`, `test_tuning_nav_link.py`, `test_smoke.py`.
5. Full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (>90%), `uv run ruff check . && uv run pyright`.
6. **UI Structure Check** (AGENTS.md rule 5) on the new page (full-width table, labels, contrast, focus-visible, aria-live on the list updates) + no CDN (rule 6).
7. **Commit** (A17): stage this phase's files (`app/**`, `alembic/**`, `frontend/**`, `tests/**`, `README.md`, `.env.example`), message `feat(sources): admin page to add and remove git sources (TODO.md L4)`, always `--no-gpg-sign`. Move `.agent/phases/todo/35_git_sources_admin/` to `.agent/phases/complete/`.
## Testing & Quality
- E2E: `tests/e2e/test_git_sources_admin.py` green **in isolation** (A16: one story, one file).
- Unit/integration: from tasks 01–03 — all green under `uv run pytest`.
- Coverage: **>90%** on `app/` (new API module + resolver fully covered).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov` green in isolation.
- [ ] `test_nav_consistency.py` updated for the fourth nav link and green; every suite in the task 06 regression list green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] README + `.env.example` document the fallback semantics.
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
@@ -1,40 +0,0 @@
# Phase 36 — Document Summary Shown Together With the Original
**Source:** `TODO.md` L5 — "When I click on a document with a summary I should be able to see the summary and the original document together."
**Story:** `.agent/user_stories/summary-in-viewer.md`
**Context:** Phase 30 stores a lite-model summary on `documents.summary` for every non-markdown document (plus an indexed `is_summary` chunk) — but the viewer never shows it: `GET /api/documents/content` omits the field and the shared renderer `renderDocument` (`frontend/assets/document.js`, used by BOTH the full-page viewer `document.html` and the chat/sources modal, phase 26) only renders the raw content. Markdown documents carry no summary (phase 30) and must render exactly as before.
## Objective
When a document **has** a summary, show it and the original content together — a labeled Summary panel above the content, on both viewer surfaces at once (shared renderer); documents without a summary are unchanged.
## Dependencies
- `30_document_summaries` (complete) — the `documents.summary` column (migration 0004), the summarizer, the `summary_kb` E2E fixture + the deterministic mock `SUMMARY_MODE` digest.
- `10_story_document_viewer` + `26_document_modal_viewer` (complete) — the viewer page, the modal, and the shared `renderDocument(doc, {titleEl, metaEl, contentEl})` core both surfaces render through.
- `16_admin_auth` (complete) — the soft rule this phase must not touch: the content endpoint stays public + stateless (catalog gated, viewer public).
## Tasks
1. `01_content_api_summary_field.md` — `DocContent.summary` + the endpoint returns it; integration tests.
2. `02_viewer_summary_panel.md` — the shared renderer draws the `.doc-summary` panel (both surfaces) + theme-matched styles.
3. `03_e2e_and_regression.md` — the story E2E suite `test_summary_in_viewer.py`; regressions; full gate; commit.
## Testing & Quality
- Unit/integration: the content endpoint returns the summary for a summarized non-markdown doc and `null` for a markdown doc; anonymous access unchanged.
- Coverage: **>90%** on `app/` (the touched endpoint stays covered).
- E2E (mandatory, A16): `tests/e2e/test_summary_in_viewer.py` — the story gate, run in isolation.
## Completion Criteria
- [ ] `GET /api/documents/content` returns `summary` (string or null); no auth/shape change beyond the added nullable field; the endpoint is still public.
- [ ] A summarized document shows the labeled Summary panel **above** the original content in the full-page viewer AND the modal; the original content (including content the summary digest doesn't contain) is fully visible.
- [ ] A markdown document (no summary) renders exactly as before on both surfaces — no empty panel.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run pytest tests/e2e/test_summary_in_viewer.py -v --no-cov` green in isolation; regressions (task 03 list) green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5): the panel is a labeled section, contrast ≥4.5:1, no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
## Locked decisions
- **A7 / A15 untouched** — retrieval, context assembly, and the SSE contract are unchanged; this is a display + API-field phase.
- **Phase 16 soft rule untouched** — `GET /api/documents/content` stays public + stateless (anyone who can open a document sees its summary; the catalog stays admin-gated).
- **Phase 30 untouched** — summaries are still generated at import, still markdown-excluded, still fail-soft (NULL possible); this phase only surfaces the existing field.
- **Shared-renderer principle (phase 26)** — the panel is drawn in `renderDocument`, so the page and the modal can never drift.
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages; summary text rendered with `textContent` (XSS contract unchanged).
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
@@ -1,32 +0,0 @@
# Task 01 — The content endpoint returns the summary
**Phase:** `36_summary_in_viewer` · **Source:** `TODO.md:5 — "When I click on a document with a summary I should be able to see the summary and the original document together."`
**Story:** `.agent/user_stories/summary-in-viewer.md`
## Objective
Surface the existing `documents.summary` field (phase 30) on the viewer's data contract: `DocContent` gains a nullable `summary` and `GET /api/documents/content` returns it — endpoint stays public, stateless, and otherwise byte-identical.
## Work
1. `app/schemas.py` — `DocContent` gains:
```python
#: Lite-model summary (phase 30) — non-markdown A9 docs only; None for
#: markdown documents, pre-phase-30 rows, and the fail-soft path where
#: summary generation failed but the document was still indexed.
summary: str | None = None
```
(Place it after `format` / before `content`, with the docstring mirroring `Document.summary`'s.)
2. `app/api/docs.py` — `get_document_content` returns `summary=doc.summary` in the `DocContent(...)` construction. Nothing else changes (no auth, no query change — `Document` is already selected in full).
3. Integration tests — extend the existing `/api/documents/content` test module (find it in `tests/integration/` — the phase-10 content-endpoint tests):
- A non-markdown document row seeded with `summary="…"` → response JSON carries `summary` verbatim.
- A markdown document row with `summary=None` → `"summary": null`.
- Anonymous (no admin cookie) still gets 200 (the phase-16 soft rule — public viewer) for both.
- The existing assertions (404 on unknown pair, content/format fields) stay green unmodified.
## Testing & Quality
- Integration: the cases above; `uv run pytest` green overall.
- Coverage: **>90%** on `app/` — the touched endpoint stays covered (the new field is exercised by the new assertions).
## Completion Criteria
- [ ] `GET /api/documents/content?source=…&path=…` returns the summary for a summarized doc and `null` for a markdown doc; anonymous access unchanged (200).
- [ ] No other field, status code, or auth behavior of the endpoint changed.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
@@ -1,44 +0,0 @@
# Task 02 — The shared renderer draws the Summary panel (both surfaces)
**Phase:** `36_summary_in_viewer` · **Source:** `TODO.md:5 — "When I click on a document with a summary I should be able to see the summary and the original document together."`
**Story:** `.agent/user_stories/summary-in-viewer.md`
## Objective
`renderDocument` — the single rendering core both the full-page viewer (`document.html`) and the chat/sources modal (`document-modal.js`) go through (phase 26) — draws a labeled Summary panel above the original content whenever `doc.summary` is non-empty; `null`/empty renders nothing, so markdown documents and fail-soft rows are byte-for-byte unchanged.
## Work
1. `frontend/assets/document.js` — in `renderDocument(doc, { titleEl, metaEl, contentEl })`, **after** the meta row is populated and **before** the content is built, append the summary section to `contentEl` (which is then filled with the usual `.doc-md` / `<pre class="doc-raw">` content after it):
```js
if (doc.summary && doc.summary.trim() !== "") {
const section = document.createElement("section");
section.className = "doc-summary";
section.setAttribute("aria-label", "Summary");
const title = document.createElement("h2");
title.className = "doc-summary-title";
title.textContent = "Summary";
const body = document.createElement("p");
body.className = "doc-summary-text";
body.textContent = doc.summary; // text node — XSS contract unchanged
section.append(title, body);
contentEl.appendChild(section);
}
```
- The meta badge row (`#doc-meta` / `#doc-modal-meta`) is untouched; the panel sits between meta and content on **both** surfaces because both call this one function.
- Update the file-header comment (the renderer now also owns the optional summary panel) and the `renderDocument` doc comment.
2. `frontend/assets/styles.css` — `.doc-summary` (dark tech theme, phase-08 palette):
- A clearly-distinct "summary, not content" look: surface `#121a2e` with a 3px brand left border (`#6d78f2`) or a brand-soft (`#232b52`) header strip — pick one and keep it simple; `border-radius` matching the existing content cards; padding ~1rem; `margin-bottom` separating it from the content.
- `.doc-summary-title` — small-caps/label treatment: `#a5b4fc` (brand-ink on the brand-soft chip, ≈6.9:1) or `#a5b4fc` on surface (verify ≥4.5:1 — if short, use the chip).
- `.doc-summary-text` — `var(--ink)` (`#e8ebf4`) on the surface (≈14.5:1); wraps inside the same width the content uses (the ≤46rem centered column for md docs, the raw-content width otherwise — the panel is a child of `contentEl`, so it inherits the column; verify for the `<pre class="doc-raw">` case where the content is wider).
- No animation (nothing for `prefers-reduced-motion` to still); the section is static content — no focusability needed (it carries `aria-label` + heading).
3. Do NOT touch `document-modal.js` (it calls `renderDocument` — the panel comes for free), `markdown.js`, or the page scripts.
## Testing & Quality
- Frontend-only — no Python change; the no-CDN integration test is unaffected.
- Coverage: `app/` gate unaffected.
- Manual smoke (dev server with an imported non-md doc, e.g. a yaml from the fixture KB via the importer + mock, or a hand-seeded `documents.summary` row): modal from the Sources table AND the full page both show panel + content; a markdown doc shows no panel.
## Completion Criteria
- [ ] A document with a summary shows the labeled panel above the original content in **both** the modal and the full-page viewer; the original content is fully visible (nothing hidden/collapsed).
- [ ] A document without a summary (`null` or empty/whitespace) renders exactly as before on both surfaces — no panel, no empty box.
- [ ] The summary text is written with `textContent` (XSS contract); the meta row is unchanged.
- [ ] `uv run pytest` green (no-CDN test); `uv run ruff check . && uv run pyright` clean.
@@ -1,31 +0,0 @@
# Task 03 — Story E2E + regression pass + commit
**Phase:** `36_summary_in_viewer` · **Source:** `TODO.md:5 — "When I click on a document with a summary I should be able to see the summary and the original document together."`
**Story:** `.agent/user_stories/summary-in-viewer.md`
## Objective
Prove the story end-to-end with its dedicated Playwright suite — summary panel + original content visible together on both surfaces, no panel for markdown docs — then the full gate + one commit.
## Work
1. `tests/e2e/test_summary_in_viewer.py` (NEW — the story gate, run in isolation). Reuse `tests/e2e/test_document_summaries.py`'s machinery as closely as possible (its fixture KB + import helpers are the reference):
- Fixtures: the standard E2E app + DB + the deterministic mock LLM (`tests/e2e/mock_llm.py` — its `SUMMARY_MODE` answer is a byte-stable 24-token digest); import `tests/fixtures/summary_kb/` so `quadlet/qwen-llamacpp.yaml` gets its stored summary + `notes/qwen-llamacpp-notes.md` stays summary-less (the existing tail sentinel `RESE-SUMMARY-SENTINEL-7f3a` sits on the yaml's LAST line — **outside** the 24-token digest, so it is a marker for "the original, not the summary").
- **Full page:** open `/document.html?source=summary_kb&path=quadlet%2Fqwen-llamacpp.yaml` → `.doc-summary` visible with the deterministic digest text AND the original content visible with the sentinel (`RESE-SUMMARY-SENTINEL-7f3a` present in the rendered content) — summary and original **together**.
- **Modal:** from the Sources table (admin session via `tests/e2e/auth_helpers.py`), click the yaml's row → the modal shows the same panel + content (sentinel present, digest present); then "Full page" still lands on the dedicated page with the panel (the two surfaces agree).
- **No-summary control:** the markdown doc (`notes/qwen-llamacpp-notes.md`) → no `.doc-summary` element on the full page and in the modal; the content renders as before.
- **API shape (cheap, via the page context's `fetch` or `context.request`):** `GET /api/documents/content` for the yaml carries `summary` (string), for the md doc `null`; anonymous fetch → 200 (soft rule unchanged).
- The E2E must not depend on a real LLM (the mock's digest is deterministic — the same pattern `test_document_summaries.py` relies on).
2. **Regression pass — each in isolation** (`uv run pytest tests/e2e/<file>.py -v --no-cov`): `test_summary_in_viewer.py` (new), `test_document_viewer.py`, `test_document_summaries.py`, `test_document_back_navigation.py`, `test_chat_rag.py` (the source-chip modal path), `test_smoke.py`.
3. Full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (>90%), `uv run ruff check . && uv run pyright`.
4. **UI Structure Check** (AGENTS.md rule 5): the panel is a labeled section (`aria-label` + heading), theme contrast ≥4.5:1, it does not break the centered 46rem chat-column-width content layout, no CDN (rule 6 — the no-CDN integration test covers it).
5. **Commit** (A17): stage this phase's files (`app/schemas.py`, `app/api/docs.py`, `frontend/assets/document.js`, `frontend/assets/styles.css`, `tests/**`), message `feat(viewer): show document summary together with the original (TODO.md L5)`, always `--no-gpg-sign`. Move `.agent/phases/todo/36_summary_in_viewer/` to `.agent/phases/complete/`.
## Testing & Quality
- E2E: `tests/e2e/test_summary_in_viewer.py` green **in isolation** (A16: one story, one file).
- Unit/integration: from task 01 — all green under `uv run pytest`.
- Coverage: **>90%** on `app/` (the endpoint change is covered).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_summary_in_viewer.py -v --no-cov` green in isolation.
- [ ] Every suite in the task 03 regression list green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
@@ -1,24 +0,0 @@
# Task 01 — Live tool-calling probe
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "The agent should be able to list the available sources as a tool and the read the ones it thinks are relevant"`
**Story:** `.agent/user_stories/agent-document-tools.md`
## Objective
Verify against the live aipi endpoint whether the `turbo` chat model supports OpenAI-style `tools` + streaming `tool_calls` before building the loop — the phase-17 "verified live" convention — and record which path the phase takes.
## Work
1. `scripts/llm_probe.py` — add a `--tools` flag: when set, send (a) a non-streaming `chat/completions` request with one trivial function (e.g. `get_time`, no parameters) and a user message that makes calling it natural, and (b) the same with `stream=True`; for each, print whether `finish_reason` is `tool_calls`, the parsed `function.name`/`arguments`, and (streaming) whether the calls arrive as `delta.tool_calls` chunks with `index`/`id`/partial `function.arguments` (the OpenAI wire convention). Reuse the existing env reading (`BOR_LLM_BASE_URL` / API key / `BOR_LLM_CHAT_MODEL`) and the script's existing output style.
2. Run it against the live endpoint (`uv run python -m scripts.llm_probe --tools`) and classify the verdict:
- **supported** → the phase uses OpenAI `tools`/`tool_calls` (tasks 02–03 as written).
- **not supported** → the phase uses the prompt-based structured-call fallback (task 03 documents it): the model is instructed to emit a single JSON block (`{"tool": "list_documents"}` / `{"tool": "read_document", "source": …, "path": …}`) before answering; `run_agent` parses it out of the content stream; the SSE `tool` contract and the budgets are identical.
- **intermittent** → treat as not supported (fail-loud house style) and note it.
3. Record the verdict + date where it will be read later: the task-03 `app/rag/agent.py` module docstring (task 03 writes it) and the phase commit message (task 06) — e.g. `probe: turbo tool_calls=streaming-ok 2026-08-26`.
## Testing & Quality
- The probe is a CLI script (no `app/` coverage impact). If the parsing of the probe response is factored into a function, add a small unit test for it.
- `uv run pytest` green (no regressions); ruff + pyright clean.
## Completion Criteria
- [ ] `uv run python -m scripts.llm_probe --tools` runs and prints a clear supported / not-supported verdict for both the non-streaming and the streaming request.
- [ ] The verdict (with date) is available for task 03's docstring and task 06's commit message.
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,36 +0,0 @@
# Task 02 — LLM client: `tools` + tool-call streaming
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "…read the ones it thinks are relevant… These values should be configured by environment variables"` (the client plumbing the env-tuned loop runs on)
**Story:** `.agent/user_stories/agent-document-tools.md`
## Objective
Teach `LLMClient.chat_stream` to pass an OpenAI `tools` list and to accumulate streaming `tool_calls` deltas into typed pieces — with `tools=None` producing a byte-identical request to today.
## Work
1. `app/rag/llm.py`
- New frozen dataclass next to `StreamPiece`:
```python
@dataclass(frozen=True)
class ToolCallPiece:
"""One model-requested tool call accumulated from stream deltas (phase 37)."""
id: str # the model's tool_call id; synthesized "call_<index>" when absent
name: str # "list_documents" | "read_document" (whatever AGENT_TOOLS names)
arguments: dict[str, Any]
```
- `chat_stream(self, messages, tools: list[dict[str, Any]] | None = None)`:
- When *tools* is not None, pass `tools=tools` to `chat.completions.create`; when None, do **not** include the key (byte-identical request to today).
- In the chunk loop, accumulate `delta.tool_calls` (a list of partials keyed by `index`; `id` and `function.name` arrive on the first partial for an index, `function.arguments` arrives in fragments to concatenate).
- At stream end (or when a chunk carries `finish_reason == "tool_calls"`), for each accumulated call **in index order** yield `ToolCallPiece(id, name, json.loads(arguments) or {})`.
- Malformed `arguments` JSON → raise `LLMError` (fail-loud house style — a silently dropped tool call would corrupt the loop).
- Return annotation becomes `AsyncIterator[StreamPiece | ToolCallPiece]`; update the docstring (wire convention + a pointer to the task-01 probe verdict).
- **If the task-01 verdict is "not supported"** (prompt-based fallback): skip the `tools` parameter and the delta accumulation entirely; keep `StreamPiece` unchanged; the JSON-block parse helper lands in `app/rag/agent.py` (task 03) instead.
2. `app/api/chat.py` — update the phase-17 typing import line (`StreamPiece # noqa: F401`) to also import `ToolCallPiece` so pyright sees the union; the dispatch wiring itself is task 04.
## Testing & Quality
- Unit (`tests/unit/` — extend the existing `chat_stream` tests): synthetic chunk sequences — (a) a content-only stream is unchanged (no `ToolCallPiece`; the captured `create()` kwargs have no `tools` key); (b) a tool-call stream with partials across chunks (name on the first, arguments in 2–3 fragments) → one `ToolCallPiece` with the merged JSON; (c) two calls in one stream (indices 0 and 1) → both, in index order; (d) malformed arguments JSON → `LLMError`; (e) `tools=[…]` present in the request when passed.
- Coverage: **>90%** on the modified module.
## Completion Criteria
- [ ] `chat_stream(messages)` (no tools) — all existing unit tests green unchanged.
- [ ] The new tool-call accumulation tests green; full `uv run pytest` green.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,50 +0,0 @@
# Task 03 — The basic agent loop (`app/rag/agent.py`)
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "it gets one opportunity to list documents and then one opportunity to add exactly one extra document to its context before being required to answer. These values should be configured by environment variables."`
**Story:** `.agent/user_stories/agent-document-tools.md`
## Objective
A testable agent loop: the model gets the two tools while budgets last, the app executes them against Postgres, and once both budgets are spent the tools are dropped so the model is required to answer.
## Work
1. `app/config.py` — two settings (documented, `BOR_` prefix per house style, near the RAG-tuning block):
```python
#: Per-turn opportunities to call the `list_documents` agent tool (phase 37);
#: 0 disables the tool entirely (pre-phase behavior with both at 0).
agent_list_calls: int = 1
#: Per-turn opportunities to call `read_document` (phase 37); 0 disables.
agent_read_calls: int = 1
```
2. `app/rag/agent.py` — new module. The module docstring carries: the phase, the loop contract, and the task-01 probe verdict + date (which path is in use — OpenAI tool_calls vs the prompt-based fallback).
- `AGENT_TOOLS`: the two OpenAI function definitions — `list_documents` (no parameters; description: "List every document indexed in the knowledge base, one `source/path — title` line each") and `read_document` (`source` + `path` required; description: "Add the full content of exactly one more indexed document to your context").
- DB accessors (module-level functions so unit tests can monkeypatch them):
- `list_catalog(db) -> list[tuple[str, str, str]]` — `select(Document.source, Document.path, Document.title).order_by(Document.source, Document.path)` (same order as `GET /api/docs`).
- `find_document(db, source, path) -> Document | None`.
- `@dataclass AgentHolder: read_docs: list[Document] = field(default_factory=list); tool_calls: int = 0` — the API layer (task 04) reads it after the stream.
- `async def run_agent(llm, db, *, system_prompt, user_message, seed_docs, settings, holder) -> AsyncIterator[StreamPiece | ToolCallPiece]`:
1. `messages = [{"role": "system", "content": system_prompt}, {"role": "user", "content": user_message}]`; `list_left = settings.agent_list_calls`; `read_left = settings.agent_read_calls`; `tools = AGENT_TOOLS if (list_left or read_left) else None`; `rounds = 0`; cap `max_rounds = 2 + settings.agent_list_calls + settings.agent_read_calls` (belt-and-braces — every tool round must consume a budget, so the cap only catches pathological streams).
2. Loop: stream `llm.chat_stream(messages, tools=tools)`; yield every piece as it arrives. Collect any `ToolCallPiece` (handle the first; if a stream yields both content and a tool call — rare — the content stays (it was already emitted) and the tool still runs).
- **No tool call** → return (the answer was streamed).
- **`list_documents`**: `list_left > 0` → `list_left -= 1`, `holder.tool_calls += 1`, result = `f"{n} documents:\n" + "\n".join(f"{s}/{p} — {t}")` (uncapped in v1 — **ASSUMPTION: the catalog is not truncated; the UI never shows it, only the model does**). Else result = `"No listing budget left — answer with what you have."` (no budget consumed).
- **`read_document`**: arguments must carry `source` and `path`. If that document is already in `seed_docs` or `holder.read_docs` → result = `"Already in your context."` (no budget consumed, no append). Else if `read_left > 0` → `doc = find_document(db, source, path)`; found → `read_left -= 1`, `holder.tool_calls += 1`, `holder.read_docs.append(doc)`, result = `f"Document {source}/{path}:\n{doc.content}"` (**full text, never truncated — the A7-revised contract**); not found → result = `f"No document at {source}/{path} — check the list_documents output."` (no budget consumed). `read_left == 0` → result = `"No reading budget left — answer with what you have."`.
- **Unknown tool name** → result = `"Unknown tool."` (no budget consumed).
- Append to `messages`: the assistant tool-call message (`{"role": "assistant", "content": None, "tool_calls": [{"id": tc.id, "type": "function", "function": {"name": tc.name, "arguments": json.dumps(tc.arguments)}}]}`) + the tool result (`{"role": "tool", "tool_call_id": tc.id, "content": result}`); then `tools = None if (list_left == 0 and read_left == 0) else AGENT_TOOLS` (once both budgets are spent, the next request must be answered).
- `rounds += 1`; if `rounds >= max_rounds` → force one final `chat_stream(messages, tools=None)` (yield its pieces) and return.
- **Prompt-based fallback (only if the task-01 verdict is "not supported")**: instead of `ToolCallPiece`s, scan each streamed content turn for a leading JSON block matching `{"tool": …}` (strip leading whitespace, parse with `json`; the block must be the first non-whitespace content of the turn). On a match: strip the block from the emitted stream (re-emit only trailing content, if any), then run the identical budget/result machinery. The SSE `tool` events and the budgets are unchanged either way.
3. `app/rag/prompts.py` — add the `<tools>` instructions section to the **HIGH prompt only** (a new constant, appended after the mode body; the LOW/deflection prompt stays byte-identical). Wording (tune against the live model if needed, but keep it stable — the E2E mock keys off the `<tools>` marker's *presence*, not the wording):
> If the documents in your context reference other files, or you need content that is not included above, call `list_documents` to see what is indexed, then `read_document` to pull in exactly one more document. Answer as soon as you have what you need — do not read more than one extra document.
## Testing & Quality
- Unit (`tests/unit/test_agent.py`, new): a scripted fake LLM (canned stream sequences) + monkeypatched `list_catalog`/`find_document` + a `Settings` with budgets set:
- list → read → answer: event order (tool pieces before content), `holder.read_docs == [doc]`, `holder.tool_calls == 2`, the request after budgets are spent carries `tools=None`, the follow-up request contains the assistant tool-call + tool-result messages.
- Budgets: `read_left` exhausted → a second `read_document` gets "No reading budget left" and appends nothing; `agent_list_calls=0` + `agent_read_calls=0` → exactly one request with `tools=None` (byte-identical single-call path).
- Edge: reading a doc already in `seed_docs` → "Already in your context." (no budget consumed); unknown path → "No document at …"; unknown tool name → "Unknown tool."; the round cap forces a final no-tools answer.
- The HIGH prompt gains the `<tools>` section; the LOW prompt is byte-identical to pre-phase (assert against the existing prompt-test fixtures).
- Integration (`tests/integration/`): `list_catalog` ordering + `find_document` hit/None against real Postgres (follow the existing DB-test patterns).
- Coverage: **>90%** on the new module.
## Completion Criteria
- [ ] `app/rag/agent.py` exists with the loop above; the module docstring records the probe verdict + date (task 01).
- [ ] The budget matrix (0/1) unit tests green, including "budgets 0/0 ⇒ one request, no tools".
- [ ] HIGH prompt carries `<tools>`; LOW prompt byte-identical to pre-phase.
- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.
@@ -1,36 +0,0 @@
# Task 04 — `POST /api/chat`: `tool` SSE events + agent wiring
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "This will require some reconfiguring of the UI since it will now need to show 'calling tool' in addition to 'thinking' and it will need a basic agent loop."`
**Story:** `.agent/user_stories/agent-document-tools.md`
## Objective
Wire the loop into the chat endpoint: the new SSE `tool` event, the agent on grounded turns only, and the read document reflected in `done.sources`, `query_log`, and the per-turn log line.
## Work
1. `app/schemas.py` — add next to `ChatThinkingEvent`:
```python
class ChatToolEvent(BaseModel):
"""SSE frame for one agent tool call (phase 37, PLAN §4 extension)."""
type: Literal["tool"] = "tool"
name: str # "list_documents" | "read_document"
argument: str | None = None # "source/path" for read_document
```
2. `app/api/chat.py`
- In `stream()`, after `plan_turn`: when `not plan.deflected` → run `run_agent(llm, db, system_prompt=plan.system_prompt, user_message=request.message, seed_docs=plan.docs, settings=settings, holder=holder)`; when deflected → keep the current direct `chat_stream` (A8 byte-identical).
- Event mapping: `StreamPiece` thinking/delta exactly as today (including the `BOR_STREAM_THINKING` kill-switch and the `thinking_chars` count); `ToolCallPiece` →
`yield sse_event(ChatToolEvent(name=tc.name, argument=f"{tc.arguments.get('source')}/{tc.arguments.get('path')}" if tc.name == "read_document" else None).model_dump())`.
- `done` event: `sources` = `plan.docs + holder.read_docs` deduped by `(source, path)`, order preserved; the `ChatDoneEvent` shape otherwise unchanged.
- `query_log.sources`: the same combined list (replaces the current `source_paths` build).
- Per-turn log line: add `tool_calls={holder.tool_calls}` after `thinking_chars=` (PLAN §9 required-line extension — the phase 17/30/31 precedent).
- Extend the module docstring with the phase-37 section (the flow, the grounded-only scope, the budgets, the "both budgets 0 ⇒ pre-phase behavior" note).
3. `.agent/PLAN.md` — record two revision notes in the project's established format (phase 17/24 precedent, "owner permission 2026-08-26"): the §4 SSE contract gains `{"type":"tool","name":…,"argument":…}` (client rule: render as a "calling tool" line/state; `delta`/`done` unchanged) and the §9 per-turn log line gains `tool_calls=N`.
## Testing & Quality
- Integration (`tests/integration/test_chat*.py` — extend the existing SSE-contract suite, mock LLM that emits tool calls): (a) grounded turn with tool calls → event sequence `thinking?/tool/tool/delta…/done`, `done.sources` includes the read doc, the `query_log` row's sources match, the log line carries `tool_calls=2`; (b) deflected turn → no `tool` events, the sequence byte-identical to today; (c) budgets 0/0 → no `tool` events, the mock received no `tools` parameter, single-request path.
- Coverage: **>90%** on the modified modules.
## Completion Criteria
- [ ] `POST /api/chat` (mock LLM, grounded) streams `tool` frames ahead of the `delta` frames; the deflected path is unchanged (existing suites green).
- [ ] `done.sources`/`query_log` include the read document (deduped); the log line carries `tool_calls=N`.
- [ ] The PLAN.md §4 + §9 revision notes exist (dated, owner permission 2026-08-26).
- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.
@@ -1,27 +0,0 @@
# Task 05 — UI: the "calling tool" state
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "…it will now need to show 'calling tool' in addition to 'thinking'"`
**Story:** `.agent/user_stories/agent-document-tools.md`
## Objective
The chat shell shows a "calling tool" state (button + status label) and a visible tool line in the bubble, distinct from the Thinking scratchpad; tool lines persist with the chat record (phase 14) and re-render after reload.
## Work
1. `frontend/assets/app.js` — in the turn handler where the SSE frames are dispatched (the same switch that handles `thinking` and `delta`):
- New `tool` branch:
- **Status/label:** keep `uiState = thinking` (the button stays disabled — never stale, PLAN §7.4) and set the typing-indicator label to "Brain of Reese is listing documents" (`name: "list_documents"`) / "Brain of Reese is reading <argument>" (`name: "read_document"`); the elapsed-seconds hint (`thinkingClock` aria-label) keeps working through tool frames.
- **Bubble line:** append a tool line to the same wrap the thinking block uses (above the answer, beside/below the Thinking summary): `<span class="tool-call">🔎 Listing documents</span>` / `<span class="tool-call">📄 Reading <code>source/path</code></span>` — a real visible line with its own icon + color, distinct from the Thinking block (styles step). Multiple calls append multiple lines, in order.
- Tolerate tool frames interleaved with thinking frames (append-only, the same rule as thinking); a tool frame after the first `delta` (should not happen in v1 — the loop completes before the answer stream) still appends rather than crashes.
- **Persistence (phase 14 convention):** the saved record gains an optional `tools: [{name, argument}]` array next to `thinking`; re-render the tool lines when a record is loaded (same code path as the thinking re-render).
- Keep the new label strings as plain literals — phase 39 centralizes brand strings; do **not** introduce a brand helper here.
2. `frontend/assets/styles.css` — `.tool-call` style: inline row, icon + text, an accent color distinguishable from the thinking block's, `code` styling for the path, contrast ≥ 4.5:1 in both themes; no layout shift on append (the centered 46rem chat column is untouched — no new container).
3. UI Structure Check (AGENTS.md rule 5) before finalizing: semantic landmarks unchanged; the tool lines live inside the `#messages` `aria-live="polite"` region (announced to screen readers); focus-visible unaffected (the lines are not interactive); no new top-level landmarks.
## Testing & Quality
- No new Python logic — gated by the story E2E (task 06) plus the existing frontend-adjacent suites staying green.
- No CDN (AGENTS.md rule 6): no new `<script>`/`<link>` tags.
## Completion Criteria
- [ ] During a tool call the button label shows the "calling tool" text (not a stale "Thinking…") and the tool lines render above the answer.
- [ ] After a page reload, the persisted record re-renders the tool line(s).
- [ ] `uv run pytest` green (no regressions); `uv run ruff check . && uv run pyright` clean.
@@ -1,24 +0,0 @@
# Task 01 — `git_sources.kind` + `path` (migration 0007)
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
**Story:** `.agent/user_stories/local-directory-sources.md`
## Objective
Extend the phase-35 `git_sources` table with a source-kind discriminator (`git` | `local`) and an optional local path, via a reversible migration — the foundation for the API, pipeline, and page tasks.
## Work
1. `alembic/versions/0007_git_sources_kind.py` — read `alembic/versions/0006_git_sources.py` first and chain from its **actual** `revision` id (filenames are not revision ids — the phase-35 task-01 convention):
- `upgrade()`: `ALTER TABLE git_sources ADD COLUMN kind TEXT NOT NULL DEFAULT 'git'` + `ADD CONSTRAINT ck_git_sources_kind CHECK (kind IN ('git', 'local'))`; `ALTER TABLE git_sources ADD COLUMN path TEXT`; a unique index on `path` — a partial unique index (`WHERE path IS NOT NULL`) where the 0006 style allows it, otherwise a plain unique index (Postgres treats NULLs as distinct, and the API enforces local-only paths anyway — mirror whatever 0006 chose for `url`).
- `downgrade()`: drop the index, constraint, and columns in reverse.
2. `app/models.py` — extend the phase-35 `GitSource` model: `kind: Mapped[str]` (default `"git"`) + `path: Mapped[str | None]` (nullable, unique) and update the docstring (phase 38: kind discriminator; local rows carry `path`, git rows keep `url`).
3. Apply to the dev database: `podman compose up -d db` (if needed) then `uv run alembic upgrade head`; verify the columns + constraint + that existing rows read as `kind='git'`, `path=NULL`.
4. Migration test — follow the 0004/0005/0006 pattern in `tests/integration/`: up adds the columns/constraint/index (existing rows keep `kind='git'`), down drops them (round-trip on the test DB).
## Testing & Quality
- Integration: the up/down test above; full `uv run pytest` green.
- Coverage: model + migration only — the `app/` gate stays >90%.
## Completion Criteria
- [ ] `alembic/versions/0007_git_sources_kind.py` chains off 0006's real revision id; up/down reversible.
- [ ] `uv run alembic upgrade head` applies cleanly; existing rows read `kind='git'`, `path=NULL`.
- [ ] The 0007 up/down integration test passes; full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,27 +0,0 @@
# Task 02 — The admin API: the local kind
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
**Story:** `.agent/user_stories/local-directory-sources.md`
## Objective
Extend the phase-35 git-sources CRUD to accept and return `kind=local` rows with fail-loud path validation, keeping the git contract byte-identical.
## Work
1. The phase-35 API module (the file behind `/api/git-sources` — identify it in the repo; phase 35 created it):
- `POST /api/git-sources` body: `{kind?: "git"|"local" (default "git"), url?, path?}`:
- `kind=git` → exactly today's `url` validation (trimmed, 1–500 chars, the `https?://` / `ssh://` / `git@` shape, 409 on duplicate without echoing the URL).
- `kind=local` → `path` required: trimmed; `Path(p).expanduser()`; must be **absolute after expansion** and an **existing directory on the server** → else `422 {detail: "local source path is not a directory: <path>"}` (a missing path is a user error — fail loud at add-time so the owner sees it immediately; the path is not a secret, so echo it); 409 on a duplicate `path` (detail may name the path).
- Wrong field combinations (git without url, local without path, both kinds' fields) → 422.
- `GET /api/git-sources` rows gain `kind` + `path` (git rows: `path: null`; the env-fallback rows report `kind: "git"`; `from_env: true` semantics unchanged — env rows are git-only).
- `DELETE /api/git-sources/{id}` — unchanged (removal prunes on the next sync, as today).
2. The phase-35 request/response models (wherever they live — `app/schemas.py` or the module) — extend for the new fields; keep the OpenAPI docs accurate.
3. Integration tests (extend `tests/integration/test_git_sources_api.py`): anonymous → 403 on all routes (regression); `kind=local` + an existing temp dir → 201 with the stored row (`kind=local`, `path` stored expanded); a relative path → 422; a missing path → 422 naming the path; a duplicate path → 409; a git row still validates exactly as before (regression); GET mixes kinds in added order.
## Testing & Quality
- Integration: the matrix above; the existing git-kind suite green unchanged.
- Coverage: **>90%** on the modified module.
## Completion Criteria
- [ ] The local-kind POST matrix (201 / 422 / 409) green; the git contract unchanged.
- [ ] GET rows carry `kind` + `path`; the env fallback stays git-only.
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,30 +0,0 @@
# Task 03 — Sync + `import_docs`: git and local together
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
**Story:** `.agent/user_stories/local-directory-sources.md`
## Objective
The canonical mirror action (the Sync button) and the CLI import DB **git + local** rows in one run: git rows clone/pull as today, local rows are walked directly; a missing local directory aborts the run loudly before anything is imported.
## Work
1. The phase-35 resolution module (`app/rag/git_sources.py` or wherever `effective_git_sources()` lives — identify it in the repo) — extend:
- `effective_sources(db) -> list[GitSource]` — the DB rows of **both** kinds (DB-wins / env-fallback semantics unchanged: while the table is empty, the env git URLs surface as synthetic `kind='git'` rows with `from_env`). Keep a backward-compatible alias (`effective_git_sources`) if other modules import the old name.
- Log the list origin as today (`origin=db|env`) plus the kind counts (`git=N local=M`).
2. `app/api/sync.py::_run_sync`:
- For each resolved row: `kind=git` → `clone_or_pull(url, sources_root / repo_name(url))` (unchanged); `kind=local` → `Path(row.path).expanduser()`, verify `.is_dir()` **at sync time** (the directory may have moved/deleted since add-time) → else raise a sync error `local source missing: <path>` (no credentials involved, but run it through the existing `_sanitize_error` for consistency).
- `import_sources(combined, llm, prune=True)` over the **single combined list** (git checkouts + local dirs) — pruning covers the union (phase-32 semantics); the overview regeneration is unchanged (change-gated).
- The fail-loud empty-config check: no git rows, no local rows, and no env URLs → `"no sources configured (git or local)"` (replaces phase 32's git-only message).
3. `scripts/import_docs.py` — `_resolve_sources` gains the DB path (after `--source`, which still wins over everything): no `--source` and the DB table non-empty → the combined list (git cloned/pulled + local direct); table empty → env git URLs (as today) → the legacy `DEFAULT_SOURCES`. A missing local dir → abort with the path named, before importing anything (the same pre-import fail-loud as a failing git clone).
4. Unit/integration:
- Unit: `effective_sources` — mixed kinds, DB-wins, env fallback git-only, both-empty (extend the phase-35 tests).
- Integration (the `test_sync_api.py` pattern, with a **temp local dir** — a host temp dir containing one fixture `.md`, since the app server runs on the same host): local-only, git-only, and mixed syncs — the local file lands in the KB (`GET /api/docs` as admin); a missing local path → status `failed` with `local source missing: …` in the sanitized error; `import_docs` (no `--source`) with a DB local row imports it; `--source` still wins over the DB.
## Testing & Quality
- The suites above; `test_git_sources_api.py`, `test_sync_api.py`, `test_import_docs_git.py` (phase 35's list) stay green through the indirection.
- Coverage: **>90%** on the modified modules.
## Completion Criteria
- [ ] A mixed git + local sync imports both in one run; union pruning works (a file deleted from the local dir is pruned on the next sync).
- [ ] A missing local dir fails the run loudly (the status error names the path) and imports nothing.
- [ ] `import_docs`: DB git + local resolution; `--source` wins; the env fallback is git-only; the both-empty message is updated.
- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.
@@ -1,24 +0,0 @@
# Task 04 — The page: the local-directory form + badges
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
**Story:** `.agent/user_stories/local-directory-sources.md`
## Objective
The sources page (phase 35) can add a **local directory** next to git repos, the list shows which is which, and the hint reflects the combined Sync semantics.
## Work
1. The phase-35 page (the git-sources HTML template + its page script — identify the actual paths in the repo):
- A second add form, **"Local directory"**: a labeled text input (placeholder `~/Notes`), an Add button, an inline error slot — the same never-stale-button + inline-error pattern as the git form (PLAN §7.4); on success the form clears and the list re-fetches.
- List rows: a kind badge — `Git` / `Local` (a small styled span, distinguishable by color **and** text, not color alone — WCAG) — plus the mono value (git URL as today; the full local path) + added date + Remove (Remove is unchanged — it prunes on the next sync).
- The hint text: "Sync clones/pulls the git repos and imports the local directories together (files removed from a source are pruned)."
- Anonymous: the sign-in gate unchanged; the admin-only nav link unchanged (phase 35/29).
2. `frontend/assets/styles.css` — the badge styles + the second form (reuse the existing form styles; contrast ≥ 4.5:1 in both themes).
3. UI Structure Check (AGENTS.md rule 5) before finalizing: landmarks intact, both inputs labeled, focus-visible on the new Add button, the page stays inside the shared layout (no new top-level structure).
## Testing & Quality
- Gated by the story E2E (task 05); no CDN (AGENTS.md rule 6 — no new external tags).
## Completion Criteria
- [ ] The admin adds a local directory through the page; the row appears with the Local badge; an invalid path shows the 422 detail inline and the button recovers (never stale).
- [ ] The git form + all existing page behavior unchanged.
- [ ] `uv run pytest` green (no frontend unit layer — the E2E is the gate); `uv run ruff check . && uv run pyright` clean.
@@ -1,39 +0,0 @@
# Phase 39 — Configurable app name (brand)
**Source:** `TODO.md` L12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."
**Story:** `.agent/user_stories/configurable-brand.md`
**Context:** `app/config.py` (`app_name`, already `BOR_APP_NAME`, currently used only for the OpenAPI title at `app/main.py:48`), the five templates (`frontend/*.html` — the `.brand-text` spans, the `<title>`s, the meta descriptions, the index empty-state h1, the `aria-label`s), `frontend/assets/app.js` (status labels ~L107/112, the elapsed-hint aria ~L540), `frontend/assets/document.js` (page titles L147/152), `app/api/health.py` (the public stateless endpoint pattern), `app/core/caching.py` (phase 33: `?v=` rewriting of `assets/…` refs — automatic for any new asset file), the `Containerfile` esbuild stage (explicit per-asset lines, L17–23).
## Objective
Make **one env var** (`BOR_APP_NAME`, default "Brain of Reese") drive the app's display name everywhere — titles, the header brand, the status labels, the aria text, the greeting — via a public `/api/config` endpoint + a small `brand.js`, with zero behavior change when the variable is unset.
## Dependencies
- `34_consistent_navbar` (todo — runs before this phase) — the standard five-page header (the `.brand-text` nodes this phase re-skins).
- `01_infrastructure` (complete) — the public stateless endpoint pattern (`app/api/health.py`).
## Tasks
1. `01_config_endpoint.md` — public `GET /api/config` → `{app_name, version}` + tests.
2. `02_frontend_branding.md` — `brand.js` + the template/JS de-hard-coding + the Containerfile build line.
3. `03_e2e_docs_commit.md` — the story E2E (its own app instance with the overridden name), `.env.example` + README, commit, move the phase dir.
## Testing & Quality
- Unit/integration: the endpoint (200, the values, anonymous access, exactly two keys — no settings may leak later).
- E2E (mandatory, A16): `tests/e2e/test_configurable_brand.py` — a **second** app instance booted with `BOR_APP_NAME` overridden; run in isolation.
- Coverage: **>90%** on `app/`.
- Regression: the default-name behavior is byte-identical — the existing suites (which assert "Brain of Reese" titles/labels against the shared conftest server) stay green **unchanged**.
## Completion Criteria
- [ ] `GET /api/config` (anonymous) → `{"app_name": "Brain of Reese", "version": "0.1.0"}` by default; the response key set is exactly `{app_name, version}`.
- [ ] With `BOR_APP_NAME="Brain of Testy"`: the index title "Brain of Testy"; the header `.brand-text` renders `Brain of <strong>Testy</strong>`; the empty-state h1 "Hey! I'm Brain of Testy."; the `#messages` aria-label "Conversation with Brain of Testy"; the chat status label "Brain of Testy is thinking"; the other pages' titles carry the name; the document viewer title "… · Brain of Testy".
- [ ] With the variable unset: the existing E2E + unit suites green unchanged (no rename leak).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run pytest tests/e2e/test_configurable_brand.py -v --no-cov` green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
## Locked decisions
- **The existing `app_name` setting is the source of truth (owner permission 2026-08-26)** — `BOR_APP_NAME` (default "Brain of Reese"); no new variable, no setting rename.
- **A11 honoured** — runtime fetch + JS application (no build-time template injection, no Jinja, no CDN); the static templates stay static.
- **A10 honoured** — `/api/config` is public and stateless; the response carries no secrets (exactly `app_name` + `version`).
- **Runtime fetch, brief flash accepted (owner permission 2026-08-26)** — the default name renders immediately and is replaced when `/api/config` answers (LAN latency; no re-paint machinery for in-flight turns — a mid-turn label keeps the previous name for that turn).
- **A16/A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
@@ -1,27 +0,0 @@
# Task 01 — `GET /api/config`
**Phase:** `39_configurable_brand` · **Source:** `TODO.md:12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."`
**Story:** `.agent/user_stories/configurable-brand.md`
## Objective
A public, stateless endpoint that hands the frontend its display name (+ version) — the single source the brand layer reads.
## Work
1. `app/api/config.py` — a new router mirroring the `app/api/health.py` pattern:
```python
@router.get("/config")
def app_config(settings: Settings = Depends(get_settings)) -> dict[str, str]:
"""Public app metadata for the frontend brand layer (phase 39)."""
return {"app_name": settings.app_name, "version": settings.app_version}
```
Public (no `require_admin` — the brand must render for anonymous users too, before any sign-in); stateless (A10); the response carries **exactly** these two keys (no other setting may leak in later — the test asserts the key set).
2. `app/main.py` — register it with the other routers (`app.include_router(config_router, prefix="/api")`), before the static mount (API routes take precedence — same order as the existing routers).
3. Tests (follow the health test's location and pattern): 200 for anonymous; default values (`"Brain of Reese"`, the current `app_version`); with an overridden `Settings` (`app_name="Brain of Testy"`) the response follows; the response key set == `{"app_name", "version"}`.
## Testing & Quality
- The tests above; full `uv run pytest` green.
- Coverage: **>90%** on the new module (keep the handler + docstring tight so it stays covered).
## Completion Criteria
- [ ] `GET /api/config` (anonymous) → `{"app_name": "Brain of Reese", "version": "0.1.0"}` (the current default version).
- [ ] The suite green; `uv run ruff check . && uv run pyright` clean.
@@ -1,34 +0,0 @@
# Task 02 — `brand.js` + de-hard-coding
**Phase:** `39_configurable_brand` · **Source:** `TODO.md:12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."`
**Story:** `.agent/user_stories/configurable-brand.md`
## Objective
Every visible brand string on every page resolves from one place (`window.BOR_BRAND`, fed by `/api/config`) — the default "Brain of Reese" renders immediately, and a fetch failure falls back to the default (the page never breaks).
## Work
1. `frontend/assets/brand.js` — a new small **classic** script (vanilla, no CDN — A11; not a module, so its top level runs at parse time):
- Top level: `window.BOR_BRAND = "Brain of Reese"` (synchronous default — module scripts execute after parsing, so the page scripts can read it from the first line).
- DOM application (deferred: `if (document.readyState === "loading") document.addEventListener("DOMContentLoaded", applyBrand); else applyBrand();`):
1. `fetch("/api/config", { cache: "no-store" })`; on success with a non-empty `app_name`: `window.BOR_BRAND = app_name`, then:
2. `document.title = document.title.replaceAll("Brain of Reese", name)`.
3. Every `.brand-text` node: if the name starts with `"Brain of "` → `innerHTML = 'Brain of <strong>' + escapeHTML(rest) + '</strong>'` (the current look for the default name); else → `textContent = name` (plain, no bold). **HTML-escape the name** (an operator-controlled string must not inject markup).
4. A `TreeWalker` over the document's text nodes: replace the literal "Brain of Reese" with the name (catches the index empty-state h1 "Hey! I'm Brain of Reese." and any prose).
5. An attribute pass over `aria-label`, `placeholder`, and meta `content` attributes containing the literal → replace (the `#messages` aria-label "Conversation with Brain of Reese", the input label, the meta descriptions).
- On fetch failure: keep the default, `console.warn` (the `loadHealth` house style — progressive enhancement, never break the page).
- Small local `escapeHTML` helper (the `markdown.js` pattern — do not import across modules unless the build makes it easy).
2. The templates (`frontend/index.html`, `sources.html`, `tuning.html`, `document.html`, `login.html`) — add `<script src="assets/brand.js"></script>` **before** the page's module script on every page (classic script → runs at parse time; the module scripts execute later). No other template changes — the walker + attribute pass is the single mechanism; do **not** add `data-brand` markers. The phase-33 `?v=` rewriting picks the new ref up automatically (`app/core/caching.py` matches any `src="…assets/…"`).
3. `frontend/assets/app.js` — replace the "Brain of Reese" literals with `window.BOR_BRAND` reads: the `UI_STATE` labels (~L107: "… is thinking" / "… is answering"), `TYPING_LABEL` (~L112), the elapsed-hint aria-label (~L540). Pattern: `const brand = () => window.BOR_BRAND || "Brain of Reese";` + template strings. (Mid-turn staleness: a label set before the fetch lands keeps the old name for that turn — accepted, see the phase's locked decisions.)
4. `frontend/assets/document.js` — the page titles (L147/152: `${doc.title} · Brain of Reese` / `"Document not found · …"`) → use the same `window.BOR_BRAND` read (document.js is a module — `window.BOR_BRAND` is set by then).
5. `Containerfile` — add the esbuild line for the new file next to the others (L17–23 pattern, classic script like `markdown.js`): `esbuild ./assets/brand.js --minify --outfile=/out/assets/brand.js`.
6. Verify the no-op property: with the default settings the rendered DOM text is byte-identical to pre-phase on all five pages (the replace is a no-op for the default name) — the existing suites' title/label assertions are the guard; if any assert a string this task moved onto `window.BOR_BRAND`, the default path must render the identical bytes.
## Testing & Quality
- No Python logic — the story E2E (task 03) is the gate; the existing suites (which assert the default "Brain of Reese" titles/labels against the shared conftest server) must stay green **unchanged**.
- No CDN (rule 6): no new external tags. UI Structure Check (rule 5): no landmark/contrast change — the brand text keeps its existing classes and styling (the `innerHTML` rewrite only re-emits the same structure with the new name).
## Completion Criteria
- [ ] `BOR_APP_NAME` unset → all five pages render exactly as today (existing suites green).
- [ ] `BOR_APP_NAME="Brain of Testy"` → title/header/greeting/labels/aria all carry the new name (asserted by the story E2E, task 03).
- [ ] The Containerfile build includes brand.js; the asset ref is versioned like its siblings (phase 33).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,27 +0,0 @@
# Task 03 — Story E2E + docs + commit
**Phase:** `39_configurable_brand` · **Source:** `TODO.md:12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."`
**Story:** `.agent/user_stories/configurable-brand.md`
## Objective
The story's isolated Playwright suite against an app instance booted with the overridden name, the env docs, and the phase commit.
## Work
1. `tests/e2e/test_configurable_brand.py` (the story gate — one story, one file, run in isolation):
- A **second app instance** — the shared `app_server` conftest fixture keeps the default name (the other suites' title/label assertions depend on it). Copy the conftest `app_server` env block (same DB, the mock-LLM base URL, `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET`, `BOR_STATIC_DIR`, `BOR_RELEVANCE_THRESHOLD`) with two changes: `BOR_APP_NAME="Brain of Testy"` and a distinct port (`APP_PORT + 1` per the conftest convention). A session-scoped fixture **inside the test file**, started after `mock_llm` is available.
- Assertions (custom instance): index `document.title` == `"Brain of Testy"`; the `.brand-text` `innerHTML` == `Brain of <strong>Testy</strong>`; the empty-state h1 text == `"Hey! I'm Brain of Testy."`; the `#messages` `aria-label` == `"Conversation with Brain of Testy"`; the sources page title `"Sources · Brain of Testy"`; the login page title `"Sign in · Brain of Testy"`; one chat turn with a pre-token window (the `think out loud` marker) → the button label shows `"Brain of Testy is thinking"`.
- Default-name assertion (cheap regression in the same file): the shared conftest server's index title still == `"Brain of Reese"`.
- The chat-turn assertion works on the default (possibly empty) KB — a deflected answer is fine; the label assertion is pre-token, so no DB seeding is required.
2. `.env.example` — document `BOR_APP_NAME` in the App section (the display name on all pages; default "Brain of Reese").
3. README — the configuration section: `BOR_APP_NAME` (what it affects: titles, the header brand, the status labels, the aria text; the default; the bold-split rendering rule: names starting "Brain of " bold the remainder, any other name renders in normal weight).
4. Regression pass: `uv run pytest` + the coverage gate (>90%) + the isolated story E2E + the suites that assert brand strings (`test_smoke.py`, `test_shared_header.py`, `test_header_consistency.py`, `test_chat_persistence.py`) green.
5. Commit — one atomic `--no-gpg-sign` Conventional Commits commit for the whole phase (AGENTS.md rule 8), e.g. `feat(brand): configurable app name — BOR_APP_NAME drives /api/config + the frontend brand layer`; move the phase directory to `.agent/phases/complete/`.
## Testing & Quality
- The gates above are this task's quality bar (A16: one story, one isolated E2E file, coverage >90%).
## Completion Criteria
- [ ] The story E2E is green in isolation, deterministic across two consecutive runs (custom-name instance + the default-name assertion).
- [ ] The step-4 regression list green; coverage >90%.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
@@ -1,26 +0,0 @@
# Task 01 — Toggle ships hidden, admin-only reveal
**Phase:** `40_tuning_toggle_flash` · **Source:** `TODO.md:3` — "Loading the page briefly shows the 'Tuning' button in the header even when the user isn't authenticated. Only show that if the user is authenticated."
**Story:** `.agent/user_stories/tuning-toggle-flash.md`
## Objective
Ship `#steering-toggle` `hidden` in all six pages and unhide it in `header.js` only when whoami says admin — zero flash for anonymous, identical admin UX.
## Work
1. `frontend/index.html`, `frontend/sources.html`, `frontend/document.html`, `frontend/git-sources.html`, `frontend/login.html`, `frontend/tuning.html` — add the `hidden` attribute to the existing `#steering-toggle` `<button>` (the element that ships `aria-expanded="false" aria-controls="steering-panel"`; keep every other attribute, icon, label, and count badge byte-identical). The `#steering-panel` section already ships `hidden` — do not touch it.
2. `frontend/assets/header.js` — in `initSharedHeader()`, in the `if (admin)` branch, add `if (steeringToggle) steeringToggle.hidden = false;` **before** `if (steeringPanel) refreshSteering();`. The anonymous branch (`steeringToggle?.remove(); steeringPanel?.remove();`) stays byte-identical. Update the module docstring: the steering controls are now ship-hidden / reveal-for-admin (2026-08-27, `TODO.md` L3), matching the admin-only nav links.
3. `tests/unit/test_steering_toggle_visibility.py` (new) — source pins in the house style (regex/substring over the HTML + JS files, see `tests/unit/test_sync_button.py`):
- `#steering-toggle` carries `hidden` in **all six** pages;
- `header.js` contains the admin unhide (`steeringToggle.hidden = false`) inside `initSharedHeader`;
- the anonymous removal (`steeringToggle?.remove()`) is still present;
- `#nav-tuning` still ships `hidden` (the contract this phase relies on).
4. Grep the existing suites for exact-markup pins of the toggle (`tests/unit/test_shared_header.py`, `tests/unit/test_steering.py`, `tests/e2e/test_global_tuning.py`, `test_steering.py`) and update any assertion that breaks purely because of the new `hidden` attribute — behavior assertions stay.
## Testing & Quality
- Unit: the new pin file above; full unit suite green.
- Coverage: **>90%** on `app/` (no Python change — TOTAL must be unchanged; run `uv run pytest --cov=app --cov-report=term-missing`).
## Completion Criteria
- [ ] All six pages ship `#steering-toggle` with `hidden`; `header.js` reveals for admin and still removes for anonymous.
- [ ] `uv run pytest` green; coverage TOTAL unchanged.
- [ ] No behavior change in completed work (admin steering flow byte-identical: open/close, count badge, delete, announcer).
@@ -1,52 +0,0 @@
# Task 01 — Server core: round cap replaces the budgets (config + loop + unit/integration)
**Phase:** `45_agent_unlimited_tools` · **Source:** `TODO.md:8` — "Allow the LLM to make as many tool calls as it wants, remove the restrictions, they're causing problems getting correct answers"
**Story:** `.agent/user_stories/agent-unlimited-tools.md`
## Objective
One coherent server-side change, landed atomically so the suite is green at the checkpoint: `agent_max_rounds` (`BOR_AGENT_MAX_ROUNDS`, default 10; `0` = no tools) replaces both per-tool budgets in config, the agent loop, and every test that pins them.
## Work
1. `app/config.py` —
- **delete** the `agent_list_calls` and `agent_read_calls` fields (with docstrings);
- **add** in their place, same "RAG tuning" section:
```python
#: Hard cap on the agent tool rounds per grounded turn (phase 45,
#: revising phase 37's per-tool budgets — owner permission
#: 2026-08-27, TODO L8: "allow the LLM to make as many tool calls
#: as it wants"). Every tool call the model emits consumes a
#: round; at the cap the loop forces one final no-tools answer.
#: ``0`` disables the tools entirely — the turn is a single
#: request with ``tools=None`` (the pre-phase-37 path — the kill
#: switch).
agent_max_rounds: int = 10
```
- optional: a `field_validator` rejecting negative values (note it in the docstring if added).
2. `app/rag/agent.py` —
- `run_agent`: `max_rounds = settings.agent_max_rounds`; `tools = AGENT_TOOLS if max_rounds > 0 else None` (the kill switch — at 0 the loop makes exactly one request with `tools=None`, byte-identical to the pre-phase-37 path);
- delete `list_left` / `read_left` and the budget-driven `tools = None if (list_left == 0 and read_left == 0) else AGENT_TOOLS` transition — `tools` stays `AGENT_TOOLS` while rounds remain;
- after each executed call: `rounds += 1`; the existing cap branch becomes the **only** forced-exit: `if rounds >= max_rounds:` → the `logger.warning("agent round cap reached …")` + final `chat_stream(messages, tools=None)` (update the warning text: it is no longer belt-and-braces — it is the cap);
- `_execute_tool(db, call, seed_docs, holder)`: drop the `list_left` / `read_left` parameters and the `LIST_EXHAUSTED` / `READ_EXHAUSTED` early returns; keep the `ALREADY_IN_CONTEXT`, `UNKNOWN_TOOL`, `MISSING_READ_ARGS` rejections (non-budget — a repeated rejected call still consumes a *round* in the loop, so a pathological stream is bounded by `max_rounds`); return type simplifies to `str`;
- delete the `LIST_EXHAUSTED` / `READ_EXHAUSTED` constants;
- `AGENT_TOOLS`: `read_document` description "Add the full content of exactly one more indexed document to your context" → "Add the full content of one more indexed document to your context";
- module docstring: the budget paragraph (points 1, 3, 4) rewritten for the round cap (owner revision 2026-08-27, `TODO.md` L8); `run_agent` docstring updated (`seed_docs` note unchanged); `AgentHolder` unchanged (`tool_calls` still counts executed calls — now including re-lists);
- the per-call `logger.info("agent tool=… budget list_left=… read_left=…")` line becomes `logger.info("agent tool=%s args=%s round=%d/%d", …)` (or equivalent — the per-turn `tool_calls=N` field in `app/api/chat.py` is untouched).
3. `tests/unit/test_agent.py` — **rewrite** the budget tests around the round cap (keep the file's fake-LLM harness):
- an always-`list_documents`-calling mock with `agent_max_rounds=3`: exactly 3 tool rounds execute, then one forced `tools=None` request streams the answer; `holder.tool_calls == 3`;
- `agent_max_rounds=0`: exactly one request, `tools=None`, no tool lines, `holder.tool_calls == 0` (kill switch);
- an always-`read_document`-with-unknown-path mock (every call rejected — `No document at …`): the loop runs to `max_rounds` and forces the final answer (rejections no longer end the loop early via budgets, the cap bounds them);
- the existing rejections tests (`Unknown tool.`, `MISSING_READ_ARGS`, `Already in your context.`) keep passing — update their `_settings(...)` calls (`agent_max_rounds=…` instead of the budget kwargs);
- a **re-list** test: `list_documents` called twice in one turn executes both (the second returns the catalog again) and counts 2 in `holder.tool_calls`.
4. `tests/unit/test_config.py` — default 10; `BOR_AGENT_MAX_ROUNDS=0` / `=5` overrides; (negative validator, if added); delete the old budget assertions.
5. `tests/integration/test_chat_api.py` — the `agent_list_calls=0, agent_read_calls=0` fixture kwargs (~line 661) become `agent_max_rounds=0`; any other budget kwarg in the file the same; the tool SSE-event and `done.sources` assertions stay untouched.
6. `.env.example` — the two `BOR_AGENT_*_CALLS` lines become one: `# BOR_AGENT_MAX_ROUNDS=10 # hard cap on agent tool rounds per turn (0 = no tools)` (file's optional-setting comment style).
7. Grep the repo for `agent_list_calls|agent_read_calls|BOR_AGENT_(LIST|READ)_CALLS|LIST_EXHAUSTED|READ_EXHAUSTED` — zero hits outside `.agent/phases/complete/**` (history).
## Testing & Quality
- Unit + integration: full `uv run pytest` green at this checkpoint (the mock/E2E multi-read flow lands in task 02 — the existing 3-step mock flow still works unmodified, so `test_agent_document_tools.py` E2E is not yet run by the gate).
- Coverage: **>90%** on `app/` — `agent.py` + `config.py` fully covered by the rewritten tests.
## Completion Criteria
- [ ] No per-tool budgets anywhere in `app/` or `tests/`; `agent_max_rounds` is the single knob (default 10, `0` = kill switch).
- [ ] Re-lists execute; non-budget rejections intact; the cap bounds pathological streams; `tool_calls=N` log field and `tool` SSE event unchanged.
- [ ] `uv run pytest` + coverage gate green at this checkpoint.
@@ -1,27 +0,0 @@
# Task 02 — Mock: deterministic multi-read tool flow
**Phase:** `45_agent_unlimited_tools` · **Source:** `TODO.md:8` — "Allow the LLM to make as many tool calls as it wants, remove the restrictions, they're causing problems getting correct answers"
**Story:** `.agent/user_stories/agent-unlimited-tools.md`
## Objective
The E2E mock gains a deterministic **multi-read** agent flow (list → read #1 → read #2 → answer) so "as many tool calls as it wants" is provable statelessly, without disturbing the existing 3-step flow.
## Work
1. `tests/e2e/mock_llm.py` —
- the existing phase-37 flow (documented in the module docstring and `_tool_flow`): marker `TOOLS_TRIGGER` ("use your tools") + `<tools>` system section → step classification **statelessly from the messages**: no tool results yet → `list`; one `tool`-role message with the catalog prefix → `read` (first catalog doc, parsed from the listing via the `rsplit("/", 1)` convention); one `tool`-role message with the `Document <source/path>:` prefix → forced answer.
- add a **multi-read variant**: when the user message contains **both** `TOOLS_TRIGGER` and a new marker `MULTI_READ_TRIGGER = "read two documents"`, the classifier reads the *count* of `tool`-role messages whose content starts with `"Document "` (the read-result prefix, `app.rag.agent`'s `_execute_tool` output):
- 0 read results (+ no catalog yet) → `list`;
- 0 read results (catalog present) → `read` the **first** catalog doc;
- 1 read result → `read` the **second** catalog doc (the listing minus the already-read doc — parse the catalog lines the same way the existing read step does, skipping the path already read);
- 2 read results → forced answer: the existing answer shape (tail echo) plus a deterministic line naming **both** read paths (e.g. `"I read <path1> and <path2>."` — byte-stable) so the E2E can assert the model actually used both;
- the single-read flow (no `MULTI_READ_TRIGGER`) stays byte-identical — the variant must be a strict superset (the existing `test_agent_document_tools.py` E2E keeps passing unmodified).
- update the module docstring's tool-flow documentation (the multi-read steps + the marker).
2. `uv run pytest` green (mock-only change; the existing 3-step E2E is not run by the unit gate but must stay conceptually intact — the regression run in task 03 proves it).
## Testing & Quality
- Unit: full suite green; if a mock-specific unit test file exists (check `tests/unit/`), add the multi-read classification case there (catalog → read #1 → read #2 → answer) so the new branch is unit-covered; otherwise the E2E (task 03) covers it.
- Coverage: **>90%** on `app/` (unchanged — `tests/` only).
## Completion Criteria
- [ ] `TOOLS_TRIGGER` + `MULTI_READ_TRIGGER` → deterministic 4-step flow (list, read #1, read #2, answer naming both paths); the 3-step flow is unchanged for marker-less turns.
- [ ] Full unit/integration suite green.
@@ -1,40 +0,0 @@
# Phase 46 — Mobile hamburger nav
**Source:** `TODO.md` L9 — "The navbar on mobile is way too squished. Make it a hamburger dropdown menu with a nice animation"
**Story:** `.agent/user_stories/mobile-hamburger-nav.md`
**Context:** The shared bar (phase 19/34 — the same header block on all six pages) carries brand + up to four text nav pills (Chat, `#nav-sources`, `#nav-git-sources`, `#nav-tuning` — the latter three ship `hidden`, revealed for admin by `header.js`) + four action controls (steering toggle, `#sync-btn`, `#new-chat-btn`, sign in/out). At ≤640px the phase-34/35 squeeze rules (0.72rem pills, 0.05rem gaps) leave a bar that is squished and hard to hit. The fix: on mobile the nav links move into an animated hamburger dropdown; the action pills stay in the bar.
## Objective
At ≤640px the nav links live in a `#nav-toggle`-opened dropdown menu (slide+fade, reduced-motion-still) with comfortable targets and the same auth visibility; at >640px the bar is byte-identical to today.
## Dependencies
- `45_agent_unlimited_tools` (todo) — sequential only (no shared files).
- `34_consistent_navbar` / `35_git_sources_admin` (complete) — the six-page bar contract, the admin-only link reveal, and the mobile squeeze rules being superseded.
- `07_story_responsive_polish` (complete) — the ≤640px conventions (44px targets, safe areas).
## Tasks
1. `01_hamburger_markup_all_pages.md` — `#nav-toggle` + `id="app-nav"` on all six pages; the mobile CSS (dropdown, animation, reduced-motion).
2. `02_toggle_behavior.md` — `header.js` open/close behavior (aria, Esc, link-close, resize-close) + source pins.
3. `03_hamburger_e2e_and_commit.md` — story E2E suite (mobile + desktop + reduced motion) + regressions + commit.
## Testing & Quality
- Unit: new `tests/unit/test_hamburger_nav.py` — `#nav-toggle` (with `aria-controls="app-nav"`, `aria-expanded`, `aria-label="Menu"`) present in **all six** pages and absent-visible on desktop (CSS `display: none` outside the media query); `<nav class="app-nav" id="app-nav">` in all six; the mobile CSS block carries the dropdown rules + `.is-open` state + the 180ms transition + the reduced-motion override; the old nav-pill squeeze rules are gone/superseded; `header.js` carries the toggle binding (click, Esc, delegated link close, matchMedia close).
- Coverage: frontend-only — `app/` TOTAL unchanged, >90%.
- E2E (mandatory, A16): `tests/e2e/test_mobile_hamburger_nav.py`, run in isolation.
## Completion Criteria
- [ ] 375px: hamburger visible (44px target), inline nav hidden, no horizontal bar overflow; menu opens with animation, closes via link/Esc/outside; anonymous sees only "Chat" in the menu, admin sees all four links.
- [ ] `reducedMotion: "reduce"`: no transition, open/close still instant and correct.
- [ ] >640px: no hamburger, inline pills exactly as today (phase-34/35 contract intact).
- [ ] `uv run pytest` green; coverage TOTAL unchanged.
- [ ] `uv run pytest tests/e2e/test_mobile_hamburger_nav.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_nav_consistency.py`, `test_header_consistency.py`, `test_shared_header.py`, `test_responsive_polish.py`, `test_tuning_nav_link.py`.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5): the toggle is a labeled 44px button with `aria-expanded`/`aria-controls`; the menu keeps `nav aria-label="Primary"`; focus-visible on the new control; contrast ≥4.5:1; no CDN.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
## Locked decisions
- **Owner-locked (2026-08-27, roadmap A5):** nav links only in the menu; action pills stay in the bar; slide-down + fade 180ms; `prefers-reduced-motion` stills it; the existing ≤640px breakpoint (no new one).
- **A11 untouched** — no new assets/CDN; the menu reuses the existing `<nav>` element (no duplicated links, so the whoami reveal keeps working unchanged).
- **Phase-34 contract kept** — the same bar on every page; the auth visibility rules apply inside the menu exactly as before.
- **A16/A17 honoured** — one story E2E suite, one atomic commit.
@@ -1,42 +0,0 @@
# Task 01 — Hamburger markup (six pages) + mobile dropdown CSS
**Phase:** `46_mobile_hamburger_nav` · **Source:** `TODO.md:9` — "The navbar on mobile is way too squished. Make it a hamburger dropdown menu with a nice animation"
**Story:** `.agent/user_stories/mobile-hamburger-nav.md`
## Objective
All six pages carry the identical `#nav-toggle` button + `id="app-nav"`, and the ≤640px stylesheet turns the nav into an animated full-width dropdown — desktop untouched.
## Work
1. **Markup — all six pages** (`frontend/index.html`, `sources.html`, `document.html`, `git-sources.html`, `login.html`, `tuning.html`), each in its `<header class="app-header">` block:
- insert the toggle button **immediately before** the `<nav>`:
```html
<!-- Phase 46 (owner permission 2026-08-27, `TODO.md` L9): the
mobile hamburger — visible ≤640px only (CSS); opens the nav as
an animated dropdown. Behavior: assets/header.js. -->
<button type="button" class="nav-toggle" id="nav-toggle"
aria-expanded="false" aria-controls="app-nav" aria-label="Menu">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"><path d="M4 7h16M4 12h16M4 17h16"/></svg>
</button>
```
- give the existing nav the id: `<nav class="app-nav" id="app-nav" aria-label="Primary">` (every other attribute/child byte-identical — the links keep their `hidden` attributes; **no links duplicated**).
- keep the six pages visually/structurally identical (the phase-34 contract) — the toggle is part of the shared bar block, positioned the same on every page.
2. `frontend/assets/styles.css` —
- **global (outside media queries):** `.nav-toggle { display: none; }` (desktop: absent);
- **inside the existing `@media (max-width: 640px)` block:**
- `.nav-toggle { display: inline-flex; align-items: center; justify-content: center; width: 44px; height: 44px; padding: 0; color: var(--ink); background: none; border: 0; border-radius: var(--radius-sm); cursor: pointer; }` + `:focus-visible` inherits the global 3px outline + a hover state matching the other pills (the `.steering-toggle:hover` family);
- `.app-nav` becomes the dropdown: `position: absolute; top: 100%; left: 0; right: 0; flex-direction: column; gap: 0; background: var(--surface); border-bottom: 1px solid var(--line); box-shadow: <existing shadow token or 0 8px 24px rgba(0,0,0,.4)>; padding: 0.5rem 0; z-index: <above the header content — check the header's z-index and use header+1>;` — note the containing block is the sticky `.app-header` (`.header-inner` is not positioned), so the menu spans the header's full width, edge to edge — intended on mobile;
- **closed state (default):** `visibility: hidden; opacity: 0; transform: translateY(-8px); pointer-events: none; transition: opacity 180ms ease, transform 180ms ease, visibility 0s linear 180ms;`
- **open state:** `.app-nav.is-open { visibility: visible; opacity: 1; transform: none; pointer-events: auto; transition: opacity 180ms ease, transform 180ms ease, visibility 0s; }`
- **menu rows:** `.app-nav .nav-link { padding: 0.75rem 1.25rem; font-size: 1rem; }` (comfortable 44px+ targets, readable — this **supersedes** the ≤640px pill-squeeze rules for `.nav-link` and `.app-nav` gap in that block: delete/replace the `.nav-link { padding: 0.3rem 0.25rem; font-size: 0.72rem; }` and `.app-nav { gap: 0.05rem; }` rules, keeping the rest of the block);
- **reduced motion:** inside the file's existing `@media (prefers-reduced-motion: reduce)` block (the one covering the 640px rules — or a new one after it): `.app-nav { transition: none; }`;
- the **900px tablet block is untouched** (inline nav still in use at 641–900px); the action-pill rules in the 640px block are untouched; the header height (`--header-h: 58px`) is untouched.
- verify at 360px: brand (clipped clean as today) + hamburger + the four icon action pills fit without horizontal overflow (the old four text pills are gone from the bar — there is now room; if the bar still overflows, the brand ellipsis target absorbs it exactly as before).
## Testing & Quality
- Unit: the new `tests/unit/test_hamburger_nav.py` starts here with the markup + CSS pins (the JS pins land in task 02): toggle markup (attributes) in all six pages; `id="app-nav"` in all six; `.nav-toggle { display: none }` outside media queries; the mobile block carries the dropdown, `.is-open`, the 180ms transition pair, the reduced-motion override, and the superseded squeeze rules are gone; full suite green (behavior not yet wired — the menu is closed by default and CSS-inert without JS, so no E2E regressions at this checkpoint).
- Coverage: **>90%** on `app/` (unchanged).
## Completion Criteria
- [ ] Six identical toggles + `id="app-nav"`; desktop rendering byte-identical (`.nav-toggle` hidden, nav inline as before).
- [ ] Mobile: closed dropdown is invisible and non-interactive; `.is-open` (added by task 02's JS) will be the only opener.
- [ ] Full suite green at this checkpoint.
@@ -1,63 +0,0 @@
# Task 02 — Toggle behavior in the shared header module
**Phase:** `46_mobile_hamburger_nav` · **Source:** `TODO.md:9` — "The navbar on mobile is way too squished. Make it a hamburger dropdown menu with a nice animation"
**Story:** `.agent/user_stories/mobile-hamburger-nav.md`
## Objective
The menu opens and closes with correct ARIA state, Esc/link/outside dismissal, and desktop-resize cleanup — one module-owned binding, like the existing sign-out/steering bindings in `header.js`.
## Work
1. `frontend/assets/header.js` — a new module-import binding (same pattern as the sign-out binding: look up at import, guard null-safe, no page-script involvement):
```js
/* ---------- mobile hamburger (phase 46; module-owned) ----------
* ≤640px only (CSS hides the button elsewhere): #nav-toggle opens the
* nav as a dropdown (#app-nav .is-open — the animated state, task 01
* CSS). One binding for all six pages; a page without either element
* is a no-op, like the rest of this module. The nav LINKS keep their
* ship-hidden whoami contract (hidden links stay hidden inside the
* menu) — this binding only toggles the container. */
const navToggle = document.querySelector("#nav-toggle");
const appNav = document.querySelector("#app-nav");
function setNavMenu(open) {
if (!appNav || !navToggle) return;
appNav.classList.toggle("is-open", open);
navToggle.setAttribute("aria-expanded", open ? "true" : "false");
}
if (navToggle && appNav) {
navToggle.addEventListener("click", () =>
setNavMenu(!appNav.classList.contains("is-open")));
// A link click navigates (or closes same-page) — shut the menu.
appNav.addEventListener("click", (e) => {
if (e.target.closest("a")) setNavMenu(false);
});
// Esc closes while open (document-level; the menu is the only
// document-level overlay this module owns).
document.addEventListener("keydown", (e) => {
if (e.key === "Escape" && appNav.classList.contains("is-open")) {
setNavMenu(false);
navToggle.focus(); // focus returns to the opener
}
});
// Resize back to desktop: the inline nav reappears — no stale open
// state (the .is-open class is scoped by the ≤640px CSS anyway, but
// dropping it keeps aria-expanded honest).
const mq = window.matchMedia("(max-width: 640px)");
const onMqChange = () => { if (!mq.matches) setNavMenu(false); };
if (mq.addEventListener) mq.addEventListener("change", onMqChange);
else mq.addListener(onMqChange); // older engines, defensive
}
```
- update the module docstring: add the hamburger bullet (phase 46, owner permission 2026-08-27, `TODO.md` L9).
2. `tests/unit/test_hamburger_nav.py` — extend (from task 01) with the JS pins: `header.js` contains the `#nav-toggle` binding, `setNavMenu` (or equivalent) syncing **both** `.is-open` and `aria-expanded`, the delegated `a`-click close, the `Escape` close (with focus return), and the `matchMedia("(max-width: 640px)")` change-close; assert the binding is null-safe (`navToggle && appNav` guard).
3. `uv run pytest` green at this checkpoint.
## Testing & Quality
- Unit: the extended pin file; full suite green.
- Coverage: **>90%** on `app/` (unchanged — frontend-only).
## Completion Criteria
- [ ] One module-owned binding: click toggles (aria-expanded in sync), Esc closes + refocuses the toggle, a link click closes, desktop resize closes; pages lacking the elements are a no-op.
- [ ] The auth visibility contract is untouched — the binding toggles the container only; `hidden` links stay hidden.
- [ ] Full suite green at this checkpoint.
@@ -1,28 +0,0 @@
# Task 03 — Hamburger E2E + regressions + commit
**Phase:** `46_mobile_hamburger_nav` · **Source:** `TODO.md:9` — "The navbar on mobile is way too squished. Make it a hamburger dropdown menu with a nice animation"
**Story:** `.agent/user_stories/mobile-hamburger-nav.md`
## Objective
Prove the full mobile contract in the browser (visibility, contents per auth state, navigation, dismissal, animation, reduced motion, desktop regression) and commit the phase.
## Work
1. `tests/e2e/test_mobile_hamburger_nav.py` (new) — mock-only, DB up. Mobile tests use a 375×812 page (new page per test, or `page.set_viewport_size` — the conftest `page` fixture is 1280×800, so create mobile pages via the `browser` fixture); per the story's Playwright Mapping Rule:
- `test_mobile_hamburger_visible_and_bar_roomy` — 375px: `#nav-toggle` visible (box ≥44px in both dimensions), `aria-expanded="false"`, the inline nav links are **not** visible in the bar (menu closed — bounding boxes outside the header band or opacity 0), and no horizontal overflow (`document.documentElement.scrollWidth <= window.innerWidth`);
- `test_anonymous_menu_contents` — anonymous at 375px: click `#nav-toggle` → `aria-expanded="true"`, exactly **one** visible link in `#app-nav` ("Chat"); `#nav-sources` / `#nav-git-sources` / `#nav-tuning` remain `hidden` inside the menu;
- `test_admin_menu_contents` — login (e2e.auth_helpers) at 375px: open → Chat / Sources / Git sources / Tuning all visible (the whoami reveal works inside the menu);
- `test_link_click_navigates_and_closes` — admin at 375px: open, click "Sources" → URL becomes `/sources.html` and on the arrival page the menu is closed (`aria-expanded="false"`, no `.is-open`);
- `test_esc_and_outside_close` — open, press `Escape` → closed **and** focus is back on `#nav-toggle`; open again, click a neutral point (e.g. the page footer/main) → closed. (If the outside-click close is not implemented per task 02's contract — it is not: only Esc/link/resize close — assert instead that the menu stays open on an outside click and **note the accepted behavior** in the test docstring; the story's AC 4 lists Esc + link + resize, not backdrop click. Do NOT add a backdrop-close — it is out of the locked scope.)
- `test_animation_and_reduced_motion` — motion allowed: computed `transition-duration` on `#app-nav` includes `0.18s` (opacity/transform pair); open → the class/aria flip; `reducedMotion: "reduce"` (new context via the `browser` fixture): computed transition is `none`/`0s` and open/close still works;
- `test_desktop_unchanged` (regression) — 1280×800: `#nav-toggle` not visible (`display: none`), the inline nav renders in the bar exactly as before (admin: all four links visible inline).
2. Regression pass (isolation runs): `test_nav_consistency.py`, `test_header_consistency.py`, `test_shared_header.py`, `test_responsive_polish.py`, `test_tuning_nav_link.py`.
3. `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged; `uv run ruff check . && uv run pyright` clean.
4. Commit (Conventional Commits, `--no-gpg-sign`), e.g. `feat(header): hamburger dropdown nav on mobile (owner permission)`, staging this phase's files; move `.agent/phases/todo/46_mobile_hamburger_nav/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_mobile_hamburger_nav.py -v --no-cov` green in isolation.
- Coverage: **>90%** on `app/` (unchanged — frontend-only phase).
## Completion Criteria
- [ ] The story E2E suite passes in isolation (all seven tests); the five regression suites pass in isolation.
- [ ] One atomic `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
@@ -1,29 +0,0 @@
# Task 01 — Config: the ten new formats (allowed + default)
**Phase:** `47_quadlet_jinja_import` · **Source:** `TODO.md:10–11` — "Add '.container', '.network', '.volume' and other quadlet files to the list of allowed/parsed files" / "Add '.j2' jinja files to the list of allowed/parsed files"
**Story:** `.agent/user_stories/quadlet-jinja-import.md`
## Objective
`app/config.py` allows and imports the ten new formats by default; the env validator keeps rejecting truly unknown extensions.
## Work
1. `app/config.py` —
- `_ALLOWED_IMPORT_EXTENSIONS`: add `"container", "network", "volume", "image", "pod", "kube", "swap", "os", "endpoint", "j2"` (with a comment: A9 revised 2026-08-27, owner permission — the full Podman quadlet family + Jinja templates);
- `import_extensions` default: `"md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2"` (the original seven first, the ten appended — order is cosmetic, the set is what matters; keep the field's docstring and the `mode="after"` validator **unchanged** — it already normalizes/lowercases/dedups and rejects unknowns, so it accepts the new names automatically);
- update the module-level comment on `_ALLOWED_IMPORT_EXTENSIONS` (it cites A9 revised 2026-08-21 — append the 2026-08-27 revision).
2. `.env.example` — the commented `BOR_IMPORT_EXTENSIONS` line updates to the new default CSV (it currently documents the old default).
3. `tests/unit/test_config.py` —
- the allowed set contains all seventeen formats;
- the default `import_extensions` includes the ten new names (assert each);
- `import_extension_set` returns the dotted lowercased set (`.container`, `.j2`, …);
- the validator **accepts** a `BOR_IMPORT_EXTENSIONS` containing the new names (e.g. `md,container,j2`) and **still rejects** an unknown one (e.g. `md,xyz`) — the never-widen contract with the widened base set.
4. `uv run pytest tests/unit/test_config.py -v` green; full unit suite green (the chunker/importer don't know the new suffixes yet — task 02; no test at this checkpoint imports a new-format file).
## Testing & Quality
- Unit: as above; coverage **>90%** on `app/` (config covered).
- No behavior change for existing formats (the default CSV only grows — every previously-imported file still matches).
## Completion Criteria
- [ ] Default settings import all seventeen formats; env narrowing/widening rules behave (new names allowed, unknowns rejected).
- [ ] `.env.example` documents the new default.
- [ ] Full suite green at this checkpoint.
@@ -1,39 +0,0 @@
# Task 02 — Chunker dispatch + fixture files
**Phase:** `47_quadlet_jinja_import` · **Source:** `TODO.md:10–11` — "Add '.container', '.network', '.volume' and other quadlet files to the list of allowed/parsed files" / "Add '.j2' jinja files to the list of allowed/parsed files"
**Story:** `.agent/user_stories/quadlet-jinja-import.md`
## Objective
Every new suffix dispatches to plain-text paragraph packing (`chunk_text`), and the fixture tree carries realistic quadlet + jinja files for the unit/integration/E2E layers.
## Work
1. `app/rag/chunker.py` —
- `_FORMAT_CHUNKERS`: add the ten entries (owner decision R1: plain-text packing — no TOML/Jinja-aware splitter):
```python
".container": chunk_text, ".network": chunk_text, ".volume": chunk_text,
".image": chunk_text, ".pod": chunk_text, ".kube": chunk_text,
".swap": chunk_text, ".os": chunk_text, ".endpoint": chunk_text,
".j2": chunk_text,
```
- update the `_FORMAT_CHUNKERS` comment (currently "A9, revised: md, markdown, txt, yaml, yml, json, py") and the module docstring's per-format list (add: "**container / network / volume / image / pod / kube / swap / os / endpoint / j2** (A9 revised 2026-08-27) — quadlet unit files (TOML) and Jinja templates; plain-text paragraph packing (`chunk_text`) — no format-specific splitter (owner decision).").
- `chunk_document`'s unknown-suffix fallback stays as-is (belt-and-braces).
2. **Fixture files** (under `tests/fixtures/docs/homelab/` — the tree the import E2E imports; hidden dirs are skipped, so no dot-dirs):
- `quadlet/compose.container` — realistic quadlet TOML (≥ ~1 500 chars to exercise sub-splitting past a single paragraph pack): `[Unit]` (Description/Wants), `[Service]` (Restart=always), `[Container]` (Image, Ports, Environment, Network, Volume mounts, Exec), comments. Include a unique sentinel token on its own line, e.g. `# RESE-QUADLET-SENTINEL-77aa`.
- `quadlet/lan.network` — small: `[Unit]` + `[Network]` (Driver=bridge, IPAMDriver, Subnets) + sentinel `# RESE-NETWORK-SENTINEL-11bb`.
- `quadlet/cache.volume` — small: `[Unit]` + `[Volume]` (Driver, Device) + sentinel `# RESE-VOLUME-SENTINEL-22cc`.
- `templates/deploy.j2` — a Jinja snippet with `{% for %}` / `{{ var }}` / `{# comment #}` constructs (realistic: an ansible-style service template) + sentinel `RESE-JINJA-SENTINEL-33dd` (no `#` prefix — it lives in a `{% set %}` line or a comment the importer keeps).
- keep the existing fixture files byte-identical (other E2E suites import this tree — `test_import_documents.py` asserts exact chunk counts: **run that suite's expectations check**: the tree grew by 4 files, so the phase-02 import E2E's document/chunk count assertions will change — update `tests/e2e/test_import_documents.py`'s count constants in this task, or fold that update into task 04's E2E work; whichever you choose, the full E2E regression pass in task 04 must be green. Prefer updating the constants here so task 03's integration test and task 04 share the same fixture state.)
3. `tests/unit/test_chunker.py` —
- dispatch: for **each** of the ten suffixes, `chunk_document(content, "x/<name>.<suffix>")` produces the same chunks as `chunk_text(content, …)` (parametrize over the suffix list);
- the `.container` fixture (read the file in the test, house pattern) chunks into ≥2 chunks, every chunk ≤ `HARD_MAX_CHARS`, and the sentinel token survives in some chunk;
- the `.j2` fixture chunks; Jinja braces are just text (no special handling — assert a `{{` line appears verbatim in a chunk);
- the unknown-suffix fallback is unchanged (a `.whatever` file still chunk-paragraphs).
4. `uv run pytest tests/unit/test_chunker.py -v` green; full unit suite green.
## Testing & Quality
- Unit: as above; coverage **>90%** on `app/` (chunker dispatch covered).
- No behavior change for the seven original formats (their chunker bindings are untouched).
## Completion Criteria
- [ ] Ten dispatch entries; docstrings/comments cite the A9 revision; fixtures exist with their sentinels and the ≥1 200-char container file.
- [ ] `test_import_documents.py` count constants updated (or explicitly deferred to task 04 — state the choice in the task's completion note).
@@ -1,30 +0,0 @@
# Task 03 — Importer parity: walk, delta, prune, titles
**Phase:** `47_quadlet_jinja_import` · **Source:** `TODO.md:10–11` — "Add '.container', '.network', '.volume' and other quadlet files to the list of allowed/parsed files" / "Add '.j2' jinja files to the list of allowed/parsed files"
**Story:** `.agent/user_stories/quadlet-jinja-import.md`
## Objective
Prove the new formats ride the existing import machinery unchanged: the default walk picks them up, delta detection re-imports on change, prune drops them on removal, and titles fall back to the file stem.
## Work
1. `tests/unit/test_importer.py` — extend (house style — the walk function `iter_importable_files` is pure and unit-tested against temp trees):
- a temp tree containing one file per new extension (all ten) + one unknown (`.xyz`) + one hidden-dir file (`.esphome/x.container`) + one exclusion (`node_modules/y.container`): the default-extensions walk returns exactly the ten new files — unknown/hidden/excluded filtered;
- the seven original extensions still walk (regression in the same test);
- title extraction: a `.container` file with no H1 gets the stem title (`extract_title` fallback — via the importer's title path, whatever the house test asserts titles through).
2. `tests/integration/` — a new `tests/integration/test_import_quadlet_jinja.py` (or extend `test_importer_e2e.py` if that file's harness fits better — choose and note):
- build a temp source dir with a `.container`, a `.volume`, and a `.j2` file (reuse the fixture files' content or small inline variants);
- run `import_sources([dir], fake_llm, prune=False)` with the house fake (`tests/fakes.py::FakeEmbedder` — it already implements `embed` + `chat`; give it an `embed_one` delegate if the import path calls one — check the `Embedder` protocol in `app/rag/importer.py` and satisfy exactly what it names):
- the three docs land in `documents` (source/path/title — stem titles) with non-zero `chunks` rows;
- **delta:** re-run with the `.j2` file's content changed → that doc `updated` (hash changed), the others `unchanged`;
- **prune:** delete the `.volume` file, re-run with `prune=True` → pruned count 1, the row gone, its chunks cascade-deleted.
- use the existing DB integration harness (the conftest app/db fixtures in `tests/conftest.py` — same pattern as `test_importer_e2e.py`).
3. `uv run pytest tests/unit/test_importer.py tests/integration/test_import_quadlet_jinja.py -v` green; full suite green (DB up for the integration part: `podman compose up -d db`).
## Testing & Quality
- Unit + integration as above; coverage **>90%** on `app/` (the importer is unchanged code — the new coverage comes from exercising it with the new formats; if TOTAL dips from task 01's config growth, add asserts — but no `app/` code change is expected in this task).
- No `app/` code change expected: if a gap is found (e.g. the walk already accepts any dotted suffix and the config set was the only gate), record that in the task completion note — the tests then prove the gate's location.
## Completion Criteria
- [ ] Default walk indexes the ten new formats; hidden-dir/exclusion/unknown filtering unchanged; stem titles.
- [ ] Delta + prune parity for the new formats (integration).
- [ ] Full suite green at this checkpoint.
@@ -1,40 +0,0 @@
# Phase 48 — Nav rename: "Sources" → "RAG", "Git sources" → "Sources"
**Source:** owner request (chat, 2026-08-28) — "The 'Git sources' navbar item should be renamed to 'Sources' and the 'Sources' navbar item should be renamed to 'RAG'."
**Story:** `.agent/user_stories/nav-sources-rag-rename.md`
**Context:** the shared header (phase 19, `frontend/assets/header.js`) ships ONE nav on all six pages — `[Chat, #nav-sources "Sources" → /sources.html, #nav-git-sources "Git sources" → /git-sources.html, #nav-tuning "Tuning"]` — with the two admin-only links ship-hidden and revealed by id once `/api/whoami` says admin (phase 16 contract, phase 34 same-header-everywhere, phase 46 mobile hamburger). Both labels are plain `<a>` text in the six page files; `header.js` toggles only the `hidden` attribute and never reads the label, so the rename is markup-only.
## Objective
Relabel the two admin-only nav items across all six pages — the document-catalog link (`#nav-sources`) becomes **"RAG"** and the source-manager link (`#nav-git-sources`) becomes **"Sources"** — with ids, hrefs, order, the ship-hidden/reveal contract, and every other page control (viewer back button, Sync button, page titles/h1s) unchanged.
## Dependencies
- `34_consistent_navbar` / `19_shared_header` (complete) — the same full header block on all six pages and the `header.js` reveal-by-id contract this phase preserves.
- `35_git_sources_admin` (complete) — the `#nav-git-sources` link (and the `/git-sources.html` page) it labels.
- `46_mobile_hamburger_nav` (complete) — the `LINK_TEXTS` assertion in `test_mobile_hamburger_nav.py` that this phase updates.
## Tasks
1. `01_nav_label_swap.md` — swap the two label texts in all six page files + the label mentions in `header.js`'s docstring/comments (no logic).
2. `02_existing_test_labels.md` — update the two test suites that assert the old label text; leave every other assertion (viewer back button, sync label, h1/title markers) untouched.
3. `03_story_e2e_regression_commit.md` — the story E2E (`test_nav_rename_sources.py`), the regression suites in isolation, ruff + pyright, the one `--no-gpg-sign` commit, and the phase-dir move.
## Testing & Quality
- Unit/integration: no new backend logic (frontend markup only). The existing no-CDN integration test (`tests/integration/test_api.py::test_html_pages_served_locally_no_cdn`) must still pass — the swap touches no tags or origins (the `/git-sources.html` "Git sources" marker keeps matching the page's unchanged `<h1>`).
- Coverage: frontend-only; the `app/` >90% gate is unaffected (unchanged).
- E2E (mandatory, A16): `tests/e2e/test_nav_rename_sources.py` — the story gate, run in isolation.
## Completion Criteria
- [ ] All six pages show **"RAG"** on `#nav-sources` (href `/sources.html`) and **"Sources"** on `#nav-git-sources` (href `/git-sources.html`); nav order and `is-active`/`aria-current` placement unchanged; no id/href/`hidden` change.
- [ ] Anonymous still sees neither link; the admin sees both (contract preserved, proven by the story E2E + `test_shared_header.py`).
- [ ] `uv run pytest` green (the two updated suites included); `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged (no backend change).
- [ ] Regression E2E suites green in isolation: `test_nav_consistency.py`, `test_shared_header.py`, `test_mobile_hamburger_nav.py`, `test_git_sources_admin.py`, `test_header_consistency.py`, `test_sources_midstream_bug.py`, `test_smoke.py`.
- [ ] `uv run pytest tests/e2e/test_nav_rename_sources.py -v --no-cov` green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean (no Python behavior change, but run the gate).
- [ ] UI Structure Check (AGENTS.md rule 5): labeled links, landmarks/contrast/focus unchanged; no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit staging only this phase's files; `.agent/phases/todo/48_nav_rename_sources/` moved to `.agent/phases/complete/`.
## Locked decisions
- **Label-only rename (owner request 2026-08-28)** — exactly the two nav item texts change. Element ids (`#nav-sources`, `#nav-git-sources`), hrefs, nav order, the ship-hidden/reveal contract, `header.js` behavior, and every other label (the document viewer's "Sources" back button, the "Sync sources" button, page `<title>`/`<h1>` — incl. "Knowledge base" and "Git sources") are OUT of scope. If the owner later wants the page titles/h1s to follow the nav labels, that is a follow-up phase, not this one.
- **A11 untouched** — vanilla HTML only, no CDN, no new packages, no new tags.
- **A10 untouched** — no endpoint, auth, or `header.js` logic change; the rename rides the existing reveal-by-id path.
- **No schema / migration** — purely a markup + test-label change.
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
@@ -1,46 +0,0 @@
# Task 01 — Swap the two nav label texts on all six pages
**Phase:** `48_nav_rename_sources` · **Story:** `.agent/user_stories/nav-sources-rag-rename.md`
## Objective
Rename the nav items themselves: `#nav-sources` ("Sources" → **"RAG"**) and `#nav-git-sources` ("Git sources" → **"Sources"") in the shared header markup of all six pages, plus the label mentions in `header.js`'s doc comments. Markup only — no ids, hrefs, classes, logic, or other text changes.
## Work
On **each** of the six page files — `frontend/index.html`, `frontend/sources.html`, `frontend/git-sources.html`, `frontend/tuning.html`, `frontend/document.html`, `frontend/login.html` — make exactly two text swaps inside `<nav class="app-nav" aria-label="Primary">`:
1. The catalog link — change the visible text only:
```html
<a href="/sources.html" class="nav-link" id="nav-sources" hidden>Sources</a>
```
becomes
```html
<a href="/sources.html" class="nav-link" id="nav-sources" hidden>RAG</a>
```
(On `sources.html` the link carries `class="nav-link is-active" aria-current="page"` — keep those attributes exactly as they are; swap only `Sources` → `RAG`.)
2. The manager link — change the visible text only:
```html
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Git sources</a>
```
becomes
```html
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Sources</a>
```
(On `git-sources.html` the link carries `class="nav-link is-active" aria-current="page"` — keep them; swap only `Git sources` → `Sources`.)
3. `frontend/assets/header.js` — **comments only** (the reveal logic is id-based and must stay byte-identical): in the module docstring (the nav-link inventory: `"Sources" (#nav-sources, phase 19), "Git sources" (#nav-git-sources, phase 35)`) and in the inline comment above the `navGitSources` reveal, update the quoted labels to the new ones (`"RAG" (#nav-sources)`, `"Sources" (#nav-git-sources)`). No executable line of `header.js` changes.
Rules for all edits:
- Do **not** touch element ids, `href`s, `hidden` defaults, `class` attributes, `aria-current`, indentation, or any other text on the page (page `<title>`, `<h1>`, the viewer's "Sources" back button, the "Sync sources" button label — all stay).
- Do **not** add `is-active` anywhere or move the links — the physical nav order stays Chat, `#nav-sources`, `#nav-git-sources`, `#nav-tuning` (the labels just swap, so it now *reads* Chat, RAG, Sources, Tuning).
- The phase-16/19/35 ship-hidden contract is untouched: the links still ship `hidden` and `header.js` still reveals them by id for admins.
## Testing & Quality
- No new unit/integration logic (frontend markup only).
- Coverage: frontend-only; `app/` coverage unaffected.
- The no-CDN integration test (`tests/integration/test_api.py::test_html_pages_served_locally_no_cdn`) still passes — no tags or origins change; its `("/git-sources.html", "Git sources")` marker still matches that page's unchanged `<h1>Git sources</h1>`.
## Completion Criteria
- [ ] All six pages: `#nav-sources` renders text **RAG** and `#nav-git-sources` renders text **Sources**; ids/hrefs/`hidden`/`is-active` unchanged (a quick `grep -c 'id="nav-sources" hidden>RAG<'` per page returns 1, and `grep -c 'id="nav-git-sources" .*>Sources<'` returns 1).
- [ ] `git diff --stat` shows only the six HTML files + `frontend/assets/header.js` (comment-only) changed.
- [ ] `uv run pytest tests/unit tests/integration -q` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,34 +0,0 @@
# Task 02 — Update the tests that assert the old nav label text
**Phase:** `48_nav_rename_sources` · **Story:** `.agent/user_stories/nav-sources-rag-rename.md`
## Objective
Keep the existing suites honest after the rename: update the (few) assertions and label-describing comments that reference the OLD label texts, without touching any assertion that is about a different control.
## Work
Only these tests assert the old label text (verified by grepping `tests/` for the strings) — update exactly these:
1. `tests/e2e/test_git_sources_admin.py` — `test_admin_nav_link_on_all_five_pages_and_click_navigates`:
- the comment `pointing at the git sources page, labeled "Git sources" — and it` → `labeled "Sources"`;
- `expect(link).to_have_text("Git sources")` → `expect(link).to_have_text("Sources")`.
- This suite has no assertion on `#nav-sources`'s TEXT (it is id-based everywhere) — do not add one here; the story E2E (task 03) owns the label checks.
2. `tests/e2e/test_mobile_hamburger_nav.py` —
- `LINK_TEXTS = ("Chat", "Sources", "Git sources", "Tuning")` → `LINK_TEXTS = ("Chat", "RAG", "Sources", "Tuning")` (DOM order unchanged — only the labels swapped);
- the comment near `test_…` that reads `Chat / Sources / Git sources / Tuning — i.e. the whoami reveal` → `Chat / RAG / Sources / Tuning …`.
3. **Comments-only drift fixes** (no assertion changes) where a suite's docstring/comment quotes the old labels as the link's identity: `tests/e2e/test_shared_header.py` (module docstring line `"the 'Sources' nav link (#nav-sources)"` → `'RAG'`), `tests/e2e/test_nav_consistency.py` (the comment `the Git sources link joined in phase 35` → mention the relabeled link), `tests/e2e/test_header_consistency.py` if it quotes the labels. Where a comment also explains the phase-35 origin, keep that history and only fix the quoted label.
**Explicitly DO NOT touch** (different controls / different text — they stay green and must stay):
- `tests/e2e/test_nav_consistency.py` lines asserting the document viewer's **back button** `span` text "Sources" / "Chat" — that is the viewer's back link (`document.js`), not the nav.
- `tests/e2e/test_document_back_navigation.py` — same back button.
- `tests/unit/test_document_viewer.py` — the `'Sources' in js` check on the viewer's back-link label.
- `tests/e2e/test_nav_consistency.py` `expect(page.locator("#sync-label")).to_have_text("Sync sources")` — the Sync button label is unchanged.
- `tests/integration/test_api.py::test_html_pages_served_locally_no_cdn` — the `("/git-sources.html", "Git sources")` marker matches the page's unchanged `<h1>`; leave it.
## Testing & Quality
- No new logic — assertion-text updates only.
- Coverage: frontend/test-only; `app/` coverage unaffected.
## Completion Criteria
- [ ] `grep -rn '"Git sources"\|Git sources' tests/e2e/*.py` shows no remaining assertion expecting the old nav label (only phase-history comments, if any, are acceptable — and none should quote it as the current label).
- [ ] `uv run pytest tests/unit tests/integration -q` green; `uv run ruff check . && uv run pyright` clean.
- [ ] The two updated E2E suites pass in isolation: `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov` and `uv run pytest tests/e2e/test_mobile_hamburger_nav.py -v --no-cov` (DB up).
@@ -1,33 +0,0 @@
# Task 03 — Story E2E, regression suites, commit, phase move
**Phase:** `48_nav_rename_sources` · **Story:** `.agent/user_stories/nav-sources-rag-rename.md`
## Objective
Prove the renamed nav end to end with the story's dedicated Playwright suite (run in isolation per A16), confirm the surrounding header suites stay green, and land the one atomic `--no-gpg-sign` commit.
## Work
1. **Create `tests/e2e/test_nav_rename_sources.py`** (one story, one file — follow the conventions of `test_git_sources_admin.py` / `test_nav_consistency.py`: the shared `app_url` / `db_ready` fixtures, the `login(page, app_url, next=…)` helper, desktop viewport 1280×800, `expect` sync-API assertions, a module docstring stating the story + the rename). The six pages under test are `/` (chat), `/sources.html`, `/git-sources.html`, `/tuning.html`, `/login.html`, and `/document.html` (viewer — seed one document row first, reusing the seeding pattern `test_nav_consistency.py` uses for its viewer pass, and wait for `#doc-title` to settle as that file does).
Test cases:
- **admin_labels_on_all_six_pages** — signed-in admin visits each of the six pages; on each: `#nav-sources` is visible, has text exactly `RAG` and href `/sources.html`; `#nav-git-sources` is visible, has text exactly `Sources` and href `/git-sources.html`; the nav DOM order is Chat, RAG, Sources, Tuning (assert the `.app-nav a` text sequence); the current page's link carries `is-active`/`aria-current="page"` (and only it does).
- **click_navigates_with_marker** — from the chat page: click `#nav-sources` (label "RAG") → lands on `/sources.html` where `#nav-sources` is the active link; from the chat page: click `#nav-git-sources` (label "Sources") → lands on `/git-sources.html` where `#nav-git-sources` is the active link.
- **anonymous_sees_neither** — anonymous (no login): on `/` and `/login.html` both `#nav-sources` and `#nav-git-sources` are present in the DOM but hidden (ship-hidden contract unchanged), `#sign-in-link` visible.
- **untouched_controls_stay** — the rename did not leak: on `/sources.html` the Sync button still reads "Sync sources" (`#sync-label`), and on the settled viewer page the back button's span still reads "Sources" (the viewer back link is a different control — regression guard for task 02's do-not-touch list).
2. **Run the regression suites in isolation** (DB up, `--no-cov`), fixing nothing unless a test genuinely asserted a renamed nav label (if one does, update it as in task 02 and note it in the commit message):
`test_nav_consistency.py`, `test_shared_header.py`, `test_mobile_hamburger_nav.py`, `test_git_sources_admin.py`, `test_header_consistency.py`, `test_sources_midstream_bug.py`, `test_smoke.py`, `test_tuning_nav_link.py`.
3. **Full gates**: `uv run pytest` (unit + integration) green; `uv run pytest --cov=app --cov-report=term-missing` with TOTAL unchanged from the pre-phase baseline (frontend-only change); `uv run ruff check . && uv run pyright` clean.
4. **UI Structure Check (AGENTS.md rule 5)** — the relabeled links are still labeled `<a>`s inside the semantic `<nav aria-label="Primary">`; landmarks/contrast/focus-visible untouched (label text only); no CDN (rule 6 — the no-CDN integration test covers it).
5. **Commit** — one atomic commit staging exactly this phase's files (the six HTML files, `frontend/assets/header.js`, `tests/e2e/test_git_sources_admin.py`, `tests/e2e/test_mobile_hamburger_nav.py`, the comment-only test files from task 02, and the new `tests/e2e/test_nav_rename_sources.py`):
`feat(ui): rename nav items — "Sources" becomes "RAG", "Git sources" becomes "Sources"` with a body citing the owner request (2026-08-28) + phase 48. `git commit --no-gpg-sign` (repo also has `commit.gpgsign=false`).
6. **Move the phase directory**: `mv .agent/phases/todo/48_nav_rename_sources .agent/phases/complete/` and `git add -f .agent/phases/complete/48_nav_rename_sources` into the SAME commit as step 5 (`.agent/` is gitignored by design — AGENTS.md rule 8), plus the story file `git add -f .agent/user_stories/nav-sources-rag-rename.md`.
## Testing & Quality
- Story E2E: `tests/e2e/test_nav_rename_sources.py` green **in isolation**.
- Coverage: frontend-only; `app/` >90% gate unaffected (TOTAL unchanged).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_nav_rename_sources.py -v --no-cov` green in isolation (DB up).
- [ ] All eight regression suites green in isolation (commands above).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit containing the code/test files, the story file, and the moved phase directory; `.agent/phases/todo/` no longer lists 48.
- [ ] No behavior change in completed phases (the suites above are the proof).
@@ -1,51 +0,0 @@
# Phase 49 — Archive upload sources (tarball/zipfile → unpack → scan)
**Source:** owner request (chat, 2026-08-28) — "The git sources page should remove local directory and should instead accept a tarball or zipfile upload which it will unpack and scan. Note that reuploading the same tarball should not create a new folder, but should unpack and overwrite the previously unpacked content." (Design confirmed by the owner in the same conversation.)
**Story:** `.agent/user_stories/archive-upload-sources.md`
**Context:** `35_git_sources_admin` (complete) — the `git_sources` table (`id, url, kind, path, added_at`), the admin-only `/api/git-sources` router, and the `/git-sources.html` manager page; `38_local_directory_sources` (complete) — `kind='local'` rows the Sync pipeline and `import_docs` walk directly, plus the page's "Add a local directory" form this phase removes; `32_admin_sync_button` + `41_sync_fail_fast_models` (complete) — the in-process pipeline parts this phase reuses: `check_models` fail-fast, `import_sources(sources, llm, prune=True)` (source name = folder basename, per-file transactions, per-source prune), `regenerate_overview`, and the sync-detail count keys (`files, added, updated, unchanged, pruned, errors, chunks, overview`); `16_admin_auth` (complete) — the `require_admin` router dependency the new route inherits.
## Objective
Replace the local-directory form on the Sources page with an **archive upload** form: `POST /api/git-sources/upload` accepts `.tar`/`.tar.gz`/`.tgz`/`.zip`, unpacks it **safely** into `BOR_UPLOAD_DIR/<name>/` (name = filename minus the archive suffix), atomically swaps it in when the name already exists, upserts the `git_sources` row (`kind=local`, no duplicates), and **scans it** — single-source `import_sources(prune=True)` + overview refresh — returning the sync-style counts. Re-uploading the same filename overwrites the previous content in place: one folder, one row, dropped files pruned from the KB.
## Dependencies
- `48_nav_rename_sources` (todo — runs first) — sequential only: it relabels the nav in the same `git-sources.html` this phase edits (keeps the diffs clean).
- `35_git_sources_admin` (complete) — the table/API/page this phase extends; the `require_admin` router; the `IntegrityError → 409` backstop pattern (`_commit_new`).
- `38_local_directory_sources` (complete) — the `kind=local` rows uploads register; the local form removed; the phase-38 story E2E rewritten in this phase.
- `32_admin_sync_button` / `41_sync_fail_fast_models` (complete) — `check_models` + `import_sources` + `regenerate_overview` + the count-key contract the upload response mirrors.
- `16_admin_auth` (complete) — admin-only surface (A10 revision).
## Tasks
1. `01_settings_and_unpack_utility.md` — `BOR_UPLOAD_DIR` + `BOR_UPLOAD_MAX_MB` settings, `.env.example`, and the new `app/rag/archive_upload.py` (name derivation, safe tar/zip unpack with traversal/symlink/size guards, atomic swap-in) + unit tests.
2. `02_upload_api.md` — `python-multipart` dependency + `POST /api/git-sources/upload` (stream-with-cap, one-at-a-time 409, upsert row, fail-fast models, single-source scan, sync-style 200 body, log line) + integration tests incl. re-upload/overwrite and no-partial-state.
3. `03_admin_page_upload.md` — the page: local form out, upload form in (§7.4 lifecycle, result line, hint/caption) + the phase-38 story E2E rewritten API-driven.
4. `04_story_e2e_docs_commit.md` — the story E2E (`test_archive_upload_sources.py`), README, regression suites in isolation, the one `--no-gpg-sign` commit, and the phase-dir move.
## Testing & Quality
- Unit: `tests/unit/test_archive_upload.py` — the name-derivation matrix (suffix stripping incl. double `.tar.gz`; empty/`..`/separator/control-char rejection); safe unpack (valid zip + tar; zip-slip `../`; absolute member; symlink + hardlink escape; device member; extracted-byte cap); `swap_in` (fresh, replace-existing with full content replacement, failure leaves the previous folder intact).
- Integration: `tests/integration/test_git_sources_upload.py` — anonymous 403 on the new route; 422 (bad extension, unsafe/empty name, traversal archive, corrupt archive); 413 (compressed cap, via the settings-override pattern of `test_git_sources_api.py`); 409 (second upload while the first is in flight); 200 happy path (real temp tarball, counts correct, row `kind=local` under `upload_dir`, docs in the KB); **re-upload same name** (one row, old folder content fully replaced, dropped file pruned, new file indexed); **failed re-upload leaves the previous folder + row + KB untouched**. The existing `test_git_sources_api.py` / `test_sync_api.py` / `test_import_docs_git.py` suites stay green through the change (the `kind=local` POST contract is untouched).
- Coverage: **>90%** on `app/` — the new module + endpoint fully covered.
- E2E (mandatory, A16): `tests/e2e/test_archive_upload_sources.py`, run in isolation.
## Completion Criteria
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
- [ ] Uploading `homelab.tar.gz` via the page: unpacked under `BOR_UPLOAD_DIR/homelab/`, scanned (result line shows the counts), one list row (Local badge, name `homelab`), documents visible on `/sources.html`.
- [ ] Re-uploading `homelab.tar.gz` (modified): still exactly one folder and one row; dropped files pruned from the KB; added/changed files indexed.
- [ ] Non-archive file → inline 422; oversized → 413; zip-slip/tar-slip archive → 422 with the previous folder/row/KB untouched; second concurrent upload → 409.
- [ ] `#local-source-form` is gone from the page; the phase-38 story E2E green in isolation, API-driven; anonymous still gets the gate.
- [ ] `uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_git_sources_admin.py`, `test_local_directory_sources.py`, `test_sync_button.py`, `test_import_documents.py`, `test_nav_rename_sources.py` (when 48 is complete), `test_smoke.py`.
- [ ] README + `.env.example` document the upload (formats, naming, in-place replace, both new settings); ruff + pyright clean.
- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
## Locked decisions (owner permission 2026-08-28 — the confirmed design)
- **Formats:** `.tar`, `.tar.gz`, `.tgz`, `.zip` only (422 naming the accepted set otherwise).
- **Identity & in-place replace:** the source name is the uploaded filename minus the archive suffix (`homelab.tar.gz` → `homelab`, case-sensitive — Linux FS). The name determines the folder under `BOR_UPLOAD_DIR`; re-uploading the same name unpacks to a temp sibling and **renames it over the existing folder** (no missing window; a failed upload never touches the existing folder, row, or KB). **No second folder, no second row** — the `git_sources` row is upserted by `path` (`kind='local'`, reusing the phase-38 discriminator — **no migration, no new table**; A13 honoured).
- **New settings:** `BOR_UPLOAD_DIR` (default `~/bor-sources/uploads` — deliberately separate from the git checkouts in `BOR_SOURCES_DIR`) and `BOR_UPLOAD_MAX_MB` (default **512**) capping BOTH the compressed upload and the total extracted bytes (zip-bomb guard).
- **The scan is synchronous in the upload request** (owner-confirmed): fail-fast `check_models` (phase 41) → `import_sources([folder], llm, prune=True)` (single source) → `regenerate_overview` when the KB changed → **200** with the sync-detail count keys so the page renders the same "N added · N pruned" line. One upload at a time — 409 while a run is in flight (the phase-32 pattern).
- **Unpack safety:** absolute member paths, `..` traversal, symlink/hardlink targets escaping the unpack folder, and device/FIFO members are rejected (422); extracted bytes are counted against the cap while writing.
- **Page:** the "Add a local directory" form is **removed**; the `POST /api/git-sources` `kind=local` **API contract is unchanged** (admin can still register a plain directory via the API — no regression; existing Local rows still list/remove, and the Sync button + `import_docs` keep walking them).
- **No auto-unwrap** of a single top-level folder — files land in the KB exactly as packed (documented in the hint/README).
- **`python-multipart`** is added to the dependencies — FastAPI's required multipart parser for file uploads (an A2 FastAPI implementation detail, not a new architectural anchor; recorded here per AGENTS.md rule 3).
- **Boundaries (deliberately out of scope):** page `<title>`/`<h1>` rename (flagged in phase 48); background/202 upload runs (synchronous locked above); deleting the uploaded archive bytes (temp file removed after unpack — only the unpacked content is kept); cross-kind source-name collisions with a git repo of the same folder name (pre-existing importer behavior, unchanged); coordinating an in-flight full Sync with an upload (accepted edge — per-file transactions + per-source-name prune keep the KB consistent).
- **A10 / A11 / A16 / A17 honoured** — admin-only surface (no new session state), vanilla frontend (no CDN), one story E2E, one atomic `--no-gpg-sign` commit.
@@ -1,36 +0,0 @@
# Task 01 — Settings + safe archive unpack utility
**Phase:** `49_archive_upload_sources` · **Story:** `.agent/user_stories/archive-upload-sources.md`
## Objective
Add the two upload settings and the reusable, unit-tested unpack machinery in a new module `app/rag/archive_upload.py`: archive-name derivation, safe tar/zip extraction (traversal/symlink/device/size guards), and the atomic swap-in that makes re-uploads replace in place without ever exposing a missing or partial folder.
## Work
1. `app/config.py` — add to `Settings` (both env-overridable, documented like `sources_dir`):
- `upload_dir: str = "~/bor-sources/uploads"` (→ `BOR_UPLOAD_DIR`) — where uploaded archives are unpacked (one subdirectory per source name). Kept **separate** from `sources_dir` (git checkouts).
- `upload_max_mb: int = 512` (→ `BOR_UPLOAD_MAX_MB`) — caps both the compressed upload and the total extracted bytes (zip-bomb guard). Add a validator rejecting `<= 0` (fail loud at startup, the `agent_max_rounds` pattern).
2. `.env.example` — document both settings next to `BOR_SOURCES_DIR` (default, meaning, the cap's dual role).
3. `app/rag/archive_upload.py` (new module) — pure file-system logic, no FastAPI imports (the API layer maps its exceptions to status codes):
- `class ArchiveUploadError(Exception)` — carries a user-safe message (no paths beyond the owner's own upload dir, never secrets).
- `ARCHIVE_SUFFIXES: tuple[str, ...] = (".tar.gz", ".tgz", ".zip", ".tar")` (longest-first — `.tar.gz` must strip before `.tar` would).
- `archive_source_name(filename: str) -> str` — take the basename (defensively strip any `/` or `\` a client could send), strip ONE trailing archive suffix from `ARCHIVE_SUFFIXES`; the result must be non-empty and not `.`/`..`, contain no path separators or control characters, or raise `ArchiveUploadError` (the API maps to 422). `homelab.tar.gz` → `homelab`; `notes.tgz` → `notes`; `a.zip` → `a`; `x.tar` → `x`; bare `tar.gz` → error (empty stem).
- `unpack_archive(archive: Path, target_dir: Path, max_extract_bytes: int) -> None` — extract into `target_dir` (the caller guarantees it does not exist yet and creates it empty):
- **zip** (`zipfile`): per member — reject absolute names and any name with a `..` part; resolve the final path and require it to stay within `target_dir`; reject symlink entries (mode bits) and non-file/non-dir entries; write file members while counting bytes — exceeding `max_extract_bytes` raises `ArchiveUploadError` (name the cap, not the archive content).
- **tar** (`tarfile.open(mode="r:*")` — handles gz/bz2/xz transparently): per member — the same name/containment checks; reject char/block devices and FIFOs; for symlinks/hardlinks, resolve the link target against the member's directory and reject any target that escapes `target_dir`; write regular files with the byte-counted cap.
- Clean up partial state: on any error, remove `target_dir` (shutil.rmtree, ignore missing) so no half-unpacked tree survives.
- `swap_in(new_dir: Path, final_dir: Path) -> None` — make `new_dir` become `final_dir` with **no missing window**: if `final_dir` exists, rename it to a same-filesystem sibling `final_dir.with_name(final_dir.name + ".old-" + uuid4().hex)`, rename `new_dir` → `final_dir`, then delete the `.old-` sibling; if it does not exist, just rename. On a rename failure, best-effort restore (`.old-` back, `new_dir` cleaned) and re-raise as `ArchiveUploadError`.
4. `tests/unit/test_archive_upload.py` (new) — full coverage of the module:
- name derivation: the matrix above incl. `upper.TAR.GZ` (case-sensitive stems preserved — `upper`), `a.tar.gz` double-strip, `..tar.gz` / `..` / `a/b.tar` / `a\tb.zip` / empty-stem rejections.
- unpack: a valid zip (nested dir + file) and a valid tar.gz extract byte-identically; zip-slip (`../evil.txt`), absolute member (`/etc/x`), tar symlink escaping (`ln -s /etc/passwd link`), tar hardlink escaping, a char-device member, and the extracted-cap (e.g. cap=10 bytes, 20-byte file) each raise `ArchiveUploadError` AND leave no partial `target_dir` behind.
- `swap_in`: fresh (final absent), replace (final's previous content fully gone, new content complete — no interleave), and restore-on-failure (monkeypatch `os.rename` to fail on the second rename → previous folder intact, new dir cleaned, `ArchiveUploadError` raised).
## Testing & Quality
- Unit: `uv run pytest tests/unit/test_archive_upload.py -v` green; the module is fully covered (the >90% gate applies to `app/` — this new module must not drag the TOTAL down; aim for ~100% here).
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` — no drop vs. baseline (no existing behavior touched).
## Completion Criteria
- [ ] `Settings` exposes `upload_dir` / `upload_max_mb` (defaults as above); `BOR_UPLOAD_MAX_MB=0` or negative fails startup with the validator message.
- [ ] `.env.example` documents both settings.
- [ ] `app/rag/archive_upload.py` exists with `ArchiveUploadError`, `ARCHIVE_SUFFIXES`, `archive_source_name`, `unpack_archive`, `swap_in`; no FastAPI/DB imports in the module.
- [ ] `uv run pytest tests/unit/test_archive_upload.py -v` green; full `uv run pytest tests/unit tests/integration -q` green; `uv run ruff check . && uv run pyright` clean.
- [ ] No behavior change in completed phases (no existing file edited beyond config + .env.example).
@@ -1,43 +0,0 @@
# Task 02 — `POST /api/git-sources/upload` endpoint
**Phase:** `49_archive_upload_sources` · **Story:** `.agent/user_stories/archive-upload-sources.md`
## Objective
Ship the admin-only upload endpoint: stream the archive with the size cap, unpack it safely to a temp sibling, swap it in atomically, upsert the `git_sources` row (no duplicates), fail-fast check the models, scan the single source (`prune=True`) with the overview refresh, and answer 200 with the sync-style counts. One upload at a time (409).
## Work
1. `pyproject.toml` — add `python-multipart` to the main `dependencies` list (FastAPI's required multipart parser; the owner-confirmed implementation detail, phase locked decisions), then `uv lock`.
2. `app/api/git_sources.py` — extend the existing admin-only router (the `require_admin` dependency covers the new route automatically):
- `@router.post("/upload", response_model=UploadOut)` (new `UploadOut` schema in `app/schemas.py`: `source: str` + the sync-detail keys `files, added, updated, unchanged, pruned, errors, chunks: int` and `overview: bool` — same names as `_run_sync`'s `detail` so the page reuses its result-line shape).
- Handler (`async def upload_archive(file: UploadFile = File(...)) -> UploadOut`):
1. **Name/format gate** — `archive_source_name(file.filename or "")`: `ArchiveUploadError` → **422** with its message (naming the accepted formats when the extension is the problem).
2. **One at a time** — module-level `asyncio.Lock` (or a `bool` flag, the phase-32 `_task` spirit): already in flight → **409** `"an upload is already in progress"`.
3. **Stream with cap** — read the upload in 1 MiB chunks into `upload_dir / f".{name}.{uuid4().hex}.upload"`; total bytes > `upload_max_mb * 1024 * 1024` → delete the temp file, **413** naming the cap. (Compute `upload_dir` once: `Path(get_settings().upload_dir).expanduser()` — create it with `mkdir(parents=True, exist_ok=True)`.)
4. **Unpack to temp sibling** — `unpack_archive(archive, upload_dir / f".{name}.{uuid4().hex}.unpack", same cap)`; `ArchiveUploadError` → delete both temps, **422** (the message is already user-safe). A **completely empty archive** (zero entries) is **422** `"the archive contains no files"`. An archive with only non-A9 files is a **valid replacement**: the swap happens, the scan indexes nothing, and prune removes that source's docs — that is the intended "replace" semantics, do not reject it.
5. **Swap in** — `swap_in(temp_unpack, upload_dir / name)`; `ArchiveUploadError` → clean temps, **422** (the previous folder/row/KB are untouched — assert this in the tests).
6. **Upsert the row** — `path = str(upload_dir / name)`: a `GitSource` row with `path == path` already exists → leave it (no new row, `added_at` preserved); otherwise create `GitSource(url=path, kind="local", path=path)` (the `url` column is the NOT-NULL location column, phase-38 convention) via the shared `_commit_new` IntegrityError → 409 backstop.
7. **Fail-fast models** (phase 41) — `llm = LLMClient()`; `await check_models(llm)`; failure → **503** with the sanitized model-unavailable message (clean up nothing else — the folder/row are already committed, and the next sync/re-upload retries idempotently).
8. **Scan** — `summary = await import_sources([upload_dir / name], llm, prune=True)`; `overview = await regenerate_overview(llm) if summary.added + summary.updated > 0 else False`.
9. **Log** (PLAN §9, AGENTS.md rule 10) — one INFO line: `upload: name=… file=… bytes_in=… files=… added=… updated=… unchanged=… pruned=… errors=… overview=… total_ms=…`.
10. **Respond 200** with `UploadOut(source=name, …counts…, overview=overview)`.
- Keep every existing route byte-identical (the `kind=local` POST contract is untouched — the page just stops offering it).
3. `tests/integration/test_git_sources_upload.py` (new) — real Postgres + TestClient, following `tests/integration/test_git_sources_api.py`'s conventions (`clean_git_sources`-style TRUNCATE autouse fixture, the monkeypatched `get_settings` pattern to point `upload_dir` at a tmp dir and shrink `upload_max_mb`, the shared `client` / `admin_client` / `db` fixtures). Build real archives in-test with `tarfile`/`zipfile` over tmp fixture files (two `.md` sentinels). Cases:
- anonymous `client` → 403 on `/api/git-sources/upload` (same body as the rest of the router).
- `admin_client` + `data={"file": ("notes.txt", b"…", "text/plain")}` → **422** (accepted formats named); `("…", …)` with a `..`/empty stem → 422.
- oversized (shrink `upload_max_mb` via the settings override, upload a bigger file) → **413**, temp file cleaned up (the upload dir holds no stray `.` files).
- zip-slip archive (a member named `../evil.txt`) and a tar-slip symlink archive → **422**; then a prior good upload's folder + row + docs are **untouched** (the no-partial-state locked decision, asserted explicitly).
- happy path: `homelab.tar.gz` (2 md files) → **200**, `source="homelab"`, `added=2`, a `git_sources` row exists (`kind=local`, `path` under the tmp `upload_dir`), the unpacked folder exists, `GET /api/docs` lists both files under source `homelab`.
- **re-upload, same name, modified archive** (drop one file, add one, change one) → 200; `git_sources` still has exactly **one** row for that path (`added_at` unchanged); the folder contains only the new archive's files; the KB: dropped file **pruned** (pruned ≥ 1), new file added, changed file updated.
- corrupt archive (truncated zip bytes) → 422, previous state intact.
- 409: hold the in-flight flag (test seam: a module-level `upload_in_progress()` helper or the lock object exposed for tests — pick the smallest seam) → second request → **409**.
- the existing `test_git_sources_api.py`, `test_sync_api.py`, `test_import_docs_git.py` suites stay green (run them).
## Testing & Quality
- Integration: all cases above green; the endpoint's every branch (422/413/409/503/200 + both upsert branches + the empty-archive 422) is covered.
- Coverage: **>90%** on `app/` (the new handler + schema fully exercised).
## Completion Criteria
- [ ] `python-multipart` in `pyproject.toml` `dependencies` + lock file updated; `uv sync` clean.
- [ ] `POST /api/git-sources/upload` implements steps 1–10; `git diff app/api/git_sources.py` shows no change to the existing GET/POST/DELETE handlers' behavior.
- [ ] `uv run pytest tests/integration/test_git_sources_upload.py -v` green (DB up); `uv run pytest tests/unit tests/integration -q` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,58 +0,0 @@
# Task 03 — Sources page: local form out, upload form in (+ phase-38 E2E rewrite)
**Phase:** `49_archive_upload_sources` · **Story:** `.agent/user_stories/archive-upload-sources.md`
## Objective
Rework the manager UI: remove the "Add a local directory" form and replace it with the labeled archive upload form (never-stale button, inline error, result line), update the hint/caption, and — because the form it drives is gone — rewrite the phase-38 story E2E to add local sources via the API instead.
## Work
1. `frontend/git-sources.html`:
- **Remove** the whole `#local-source-form` block (the phase-38 form: label, `#local-source-path` input, `#local-source-add` button, `#local-source-error` — including its phase-38 comment).
- **Add**, in its place (same `.git-source-error` / never-stale visual language as the git form):
```html
<!-- Phase 49 (owner permission 2026-08-28): the archive upload form
replaces the phase-38 local-directory form — an uploaded
.tar/.tar.gz/.tgz/.zip is unpacked under BOR_UPLOAD_DIR and
scanned immediately; the same filename replaces the source in
place (no new folder, no duplicate row). -->
<form id="archive-upload-form">
<label for="archive-upload-file">Upload a source archive (.tar, .tar.gz, .tgz, .zip)</label>
<input id="archive-upload-file" name="file" type="file"
accept=".tar,.tar.gz,.tgz,.zip" required>
<button type="submit" id="archive-upload-btn">Upload &amp; scan</button>
<p class="git-source-error" id="archive-upload-error" role="alert" hidden></p>
<p class="git-source-result" id="archive-upload-result" role="status"
aria-live="polite" hidden></p>
</form>
```
(No `Content-Type` concerns: the JS posts a `FormData` and the browser sets the multipart boundary. The visible `<label>` satisfies the WCAG input-label rule for the file control.)
- **Update the hint** `#git-sources-hint`: uploads unpack + scan immediately and a same-name re-upload replaces in place; the Sync button still imports git checkouts and local directories together (union prune), and removing a source prunes on the next sync. Keep it one `role="note"` paragraph.
- **Update the table `<caption>`** (visually hidden): it reads "git repositories it clones and local directories it walks" — add uploaded archives (unpacked under the upload dir) to the description.
- Do **not** touch the page `<title>`/`<h1>` ("Git sources") — out of scope (flagged in phase 48); the nav link already reads "Sources" after phase 48.
2. `frontend/assets/styles.css` — one small block near the git-sources page styles: the file input (mono-ish, on-surface, ≥44px touch target, `:focus-visible` 3px outline like the other controls) and the `#archive-upload-result` success line (ink-soft on surface, ≥4.5:1 — reuse the existing palette; `prefers-reduced-motion` already global). Keep it minimal — no new layout regions.
3. `frontend/assets/git-sources.js`:
- Remove the local-form element refs (`localFormEl`, `pathInput`, `localAddBtn`, `localAddError`) and the second `wireAddForm(…)` call; update the module docstring (the page now wires the git add form + the archive upload).
- **Upload wiring** (one new `wireUploadForm`-style submit handler on `#archive-upload-form`), the §7.4 never-stale lifecycle:
- submit → `e.preventDefault()`; no file selected → inline error (the input is `required` too — browser prompt first); hide a previous result line; disable `#archive-upload-btn`, label **"Uploading…"**.
- `fetch("/api/git-sources/upload", { method: "POST", body: new FormData([["file", file]]) })` — **no** manual `Content-Type` header.
- **200**: clear the file input; hide the error; show `#archive-upload-result` with the counts in the sync-result shape (`2 added · 1 updated · 3 unchanged · 1 pruned` — omit zero parts, the `fmtSyncResult` convention from `sources.js`); announce through `#git-sources-announcer` (`"Archive uploaded: …"`); `await loadSources()` (the new/updated row lands with the Local badge; on a re-upload the row simply refreshes — no duplicate).
- **non-2xx**: inline the server detail via the existing `apiDetail(r, fallback)` (422 format/name/traversal, 413 size, 409 busy — the server messages are already user-safe); keep the file selection; restore the button (finally block, success AND failure).
- **network failure**: the `networkMessage` line, button restored.
4. **Rewrite `tests/e2e/test_local_directory_sources.py`** (phase-38 story — the form it drives is gone; its *acceptance* stands):
- Replace every form interaction (`page.fill("#local-source-path", …)` + `page.click("#local-source-add")`) with the authenticated API call the page's own JS no longer makes: `r = page.request.post(f"{app_url}/api/git-sources", json={"kind": "local", "path": str(local_dir)})` (the cookie rides the browser context — the established `page.request` pattern, cf. `test_git_sources_admin.py`).
- The "missing path → inline 422 naming the path + input kept" test becomes: the API returns **422** whose `detail` names the path (assert on the JSON body); drop the input-value assertions.
- Everything else stays: the Local badge on list rows, add/remove lifecycle, Sync importing the local dir (visible via `GET /api/docs` / the page), prune-on-file-deletion after re-sync, anonymous 403s, and the gate.
- The module docstring must note the phase-49 rewrite (form → API) so a future reader doesn't "restore" the form.
5. **UI Structure Check (AGENTS.md rule 5)** while in there: the new form is inside the existing `<main>` region, labeled, focus-visible, error `role=alert`, result `role=status`; no CDN (rule 6 — the no-CDN integration test re-proves it).
## Testing & Quality
- No new unit/integration logic (frontend + one E2E rewrite). `app/` coverage unaffected.
- The no-CDN integration test still passes (markup only, same-origin).
- `uv run pytest tests/e2e/test_local_directory_sources.py -v --no-cov` green in isolation **after** the rewrite (DB up) — this is the proof the form removal caused no regression in the phase-38 story.
## Completion Criteria
- [ ] `#local-source-form` / `#local-source-path` / `#local-source-add` are gone from `frontend/git-sources.html` (and `git-sources.js`); `#archive-upload-form` with file input (accept set), button, `role=alert` error and `role=status` result line is in its place.
- [ ] Hint + caption updated; page `<title>`/`<h1>` untouched.
- [ ] `tests/e2e/test_local_directory_sources.py` green in isolation, fully API-driven for local adds.
- [ ] `uv run pytest tests/unit tests/integration -q` green; `uv run ruff check . && uv run pyright` clean (no Python behavior change expected in `app/` beyond nothing).
- [ ] Manual-feel check via the story E2E in task 04 (upload → counts → row) — this task leaves the page in a usable state on its own (independent viability).
@@ -1,34 +0,0 @@
# Task 04 — Story E2E, README, regression suites, commit, phase move
**Phase:** `49_archive_upload_sources` · **Story:** `.agent/user_stories/archive-upload-sources.md`
## Objective
Prove the archive-upload story end to end with its dedicated Playwright suite (A16, in isolation), document the feature in the README, confirm no regression in the surrounding suites, and land the one atomic `--no-gpg-sign` commit.
## Work
1. **Create `tests/e2e/test_archive_upload_sources.py`** (one story, one file — conventions of `test_git_sources_admin.py` / `test_local_directory_sources.py`: shared `app_url` / `db_ready` fixtures, the `login(page, app_url, next=…)` helper, `page.request` for API assertions, desktop viewport 1280×800, sync `expect`). Build real archives in-test with Python's `tarfile` over `tmp_path` fixture files (markdown sentinels, e.g. `ALPHA-…` / `BETA-…` / `GAMMA-…`), named `e2e-upload.tar.gz` (source name will be `e2e-upload`).
Test cases:
- **form_swapped** — signed-in admin on `/git-sources.html`: `#local-source-form` count is 0; `#archive-upload-form` is visible with the file input (`accept` contains the four extensions) and the "Upload & scan" button; the hint mentions unpack/scan + in-place replace.
- **upload_scans_and_lists** — `page.set_input_files("#archive-upload-file", tarball_v1)` + click: the button shows "Uploading…" while in flight, then restores; `#archive-upload-result` shows the added count (2); the list gains exactly one row for `e2e-upload` with the **Local** badge; `page.request.get("/api/docs")` lists both sentinel files under source `e2e-upload`; on `/sources.html` (the RAG catalog) the table shows them (the admin sees the content where they expect it).
- **reupload_replaces_in_place** — back on the page, upload `tarball_v2` under the **same filename** (`e2e-upload.tar.gz`; v2: `alpha` modified, `beta` removed, `gamma` added): result line shows pruned ≥ 1; the list still has exactly **one** `e2e-upload` row (no duplicate — the row count for that source is invariant); `/api/docs` now shows `gamma` + the changed `alpha` and NOT `beta`.
- **bad_file_inline_error** — upload a `.txt` via the file input: `#archive-upload-error` (role=alert) shows the 422 detail naming the accepted formats; the button restores; the list is unchanged; a subsequent good upload still works (the form isn't wedged).
- **anonymous_gate** — anonymous on `/git-sources.html`: the gate (`#git-sources-gate`) shows, `#git-sources-content` (and thus the upload form) stays hidden, and `page.request.post(f"{app_url}/api/git-sources/upload", …)` is 403.
2. **README** — in the sources section (next to the phase-35/38 admin-sources docs): the upload form (accepted formats), the naming rule (filename minus archive suffix = source/folder name), the in-place replace + prune semantics on re-upload, the unpack destination (`BOR_UPLOAD_DIR`, default `~/bor-sources/uploads`) and the size cap (`BOR_UPLOAD_MAX_MB`, default 512, compressed + extracted), and the note that the local-directory *form* is gone but `POST /api/git-sources` with `kind=local` still works and existing Local rows are unchanged.
3. **Run the regression suites in isolation** (DB up, `--no-cov`): `test_git_sources_admin.py`, `test_local_directory_sources.py` (task 03's rewrite), `test_sync_button.py`, `test_import_documents.py`, `test_nav_rename_sources.py` (if phase 48 is complete — otherwise note it as the 48 gate), `test_smoke.py`, `test_shared_header.py`.
4. **Full gates**: `uv run pytest` (unit + integration) green; `uv run pytest --cov=app --cov-report=term-missing` **>90%**; `uv run ruff check . && uv run pyright` clean.
5. **Commit** — one atomic commit staging exactly this phase's files (config, `.env.example`, `app/rag/archive_upload.py`, `app/api/git_sources.py`, `app/schemas.py`, `pyproject.toml` + lock, the two new test files, the rewritten `tests/e2e/test_local_directory_sources.py`, `frontend/git-sources.html`, `frontend/assets/git-sources.js`, `frontend/assets/styles.css`, README):
`feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place` with a body citing the owner request (2026-08-28) + phase 49. `git commit --no-gpg-sign`.
6. **Move the phase directory**: `mv .agent/phases/todo/49_archive_upload_sources .agent/phases/complete/` and `git add -f` the moved directory + the story file `.agent/user_stories/archive-upload-sources.md` into the SAME commit (`.agent/` is gitignored by design — AGENTS.md rule 8).
## Testing & Quality
- Story E2E: `tests/e2e/test_archive_upload_sources.py` green **in isolation**.
- Coverage: `app/` >90% (the phase's Testing & Quality bar, re-proven on the final pass).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_archive_upload_sources.py -v --no-cov` green in isolation (DB up).
- [ ] All regression suites green in isolation (list above).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] README + `.env.example` document the upload semantics and both new settings.
- [ ] One `--no-gpg-sign` commit containing code, tests, story file, and the moved phase directory; `.agent/phases/todo/` no longer lists 49.
- [ ] No behavior change in completed phases (the suites above are the proof — incl. the Sync button and the `kind=local` API contract).
@@ -1,55 +0,0 @@
# Phase 21 — Thinking Window: No Scroll Back, Just Live
**Status:** Complete
**Date:** 2026-08-25
## Objective
Disable user scrolling in the Thinking window so it always shows the live tail
of the reasoning stream in a fixed 320px clip.
## What was implemented
### Task 01 — CSS change + unit pins (completed by prior agent)
- `frontend/assets/styles.css`: `details.thinking .thinking-text` changed from
`overflow-y: auto` to `overflow-y: hidden` (with owner-choice comment).
- `tests/unit/test_thinking_no_scroll.py`: source-level pins for CSS value,
comment, and JS bottom-pin integrity.
### Task 02 — E2E story suite, story file, validation, commit (completed this run)
- `tests/e2e/test_thinking_no_scroll.py` (5 tests):
1. `test_thinking_window_not_user_scrollable` — wheel/drag/keyboard on the
frozen live tail do not move the window.
2. `test_thinking_window_tracks_live_tail` — per-chunk pin keeps the window
glued to the tail; last chunk's text is inside the visible rectangle.
3. `test_thinking_window_css_contract` — computed `overflow-y` is `hidden`,
`max-height` is `320px`, clip is real.
4. `test_answer_bubble_still_scrollable` (regression, phase 11) — answer
bubble overflow is untouched, page scrolls normally.
5. `test_restored_collapsed_thinking_unaffected` (regression, phase 17) —
stored thinking restores collapsed with full text.
- `.agent/user_stories/thinking-no-scroll.md` — story file with bug report,
narrative, acceptance criteria, owner-confirmed decisions, and Playwright
mapping.
## Test / lint / coverage results
| Check | Command | Result |
|-------|---------|--------|
| E2E story suite | `uv run pytest tests/e2e/test_thinking_no_scroll.py -v --no-cov` | **5/5 passed** |
| Regression: thinking display | `uv run pytest tests/e2e/test_thinking_display.py -v --no-cov` | **5/5 passed** |
| Regression: long answers | `uv run pytest tests/e2e/test_long_answers.py -v --no-cov` | **2/2 passed** |
| Full suite | `uv run pytest --no-cov` | **319 passed** |
| Coverage | `uv run pytest --cov=app --cov-report=term-missing` | **99%** (gate: >90%) |
| Lint | `uv run ruff check .` | **All checks passed** |
| Types | `uv run pyright` | **0 errors** |
## Notable decisions
- No JS changes: the phase-17 bottom-pin (`textEl.scrollTop = textEl.scrollHeight`)
already works under `overflow-y: hidden`, so the CSS-only change is sufficient.
- The E2E tests use the mock's `think out loud then hesitate` trigger (phase 20)
to create a 4s frozen window where user scroll would be observable if regressed.
- The long thinking body (~2,700 chars, lengthened in phase 21) overflows the
320px window by ~2x, making the live-tail clip contract E2E-observable.
## Deviations
None. All changes follow LOCKED DECISIONS A11 (no CDN, pure CSS), A16 (one E2E
suite per story), and the owner-confirmed roadmap A2 decisions.
@@ -1,186 +0,0 @@
# Phase 22 / Task 01 — Background animation diagnosis + fix
**Owner report (2026-08-24):** "background animation not working, just
blinking". **Method:** live dev server (`uv run uvicorn app.main:app`,
db up), headless Chromium 1280×800 (Playwright), ≥15s observation per
session, computed-style sampling + pixel-diffed screenshots. All
artifacts in `.agent/screenshots/22_background_animation/`.
## Diagnosis (per the 00_phase.md checklist)
**1. Per-layer visibility.** Both pseudo-layers exist, computed
`position: fixed`, `z-index: -1`, `pointer-events: none`, both animating.
Per-layer screenshots (`before_glow_only.png`, `before_grid_only.png`):
the glow is clearly visible in its two corners; the grid is drawn **only
in a narrow band below the header** (top ~25% of the viewport) and is
barely distinguishable from the page bg.
**2. Grid layer — moving but imperceptible.** `background-position` on
`body::before` sampled over 16.8s: `0.86px → 13.2px` (≈0.79px/s — the
spec'd 44px/60s, seamless one-cell drift). The drift is *real*.
Isolating it (glow hidden, two frames 5s apart,
`before_gridonly_frame1/2.png`):
| metric | value |
|---|---|
| pixels changed (>12/765) | 1.5% |
| mean pixel change | **0.48/765** (≈0.2/255 per channel) |
| bottom-half quadrants | **≈0.07** (no grid there at all — masked out) |
Why: 1px lines at **35% of `--line`** blend to ≈rgb(20,26,41) on
rgb(10,14,23) — a ≈10/12/18-level delta — and the radial mask
(`120% 90% at 50% 0%, black 25%, transparent 78%`) fades the layer to
~zero past ~560px at center (probe: lines gone by y≈254 at the edges).
A 0.73px/s drift of that faint texture sits below the perception
threshold → the grid contributes no visible motion.
**3. Glow layer — the only visible motion = the "blink".** Full-background
frame diff (5s apart, `before_frame1/2.png`): 1.4% of pixels changed,
concentrated exactly on the two glow centers — top-left mean 3.07,
bottom-right 2.92 (glow positions 12%/8% and 88%/92%), top-right 1.23,
bottom-left 0.70. The 0.65↔1.0 opacity swing (35% amplitude, + scale)
is the entire perceived animation → reads as a blink/pulse, not
breathing.
**4. Occlusion check — ruled out.** Forced the grid lines to bright red:
3127 red-line pixels render (`before_occlusion_probe.png`, x 132–1276,
y 64–254 — the mask shape). Computed: `html` bg `rgb(10,14,23)`
(= `var(--bg)`), `body` bg `rgba(0,0,0,0)` (transparent), no later rule
paints over the layers. Assumption (c) **does not fire**.
### Root cause
**Assumptions (a)+(b) confirmed, (c) disproven.** The grid drift works
but is invisible (faint 35%-alpha lines × small masked region × 0.73px/s),
so the only perceived motion is the glow's large opacity swing — which
is exactly what "just blinking" is.
## Fix (styles.css — pure CSS, zero JS, no blur, palette untouched)
| property | before (phase 08) | after (phase 22) |
|---|---|---|
| grid line alpha | `rgb(38 48 74 / 0.35)` ×2 | **`rgb(38 48 74 / 0.6)`** ×2 |
| grid mask | `120% 90% at 50% 0%, black 25%, transparent 78%` | **`140% 110% at 50% 0%, black 40%, transparent 90%`** |
| grid drift | 60s, 0→44px | **unchanged** (60s, 0→44px — one-cell seamless) |
| glow opacity | 0.65 ↔ 1 | **0.85 ↔ 1** |
| glow period/scale | 14s ease-in-out alternate, 1↔1.05 | **unchanged** |
- Grid: +71% line contrast and a wider fully-visible mask radius make
the same 60s one-cell drift readable as smooth motion; the grid now
extends across most of the viewport, fading to the corners.
- Glow: amplitude cut 35%→15% — breathing, not pulsing.
- Durations deliberately untouched: the owner-confirmed intended effect
is the phase-08 design (60s drift + 14s breathe), and
`tests/e2e/test_dark_tech_theme.py::test_animated_background` pins the
live 60s/14s durations. Both layers keep `fixed; inset: 0; z-index:
-1; pointer-events: none`; `html` keeps `var(--bg)`, `body` stays
transparent; reduced-motion stilling block untouched.
## After (same viewport, same instrumentation)
| metric | before | after |
|---|---|---|
| grid-only 5s diff, mean \|d\| | 0.48 | **1.82** (3.8×) |
| grid-only, pixels changed | 1.5% | **6.1%** |
| grid-only bottom-half | ≈0.07 (absent) | **1.43–1.47** (present + moving) |
| full 5s diff, pixels changed | 1.4% | **4.2%** |
| full quadrant means (TL/TR/BL/BR) | 3.07 / 1.23 / 0.70 / 2.92 | 2.92 / 2.11 / 1.69 / 2.48 |
Motion is now distributed across the whole viewport (drift everywhere +
gentler corner breathing) instead of concentrated in the two glow
corners; after frames (`after_frame1/2.png`, 5s apart) confirm no static
frame. Visual check of `after_frame1.png`: grid clearly visible, still
subtle — no text/contrast impact, no jank (compositor-friendly
`background-position`/`opacity`/`transform` only), no 360px overflow
(layers remain `fixed; inset: 0`; `test_responsive_polish.py` 7/7 green).
## Screenshots
`before_frame1/2.png` (5s pair), `before_glow_only.png`,
`before_grid_only.png`, `before_occlusion_probe.png`,
`before_gridonly_frame1/2.png` (isolated grid pair), `after_frame1/2.png`
(5s pair), `after_gridonly_frame1/2.png`, `diagnosis_log.txt` (timeline
samples), `diagnosis_diff.txt` (all diff numbers).
## Tests (task 01)
- New: `tests/unit/test_background_animation.py` (10 source pins on the
final values + the no-occlusion + no-blur/zero-JS anchors) — green.
- `uv run pytest --cov=app` — 329 passed, app coverage 99%.
- `uv run ruff check . && uv run pyright` — clean.
- Regressions (isolated): `test_dark_tech_theme.py` 6/6,
`test_responsive_polish.py` 7/7.
---
# Task 02 — E2E story suite, story file, validation, commit
## Story gate: `tests/e2e/test_background_animation.py` (5 tests)
The layers are CSS pseudo-elements, so the suite asserts via computed
style + the Web Animations API. **Implementation note (deviation from
the task file, documented per the plan's "found, not guessed" rule):**
the task assumed Chromium reports pseudo-element CSS animations through
`element.getAnimations()` — verified false on Chromium 151
(`document.body.getAnimations()` returns `[]` for pseudo-layers); the
animations ARE enumerated by `document.getAnimations()`, with
`animationName` / `playState` / `currentTime` intact. The suite uses the
document-level list and matches on `animationName`. A second finding:
headless Chromium starts the document animation timeline ~1.4s after
navigation (currentTime stays 0 until then), so `test_animations_advance`
polls until both timelines are alive before sampling.
1. `test_grid_layer_animation_running` — computed `body::before`:
`animationName = bg-grid-drift`, timing `linear`, iterations
`infinite`; matching `CSSAnimation` with `playState === "running"`.
2. `test_glow_layer_animation_running` — same for `body::after`
(`bg-glow-breathe`); `playState === "running"`.
3. `test_animations_advance` — `currentTime` of both layers advances by
≥200ms over a 500ms wait — the timelines are truly running, not
paused (no static frame).
4. `test_background_layers_contracts` — both pseudo-elements: `position:
fixed`, `z-index: -1`, `pointer-events: none`, `inset: 0` (0px on all
four edges); `<html>` computed background `rgb(10, 14, 23)` (canvas
stays on html), `<body>` computed background `rgba(0, 0, 0, 0)`
(no occlusion).
5. `test_no_horizontal_overflow_with_layers` — 360px viewport:
`documentElement.scrollWidth <= clientWidth` (phase-07 pin,
replicated locally).
## Results (all commands, final state)
| command | result |
|---|---|
| `uv run pytest tests/e2e/test_background_animation.py -v --no-cov` | **5 passed** (run 4× — stable) |
| `uv run pytest tests/e2e/test_dark_tech_theme.py -v --no-cov` | **6 passed** (isolated) |
| `uv run pytest tests/e2e/test_responsive_polish.py -v --no-cov` | **7 passed** (isolated) |
| `uv run pytest` | **329 passed** |
| `uv run pytest --cov=app --cov-report=term-missing` | 329 passed, **app coverage 99%** (≥90% gate; unchanged from task 01) |
| `uv run ruff check . && uv run pyright` | **clean** (0 errors) |
Prerequisite honored: `podman compose up -d db` — `brain-of-reese_db_1`
Up (healthy) throughout.
## UI Structure Check (AGENTS.md rule 5)
- **Layers behind content:** E2E `test_background_layers_contracts`
pins `position: fixed`, `z-index: -1`, `pointer-events: none`,
`inset: 0` on both pseudo-layers in the live viewport (plus the
html-canvas/body-transparent no-occlusion pair).
- **No text/contrast impact:** the fix touches only line alpha, mask
radius, and glow opacity — no palette token, no text on the layers,
all WCAG AA pairs unchanged (phase-08 suite green).
- **No 360px overflow:** E2E pin `scrollWidth <= clientWidth` at
360×740 — live check `scrollWidth=360 clientWidth=360`; the
`fixed; inset: 0` layers add no width (phase-07 suite green).
## Final screenshots (fresh server, headless Chromium 151)
- `final_frame1.png` / `final_frame2.png` — 1280×800, 3s apart: grid
drift visible across the viewport (live sample at frame 2:
`background-position 2.55px` after ≈3.5s ≈ 0.73px/s — the spec'd
one-cell/60s drift), glow corners breathing gently.
- `final_360px.png` — 360×740, no horizontal overflow.
- Task-01 before/after evidence (root-cause + fix delta) in the same
directory (`before_*`, `after_*`, `diagnosis_log.txt`,
`diagnosis_diff.txt`).
## Story file
`.agent/user_stories/background-animation.md` — goal, bug report
verbatim (TODO.md L5), owner-confirmed A3 decisions, found root cause,
fix table, acceptance criteria, UI structure, Playwright mapping table.
@@ -1,120 +0,0 @@
# Phase 23 report — Containerfile: build the whole app image again
**Date:** 2026-08-24 · **Source:** `TODO.md` L6 ("Fix Containerfile build
not working") · **Story:** `.agent/user_stories/containerfile-build.md`
## Root cause (verified, not guessed — logs in this directory)
1. **Build failure:** phase 19's absolute `import … from
"/assets/header.js"` is resolved by esbuild as the *filesystem* path
`/assets/header.js` → `✘ [ERROR] Could not resolve "/assets/header.js"`,
reproduced with the exact pinned esbuild **0.25.5**
(`repro_esbuild_0.25.5.log`, `repro_podman_build_before_fix.log`).
2. **Under-shipped image:** stage 1 bundled only `app.js` +
`sources.js` and copied only `index.html` + `sources.html` —
`document.html` / `login.html` / `document.js` / `login.js` /
`markdown.js` were all missing from the image.
3. **Double-evaluation trap:** the four direct `<script
src="/assets/header.js">` tags would evaluate the header module
twice in the image (bundled copy + raw module) → duplicate sign-out
listener, double init.
4. **Second blocker (task 01):** a local `npm install` of esbuild left
the binary off the next `RUN`'s PATH (`esbuild: not found`) → the
stage installs it globally.
## Fix
| piece | after |
|---|---|
| page-script header import | `from "./header.js"` (relative — owner-confirmed over an esbuild alias) |
| direct header.js `<script>` tags | **removed from all four pages** (the hoisted import covers evaluation order) |
| esbuild install | `npm install --no-audit --no-fund -g esbuild@0.25.5` (pinned, on PATH) |
| stage-1 bundles | `app.js`, `sources.js`, `document.js`, `login.js` |
| stage-1 minifies | `styles.css` + `markdown.js` (no `--bundle` — classic script) |
| stage-1 copies | all four HTML pages into `/out` |
## Build gate — `podman build` (task 01, `podman_build_after_fix.log`)
```
[1/3] STEP 3/5: RUN npm install --no-audit --no-fund -g esbuild@0.25.5
[1/3] STEP 5/5: RUN mkdir -p /out/assets && esbuild ./assets/app.js --bundle … && … && cp ./index.html ./sources.html ./document.html ./login.html /out/
../out/assets/app.js 14.3kb ⚡ Done in 2ms
../out/assets/sources.js 2.3kb ⚡ Done in 2ms
../out/assets/document.js 2.4kb ⚡ Done in 2ms
../out/assets/login.js 1.5kb ⚡ Done in 2ms
[3/3] COMMIT brain-of-reese/app:phase23test
Successfully tagged localhost/brain-of-reese/app:phase23test
39f5563a1b101b274d81d86aa06f86a4d6d1bba453e8dba40d6f48f987537019
BUILD_EXIT=0
```
## Image smoke (task 01, `image_smoke.log`)
Throwaway Postgres 17 + pgvector (`bor-smoke-db`, isolated network),
production env, teardown done:
- app boots: migrations applied, uvicorn up, `/api/health` →
`{"status":"ok","db":"up",…}`
- pages: `GET /`, `/sources.html`, `/document.html`, `/login.html` → **200**
- assets: `app.js`, `sources.js`, `document.js`, `login.js`,
`markdown.js`, `styles.css` → **200**
- single-evaluation: raw `/assets/header.js` → **404** (expected — the
header code ships inside each page bundle); all four bundles are
minified and contain the header markers (`sign-out-btn`,
`/api/logout`)
- No CDN rule: `cdn_violations=0` across all four served pages
- `direct_header_script_tags=0`
- → **`SMOKE_PASS`**
## Task 02 — hermetic coverage pin + validation
`tests/integration/test_containerfile_assets.py` (6 tests, no podman /
no network / no DB — parses `Containerfile` + `frontend/` as text):
1. `test_every_html_page_is_copied_into_stage1` — cp set == pages on disk
2. `test_every_local_asset_reference_is_produced` — every `assets/`
src/href has a stage-1 producer
3. `test_page_module_scripts_are_bundled` — HTML module set == esbuild
`--bundle` input set
4. `test_header_module_is_imported_not_directly_loaded` — no direct
header.js tag anywhere; every page script imports `./header.js`
relatively
5. `test_markdown_js_is_a_produced_classic_script` — minified without
`--bundle`; no top-level import/export in the source
6. `test_esbuild_stays_pinned` — concrete `esbuild@X.Y.Z`, no floating
spec
Mutation-checked while authoring: removing `login.html` from the cp
line (→ test 1 fails), `esbuild@latest` (→ test 6 fails), deleting the
`markdown.js` line (→ tests 2+5 fail), deleting the `document.js`
bundle line (→ test 3 fails), re-adding a direct header.js tag (→
tests 3+4 fail). All pins fire.
### Test / lint / coverage results (exact commands)
| command | result |
|---|---|
| `uv run pytest tests/integration/test_containerfile_assets.py -v --no-cov` | 6 passed |
| `uv run pytest tests/e2e/test_smoke.py -v --no-cov` (isolated) | **3 passed** (`e2e_smoke.log`) |
| `uv run pytest tests/e2e/test_shared_header.py -v --no-cov` (isolated) | **6 passed** (`e2e_shared_header.log`) |
| `uv run pytest tests/e2e/test_chat_persistence.py -v --no-cov` (isolated) | **4 passed** (`e2e_chat_persistence.log`) |
| `uv run pytest` | **335 passed** (329 → 335: the six new pins) |
| `uv run pytest --cov=app --cov-report=term-missing` | **335 passed; TOTAL 1109 stmts, 14 missed, 99%** (≥ today's 99%; >90% gate) |
| `uv run ruff check . && uv run pyright` | **All checks passed! / 0 errors, 0 warnings** |
Dev-server behavior unchanged: the tag removal is covered by the three
isolated E2E regressions above (browser-level header init, sign-out
binding exactly once, chat persistence across navigation).
## Decisions
- Relative imports over an esbuild alias (owner-confirmed A4-1).
- Direct header.js tags removed over shipping a raw header.js
(owner-confirmed A4-2 — single module evaluation).
- No Playwright suite for this phase (build/infrastructure): the gate
is the hermetic integration pin + the real `podman build` + image
smoke recorded above + the dev-server E2E regressions (A16 note in
the phase plan).
- No LOCKED anchor changed (A11 No-CDN and A16 honored).
- Build-evidence image `localhost/brain-of-reese/app:phase23test` kept
locally.
@@ -1,97 +0,0 @@
# Phase 25 — Background: No Motion, Only Fading Light
**Owner report (2026-08-25, chat, verbatim):** "It should be smooth,
fluxuating, dimming and brightening, but not moving. Different bright
spots should slowly fade in and out." **Story:**
`.agent/user_stories/background-no-motion.md`.
## Root causes (phase-22 measurements as context)
1. **"Jitters down and to the right every second" = the grid drift.**
`bg-grid-drift` translated the 44px grid 44px per 60s (≈0.73px/s)
diagonally down-right — exactly the reported direction. A 1px line
moved sub-pixel by sub-pixel rasterizes with per-frame stepping, not
smooth motion, so it reads as a once-per-second jitter. Phase 22
had made that drift *visible* (60% line alpha + widened mask;
measured 1.82/765 mean pixel change over 5s in the grid zone vs
0.48 pre-phase-22) — that is precisely why it now reads as jitter.
Confirmed in the `before.png`/`before_4s.png` pair below: a
grid-only region far from every glow changes by 2.98/765 mean over
4s (4.7% of its pixels) — the grid itself is moving.
2. **"Slowly blinks brighter and darker" = the whole-layer breathe.**
`bg-glow-breathe` swung the entire glow layer's opacity 0.85↔1 over
14s (alternate) plus `scale(1)↔scale(1.05)` (a faint zoom). One
synchronized pulse of the whole background reads as a blink; the
before pair's uniform change across both glow corners (mean
2.19/765 over the full frame) is that single pulse.
## Design change (styles.css — pure CSS, zero JS, no blur, palette
untouched; task 01)
| property | phase 22 | phase 25 |
|---|---|---|
| grid (`body::before`) | `bg-grid-drift 60s linear infinite` (0→44px, 0.73px/s) | **static** — animation removed, `bg-grid-drift` deleted (44px cells, 60% `--line` 1px lines, widened mask kept) |
| glow | one whole-layer `bg-glow-breathe` (14s, opacity 0.85↔1 + scale 1↔1.05) | **three independent spot layers, opacity-only fades** |
| spot A `body::after` | indigo + cyan on one layer | indigo `rgb(109 120 242 / 0.14)` 56rem @ 12%/8% — `bg-glow-a` **26s** ease-in-out infinite, low 0.25 |
| spot B `html::before` | — | cyan `rgb(34 211 238 / 0.10)` 60rem @ 88%/92% — `bg-glow-b` **34s** ease-in-out **−12s** infinite, low 0.20 |
| spot C `html::after` | — | indigo `rgb(109 120 242 / 0.09)` 52rem @ 14%/86% — `bg-glow-c` **42s** ease-in-out **−23s** infinite, low 0.15 |
| keyframes | `bg-grid-drift` (background-position), `bg-glow-breathe` (opacity + transform) | `bg-glow-a/b/c` — **opacity only** (0%/100% low → 50% 1) |
| reduced motion | stills the two body layers | stills **all four** layers |
`html::before`/`html::after` join as background layers: `<html>` is
the root stacking context, so their `z-index: -1` pseudo-elements
paint above the `var(--bg)` canvas and below the transparent,
non-stacking `<body>`'s content (verified live — E2E test 6). 26/34/42s
with negative delays (LCM 4641s) keep the cycles out of phase, so the
composite pattern effectively never repeats within a viewing session.
No LOCKED anchor changed (A11 pure CSS / zero JS / no CDN / no new
assets; no `filter: blur`).
## Before / after evidence (1280×800 headless Chromium, shots ~4s
apart; `.agent/screenshots/25_background_no_motion/`)
- `before.png` / `before_4s.png` (captured **pre-change**, task 01):
down-right grid jitter + the uniform whole-layer pulse.
- `after.png` / `after_4s.png` (captured post-change, task 02):
brightness changes; the grid is bit-identical.
Region diffs (pure-stdlib PNG decode; "changed" = pixels with
per-channel Δ sum > 12/765):
| region | before pair: changed / mean\|d\| | after pair: changed / mean\|d\| |
|---|---|---|
| full frame | 42,316 (4.1%) / 2.190 | 8,357 (0.8%) / 1.754 |
| grid-only patch (900,30)–(1250,120) | 1,484 (4.7%) / **2.979 — the grid moves** | 0 (0.0%) / **0.000 — bit-identical** |
| spot A center (154,64) patch | 406 (4.1%) / 3.198 | 4,018 (40.2%) / 6.595 — mid-fade |
| spot C clip (0,500)–(500,800) | 6,469 (4.3%) / 1.274 | 0 / 3.247 — gentle fade, no >12/765 steps |
| spot C center (179,688) patch | 63 (0.6%) / 0.789 | 0 / 6.003 — visibly brightening |
| spot B clip (780,500)–(1280,800) | 5,784 (3.9%) / 2.037 | 0 / 1.211 — slow fade at this phase |
Read: after the change, the only region that moves pixel-by-pixel is
the light itself (spot A's 26s cycle passes through its steepest
mid-fade in the 4s window); every region that the grid drift used to
scrub through is now byte-identical — no positional shift anywhere.
## E2E story gate (task 02) — `tests/e2e/test_background_no_motion.py`
8 tests, green in isolation (see "Results"). The deterministic
no-movement proof is test 3: a live Chromium walk of
`document.styleSheets` shows the set of properties declared across
every frame of every `bg-*` @keyframes rule is exactly `{opacity}`.
Regression suites adapted to the new contract:
`tests/e2e/test_background_animation.py` (grid static + `bg-glow-a`
running + the three spot timelines advance + 4-layer contracts + 360px
pin) and `tests/e2e/test_dark_tech_theme.py`
(`test_animated_background` → static grid + 26/34/42s spots;
`test_reduced_motion_honored` → all four layers stilled).
## UI Structure Check (AGENTS.md rule 5)
- **Layers behind content:** E2E test 6 pins `position: fixed`,
`z-index: -1`, `pointer-events: none`, `inset: 0` on all four
pseudo-layers live, plus the html-canvas / body-transparent
no-occlusion pair (`rgb(10, 14, 23)` / `rgba(0, 0, 0, 0)`).
- **No text/contrast impact:** the change touches only background
layers (no palette token, no text on the layers); the phase-08
contrast suite stays green.
- **No 360px overflow:** E2E test 8 — `scrollWidth <= clientWidth` at
360×740 with all four `fixed; inset: 0` layers live.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 134 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 140 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 112 KiB

Some files were not shown because too many files have changed in this diff Show More