Compare commits

..
143 Commits
Author SHA1 Message Date
ducoterra bef24e05e2 phase: 123_chat_image_questions
Build and Push Containers / build-and-push-app (push) Successful in 1m54s
Build and Push Containers / build-and-push-db (push) Failing after 13s
All gates green. Verification complete.

**Phase 123 — final verification pass (all 4 tasks already in `complete/`)**

- Verified the full implementation is in the working tree: `app/api/chat_images.py` (upload/serve pair), `ChatRequest.image`/`ChatMessage.image` (path-validated, omitted-when-None), toggle-off + stale-file hinted error frames, `build_user_content` multimodal build at both sites (chat.py deflected branch + `run_agent`), config-gated composer attach/preview/upload-then-send, restore + shared rendering, CSP `img-src 'self' data:` carve-out, mock-LLM capture buffer.
- `uv run pytest` → **2796 passed**, exit 0 (unit + integration).
- `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (29/4615 missed; phase-123 modules 99–100%).
- `uv run pytest tests/e2e/test_chat_image_questions.py -v --no-cov` → **5 passed** in isolation.
- `uv run ruff check . && uv run pyright` → clean (0 errors).

**Completion criteria:** (1) attach→send→multimodal text+image to the model, bubble/reload/shared all render it, saved chat stores the PATH with `"base64" not in json.dumps(stored)` — **verified** (E2E tests 1–4 + integration round-trip); (2) `BOR_IMAGES=false` — control hidden, exact hinted error frame, zero model calls / no query_log row — **verified** (E2E test 5 + integration); (3) text-only byte-identical (`content` stays a plain `str`) — **verified** (unit + integration); (4) all gates green — **verified**; (5) commit + phase move — left to the harness per pipeline rules (no `git add`/`commit` run).

No defects found; no live-infrastructure changes (repo + local dev DB only). **Next pending phase: none** — 123 is the last phase in `todo/`.
2026-09-25 05:19:18 -04:00
ducoterra a19d78d284 phase: 122_image_documents
Build and Push Containers / build-and-push-app (push) Successful in 1m57s
Build and Push Containers / build-and-push-db (push) Failing after 13s
**Phase 122 (image documents) — final verification pass: all green. No code changes were needed; defects found: none.**

**Verified (implementation already complete in working tree, reviewed end-to-end):**
- Toggle (`BOR_IMAGES`/`BOR_IMAGE_EXTENSIONS`/`BOR_IMAGE_DIR`, off by default) + `GET /api/config` `images` flag
- Ingest: bytes digest, `image_dir` persistent copy, `content = summary = vision description` (chat-model call; only text embedded), fail-soft skip + `images_failed` counter
- Serve/display: `/api/documents/{id}/image` route (404 matrix), viewer `<img>` + description, Sources 48px lazy thumbnails, chat inline source figure (alt = summary), agent `read` marker
- Prune guard: images-off syncs never prune `is_image` docs

**Test / lint / coverage (exact commands & outcomes):**
- `uv run pytest` → exit 0 (green; note: pytest 9.1.1 `-q` omits the final count line in output — exit code authoritative)
- `uv run pytest --cov=app --cov-report=term-missing` → **2715 passed, exit 0, TOTAL 99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → "All checks passed!" / "0 errors, 0 warnings, 0 informations"
- `uv run pytest tests/e2e/test_image_documents.py -v --no-cov` → **4 passed, exit 0** (isolation)

**Completion criteria:** (1) images=true → described/embedded/displayed docs: ✅ (E2E + integration) · (2) images=false byte-identical + image docs survive sync: ✅ (E2E negative app + unit/integration) · (3) viewer + chat rendering with alt text; failed description skips + logs, sync completes: ✅ · (4) test/lint/coverage gates: ✅ · (5) commit + phase move: deferred to harness per this pass's rules (working tree left uncommitted).

**Notable deviation (pre-existing, documented in code):** image route uses `require_user` (phase-79 posture, same gate as the document content endpoint) rather than the phase text's "public" parenthetical — matches the endpoint it mirrors.

**Next pending phase:** `123_chat_image_questions`.
2026-09-25 01:54:23 -04:00
ducoterra 0f77e9a876 phase: 121_git_source_tokens
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Build and Push Containers / build-and-push-db (push) Failing after 14s
**Phase 121 final verification pass — all green** (all 4 tasks already in `complete/`; verified, no defects found, no changes needed)

- Verified implementation vs phase design: migration `0021` (reversible, round-tripped via `alembic downgrade base` + `upgrade head` → head `0021`), `GitSource.token` column, `normalize_credential`/`clone_url_for`/`sanitize_url`, clone callers switched (`sync.py`, `import_docs.py`), masked token fields in add form + editor, `extra="forbid"` output shapes
- Tests: `uv run pytest` → 2662 passed, 0 failed (exit 0); `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (≥90% gate)
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov` → **4 passed**

Completion criteria:
1. Private repo (UI add or pasted embedded-token URL) clones with injected token; token absent from every API response, page text, title attr, and full HTML — **PASS** (integration raw-JSON assertions + E2E `_assert_token_nowhere`)
2. Legacy embedded-token rows still clone from stored URL; output sanitized — **PASS** (`test_sync_legacy_row_clones_with_original_stored_url`, `test_get_masks_legacy_embedded_token_row`, env-fallback masking)
3. Public/local sources byte-identical — **PASS** (verbatim-URL + no-userinfo-unchanged tests)
4. pytest / coverage / ruff / pyright — **PASS** (see above)
5. Commit + phase move — harness responsibility; task files already in `complete/`, changes left in working tree (no commit made, per protocol)

Notable: no deviations; DB left at head, functional. Next pending phase: **122_image_documents** (then 123_chat_image_questions).
2026-09-24 20:51:39 -04:00
ducoterra 3a0fc3db05 phase: 120_failed_turn_retry
Build and Push Containers / build-and-push-app (push) Successful in 3m5s
Build and Push Containers / build-and-push-db (push) Failing after 2m13s
All verification complete. Final report:

**Phase 120 — Failed-turn retry: verification pass (all 3 tasks were done; final verification + 1 regression fix)**

**Verified:** `ChatMessage.failed`/`error` (≤500, `extra="forbid"` intact); `finalizeFailedTurn` funnel on the 3 failure paths (catch-else, stream-drop guard, zero-frame fallback) with `failed: true` + capped detail + `markLastRetryable`; `appendFailedNote` restore branch (Save-as-doc/Tune excluded); `showErrorBanner`/`retryLastTurn` byte-pinned untouched; only the three paths persist `failed: true` (grep + unit pin); no test asserts the old broken behavior.

**Defect found & fixed (rule 7):** a real navigate-away mid-turn let the browser's teardown fetch rejection (TypeError, not AbortError) leak into the failed funnel, persisting a phantom failed brain record — `test_sources_midstream_bug.py::test_no_orphan_brain_message_when_navigated_before_first_token` failed (2 `.msg` after reload) and violated the phase-20 navigate-away convention. Fixed: turn-scoped `leftThePage` flag (set unconditionally on `pagehide`, reset in `runTurn`) skips the funnel in the catch-else branch; pinned by new unit test `test_navigate_away_is_not_a_failed_turn`. No phase-overview/PLAN/todo/complete files touched; no commits made.

**Gates (exact):**
- `uv run pytest` → 2577 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL 4271 stmts, 99% (>90%)
- `uv run pytest tests/e2e/test_failed_turn_retry.py -v --no-cov` → 4 passed (isolated)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- Regression E2E, isolated: `test_sources_midstream_bug.py` 6/6 (was 5/6); `test_llm_retry`/`test_tool_scaffolding_guardrails`/`test_stop_generation`/`test_navbar_refresh` 17/17

**Completion criteria:** (1) network error → banner + in-bubble Retry, re-ask without re-typing ✅ (E2E A); (2) refresh restores failed bubble + working Retry, no "new chat" ✅ (E2E C); (3) stopped/successful turns byte-identical ✅ (negative E2E, stop suite, byte-identity units); (4) pytest/coverage/lint/types ✅; (5) commit + phase move — left to the harness per pass rules.

**Notable:** deviation = the regression fix above (a navigation is not a failed turn; phase-20 partial-persist convention restored). Next pending phase: `121_git_source_tokens`.
2026-09-24 18:50:36 -04:00
ducoterra 0ff1f8c4d6 chore(agent): phase roadmap from TODO.md — 4 phases (120–123)
Protocol B append: failed-turn retry (L3–4), git source tokens (L5),
image documents (L6 ingest), chat image questions (L6 chat side).
TODO.md items now live in .agents/phases/todo/ and the file is cleared.

LLM-Generated: true
2026-09-24 16:27:44 -04:00
ducoterra a5b63f83ad phase: 119_name_signal_read_chips
Build and Push Containers / build-and-push-app (push) Successful in 2m1s
Build and Push Containers / build-and-push-db (push) Successful in 18s
All verification complete. Final report:

**Phase 119 final verification pass — all criteria verified, one stale pin fixed.**
- Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry.
- Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged.
- New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2.
- Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed).
- Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met).
- Next pending phase: **none** — `todo/` holds only phase 119.
2026-09-16 15:50:48 -04:00
ducoterra 795fb56425 add phase 119
Build and Push Containers / build-and-push-app (push) Successful in 15s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-16 10:05:52 -04:00
ducoterra 9820c361b0 phase: 118_summary_seed_context
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 14s
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.

- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
2026-09-16 06:57:49 -04:00
ducoterra 21aad84a6d chore(agent): phase roadmap from TODO.md — phase 118 summary seed context (top-5 summary seeding, all-doc summaries)
Build and Push Containers / build-and-push-app (push) Successful in 17s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-15 22:43:01 -04:00
ducoterra c851d1a1c5 phase: 117_tool_call_compact
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification complete — every gate green. Final report:

**Phase 117 final verification pass — all criteria verified** (tasks 01–04 were already in `complete/`; I re-verified the full gate myself, no code changes needed)
- **Verified:** `appendToolLine` wraps lines in `details.tool-calls-disclosure` (open live, `closeToolCalls` at delta/done/stop + restore-closed, count via textContent); `addToolLines` parity (closed); `.tool-call` deboxed / `code` dechipped; new disclosure/summary CSS; only the one pinned CSS assert updated
- `uv run pytest` → **2466 passed**
- `uv run pytest --cov=app --cov-report=term-missing` → **99%** on `app/` (>90% floor)
- E2E in isolation: `test_tool_call_compact.py` **4 passed** (fold/expand/deboxed-flow/restored-folded); `test_agent_document_tools.py` **4 passed**; `test_big_read_progress.py` **4 passed** (live disclosure open + `.tool-elapsed` visible); `test_thinking_display.py` **5 passed**; `test_smoke.py` **3 passed**
- `uv run ruff check . && uv run pyright` → clean (pyright 0 errors, 0 warnings)
- Completion criteria: pins 1–4 ✅ (E2E above); live "calling tool" state byte-identical ✅ (`test_frontend_tool_states.py` 10 passed incl. `test_calling_tool_label_strings`); scope ✅ — `git diff` limited to the 3 frontend assets + 2 unit tests + E2E story + phase files, **0 changes in `app/`**
- Notable: `TODO.md` carries a **pre-existing uncommitted owner edit** (new retrieval-context idea) already in the tree before this pass — left untouched, will ride along in the harness commit; pre-existing committed quirk: the brain-wave commit's styles.css comment mislabels itself "Phase 117" (cosmetic, out of this diff)
- No commit made (harness commits + moves the phase); **next pending phase: none** — `todo/` holds only phase 117
2026-09-15 18:26:38 -04:00
ducoterra 2ac3fc89c2 feat(ui): swap the in-turn loader for a brain-wave sweep left of the send button
Build and Push Containers / build-and-push-app (push) Successful in 2m11s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Phase 117 (owner request, live-mockup-confirmed): the phase-109 3-dot
cue becomes a compact ECG trace (49px, P/QRS/T) with a brand sweep
traveling the path (bwdraw, 42/140 dash segment, 0.9s loop), the
loader repositioned left of the button so its appearance never shifts
it. setUiState stays the sole owner of the loader's hidden attribute;
the reduced-motion variant stills the sweep. Unit + lifecycle-E2E pins
updated for the new contract.
2026-09-15 11:55:59 -04:00
ducoterra adf31a4a35 phase: 116_modal_scrollbar_theme
Build and Push Containers / build-and-push-app (push) Successful in 2m1s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification complete — every gate re-run independently and green. Final report:

**Phase 116 final verification pass — all green (work was already in the tree; verified, nothing to fix)**

- **Verified** `styles.css`: `--scrollbar-thumb`/`--scrollbar-track` tokens in `:root` (color-mix derivations — follow the saved theme, not in BUILTIN_COLORS; contrast 3.73:1 ≥ 3:1, recomputed independently) + scoped rules `.doc-modal .doc-raw` (overflow-x already owned by the element)
- **Verified** E2E test 8 `test_raw_code_block_themed_scrollbar`: overflow, scroll action, computed-style theme assertions, screenshot
- **Screenshots visually confirmed**: headed `doc_modal_scrollbar_themed_1280.png` shows muted rounded thumb on dark track, no native light-gray bar; headless suite shot records scrolled state (headless Chromium paints no bar UI — documented in test docstring)

**Test / lint / coverage (re-run by me):**
- `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` → **8 passed** (isolation)
- `uv run pytest --cov=app --cov-report=term` → **2457 passed, app/ 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_dark_tech_theme.py -v --no-cov` → **6 passed** (no token collision)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:**
1. Modal scrolls horizontally, scrollbar themed, screenshot shows no native bar — ✅
2. No global scrollbar change (all new selectors under `.doc-modal`; grep-confirmed), regressions green — ✅
3. pytest green, coverage >90%, e2e isolated green, ruff+pyright clean — ✅
4. `--no-gpg-sign` commit + phase move — harness job (left uncommitted in working tree, per rules)

**No defects found; no deviations. Next pending phase:** none — `todo/` contains only phase 116.
2026-09-15 06:11:38 -04:00
ducoterra 990c8adf13 phase: 115_doc_draft_discard
Build and Push Containers / build-and-push-app (push) Successful in 2m12s
Build and Push Containers / build-and-push-db (push) Successful in 14s
All green — this was the final verification pass; everything from the four completed tasks was already in the working tree and verified.

**Phase 115 — Doc drafts: Discard + DELETE route + title fix — verification report**

- Verified all 4 task deliverables present: DELETE route (`app/api/doc_drafts.py`), Discard UI (`doc-edit.html` + `doc-edit.js` + `.discard-draft` CSS), title fix (`defaultDocTitle(wrap)` pairing + `saveAsDoc` call site), and all test pins (integration, frontend unit, E2E). No code changes needed.
- **Completion criteria:**
  1. ✅ Orphaned draft discardable from edit screen; row gone — `test_delete_removes_row_and_invalidates_token` (204 → GET 404), unknown-token 404, admin-gate 403 on all routes, E2E `test_discard_draft_from_edit_screen` all pass.
  2. ✅ Title after retry redo = redone answer's own question — E2E `test_save_title_is_the_redo_question_after_retry` passes.
  3. ✅ Push flow byte-identical — `git diff` shows only the new DELETE route + module docstring; all 7 existing push tests green.
  4. ✅ `uv run pytest --cov=app` → **2457 passed**, app coverage **99%** (>90%); `uv run pytest tests/e2e/test_save_doc_session.py -v --no-cov` → **4 passed**; `uv run ruff check .` → clean; `uv run pyright` → 0 errors.
  5. ⏳ Commit + phase-dir move left to the harness (per executor rules, no `git` run; all changes left in the working tree).
- No defects found; no deviations.
- Next pending phase: none in `todo/` other than this one (`115_doc_draft_discard` is the last).
2026-09-15 05:35:05 -04:00
ducoterra 3846f26a58 phase: 114_embed_question_length
Build and Push Containers / build-and-push-app (push) Successful in 2m6s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification passes complete — the phase was already fully implemented in the working tree, and every gate is green. No defects found; no code changes were needed.

**Final verification pass — Phase 114 (embed question length):**
- Verified truncation: `chat.py:459` embeds `request.message[:settings.embed_question_max_chars]` (default 1200, `BOR_EMBED_QUESTION_MAX_CHARS`, `>0` validator); full question still reaches the LLM prompt/log.
- Verified error mapping: `EmbeddingInputTooLargeError(EmbeddingError)` (byte-identical message) caught **before** `EmbeddingError` → no retry, terminal frame `detail="Question too long — trim it and re-ask."` + reachability-fine hint; `ChatErrorEvent.hint` additive.
- Verified frontend chain: frame `hint` → `err.hint` → `setUiState(error, …, {hint})` → `showErrorBanner(…, {retryable: true})` — hint replaces default `ERROR_HINT`, phase-111 `#banner-retry` button revealed. 4,000-char clamp untouched (A2).
- `uv run pytest tests/unit/test_embed_question_length.py -v --no-cov` → 21 passed
- `uv run pytest tests/e2e/test_embed_question_length.py -v --no-cov` (isolation, DB up) → 1 passed (4,000-char question → done, no banner)
- Regression: `test_llm_retry.py` 4 passed · `test_oneshot_llm_retry.py` 2 passed · `test_chip_sizing_question_cap.py` 6 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2444 passed, TOTAL **99%** (>90% gate)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:** (1) 4,000-char question embeds prefix + full prompt ✅ · (2) too-large → accurate frame + hint + Retry button ✅ · (3) reachability failure byte-identical (retries + old copy) ✅ · (4) all gates green ✅ · (5) commit/phase-move → left to the harness per instructions (no `git add`/`commit` run).
**Deviations:** none. **Next pending phase:** `115_doc_draft_discard`.
2026-09-15 04:16:55 -04:00
ducoterra 97d663d16d phase: 113_source_chip_quality
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 15s
All gates green — no defects found; this pass was verification only.

**Phase 113 final verification pass — report**

- Verified (no code changes needed): `select_documents_tiered` cited/related tiering + `select_documents` wrapper, `TurnPlan.related_docs`, `ChatDoneEvent.related` (additive, old payloads parse), `appendRelated` UI row (`.related-doc`, never `.source-chip`), done-frame + restore-path wiring, two settings with validators, `.env.example` entries
- `uv run pytest --cov=app --cov-report=term-missing` → 2422 passed, app/ coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov` (isolated) → 2 passed
- Regression E2E `test_retrieval_quality.py` + `test_honest_deflection.py` + `test_chat_rag.py` + `test_sources_midstream_bug.py` → 17 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors); `bash .agents/validate.sh` → "validation OK"

Completion criteria:
1. Single-doc question → exactly one `.source-chip` (E2E): ✅ passed
2. Weak 2nd doc only in de-emphasized related row, never `.source-chip` (unit + E2E): ✅ passed
3. Deflected turn → zero citation chips, weak hits in related row: ✅ passed
4. Full suite green, coverage >90%, isolated E2E green, lint/types clean: ✅ passed
5. `--no-gpg-sign` commit + phase dir move: left to harness per pass rules (task files already in `complete/`)

No deviations. Next pending phase: `114_embed_question_length`.
2026-09-15 03:11:05 -04:00
ducoterra 1374faf136 phase: 112_honesty_gate_weak_hits
Build and Push Containers / build-and-push-app (push) Successful in 2m15s
Build and Push Containers / build-and-push-db (push) Successful in 14s
**Phase 112 — final verification pass (all 4 tasks already complete in `complete/`):**

- Verified gate fix: `app/api/chat.py::plan_turn` — HIGH iff `best_cosine >= relevance_threshold` OR (`fts_hits > 0` AND `best_cosine >= lexical_support_floor`); `lexical_support_floor` (default 0.35, `BOR_LEXICAL_SUPPORT_FLOOR`, bounds-validated) in `app/config.py` + `.env.example`; A8 revision note (2026-09-14) in `.agents/PLAN.md`.
- Verified prompt contract: `app/rag/prompts.py` diff is docstring-only (dated owner-decision-iii entry); `tests/unit/test_prompt_lock.py` byte-pins PERSONA/TOOLS_SECTION/DEFLECT body (sha256+length).
- Verified README: L11 + L575 deflection copy refreshed; `grep "haven't done anything" README.md` → no hits; disclosed-answer behavior documented.
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → **2378 passed, 99% coverage (>90%)**; includes Mongolia-quadrant unit pins (fts>0 + cosine<floor → LOW).
- E2E in isolation: `uv run pytest tests/e2e/test_honest_deflection.py -v --no-cov` → **4 passed** (out-of-KB question: `deflected=true`, `sources==[]`, 2–3 suggestions); regression `test_chat_rag.py` + `test_retrieval_quality.py` → **7 passed**.
- Lint/types: `uv run ruff check .` → clean; `uv run pyright` → 0 errors.

**Completion criteria:** weak-FTS→LOW unit-pinned ✅ · no false citations + 2–3 alternatives E2E ✅ · prompts byte-identical (test-pinned) + README matches ✅ · suite/coverage/e2e/lint all green ✅ · commit + phase move → left to harness (no `git commit` run, per rules; changes in working tree).

**Deviations:** none. Next pending phase: `113_source_chip_quality`.
2026-09-15 00:37:38 -04:00
ducoterra 2683128876 phase: 111_chat_banner_retry
Build and Push Containers / build-and-push-app (push) Successful in 2m24s
Build and Push Containers / build-and-push-db (push) Successful in 14s
## Phase 111 Completion Report

**Implemented/Verified:**
- `#kb-banner` contains a `<button type="button" class="banner-retry" id="banner-retry">` (hidden by default, Retry label + SVG)
- `showErrorBanner(detail, opts)` reveals the button only when `opts.retryable` is true AND `lastBrainWrap` exists
- Turn-error path passes `{ retryable: true }`; all non-turn callers (share, save-doc, stale) remain text-only
- `clearErrorBanner()` re-hides the button
- `ERROR_HINT` changed from "Try again — …" to "If this persists, check the LLM is reachable."
- `.banner-retry` CSS styled as a pill (matching `.stale-regenerate` family)
- 12 source-assertion unit tests in `tests/unit/test_frontend_banner_retry.py`

**Test / Lint / Coverage:**
- `uv run pytest tests/unit/test_frontend_banner_retry.py -v --no-cov` → 12 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 2362 passed, 99% coverage
- `uv run ruff check .` → All checks passed
- `uv run pyright` → 0 errors
- `tests/e2e/test_llm_retry.py` → 4 passed (in isolation)
- `tests/e2e/test_smoke.py` → 3 passed (in isolation)

**Completion Criteria:**
- ✅ Retry button visible after failed chat turn, re-runs last question
- ✅ Non-turn callers show text-only banner (no button)
- ✅ pytest green, coverage >90%, ruff + pyright clean
- ✅ Phase dir to be moved by pipeline gate

**Next pending phase:** `112_honesty_gate_weak_hits`
2026-09-14 23:07:56 -04:00
ducoterra f37c517590 chore(agent): phase roadmap from TODO.md — 6 phases (111–116): banner retry, honesty gate, chip quality, embed length, draft discard, modal scrollbar
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 13s
2026-09-14 22:07:32 -04:00
ducoterra 2b75f3cc85 docs(todo): log live testing findings L1-L7 (brain.reeseapps.com session)
Interactive browser test session (15 tests) against the deployed
instance. Findings: L1 banner 'Try again' is plain text, L2 honesty
gate eagerness + stochastic deflection compliance, L3 unstyled modal
scrollbar, L5 recurring weak-hit 2nd source chip, L6 4000-char
question clamp exceeds the embed input cap (500 surfaced as
'unreachable'), L7 save-as-doc drafts have no discard path. Each
entry carries observed evidence, suggested fixes and acceptance.
2026-09-14 21:31:08 -04:00
ducoterra 9ce13b6290 add vulns.md
Build and Push Containers / build-and-push-app (push) Successful in 29s
Build and Push Containers / build-and-push-db (push) Successful in 13s
2026-09-14 16:01:58 -04:00
ducoterra 3a4035fc96 phase: 110_fix_sse_db_pool_exhaustion
Build and Push Containers / build-and-push-app (push) Successful in 2m14s
Build and Push Containers / build-and-push-db (push) Successful in 13s
---

**Phase 110 — Fix SSE DB Connection Pool Exhaustion (SEC-14-04): COMPLETE**

**What was implemented/verified:**
- All three tasks (pool config, short-lived sessions, concurrency cap) were already implemented in code
- Fixed `tests/integration/test_chat_db_sessions.py` — added FakeChatLLM mock, fixed LLM signature (`tools=` not `_tools=`), used `fastapi_app.dependency_overrides` instead of `client.app.dependency_overrides`
- Fixed `tests/e2e/test_chat_db_pool.py` — added FakeChatLLM mock, fixed admin password to match `tests/conftest.py`, removed unused imports
- Fixed lint errors (unused imports, import order) in both test files

**Test / lint / coverage results:**
- `uv run pytest` → 2350 passed, 1 warning, 56.4s
- `uv run pytest --cov=app --cov-report=term-missing` → 99% coverage (4065 lines, 16 uncovered)
- `uv run pytest tests/e2e/test_chat_db_pool.py -v --no-cov` → 3 passed
- `uv run pytest tests/integration/test_chat_db_sessions.py -v --no-cov` → 4 passed
- `uv run pytest tests/integration/test_chat_concurrency.py -v --no-cov` → 11 passed
- `uv run pytest tests/unit/test_db_pool_config.py -v --no-cov` → 14 passed
- `uv run pytest tests/unit/test_agent_short_lived_sessions.py -v --no-cov` → 7 passed
- `uv run ruff check .` → all checks passed
- `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:**
- [✓] `app/db.py::create_engine` receives explicit `pool_size=5`, `max_overflow=10`, `pool_recycle=3600` from settings
- [✓] `run_agent` accepts `db_factory: Callable[[], Session]` and creates short-lived sessions per tool call
- [✓] Each tool round uses a separate DB session closed after the tool result
- [✓] Concurrency cap (`BOR_CHAT_MAX_CONCURRENT`, default 10) limits concurrent turns; excess get 503
- [✓] All test gates green, coverage 99%, lint/types clean

**Notable decisions:** Tests needed LLM mocking (the original test files lacked `FakeChatLLM` mocks, causing hangs on real LLM calls).

**Next pending phase:** None — this is the last phase in `todo/`.
2026-09-14 15:55:13 -04:00
ducoterra 35d65d2f25 feat(rag): summarize single-document folders (MIN_DOCS_PER_FOLDER 2 → 1)
Build and Push Containers / build-and-push-app (push) Successful in 2m10s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Relax the phase-94 folder-summary scope rule from ≥ 2 documents to
≥ 1: a folder (or source root) is a candidate while ANY document
lives under it, so single-file folders and single-file source roots
get their own lite-written description. A row is now pruned only
when its folder loses its last document (vanishes from the
catalogue).

The constant is the single source of truth, so the flip propagates
to the generator's candidate set, the prune pass, the
missing_folder_summaries gap probe (the next sync self-heals the
new gaps), and the KB-tree summary_pending markers (1-doc folders /
sources now read "Summary pending" until their row lands).

Docstrings/comments across app/, scripts/import_docs.py, and the
E2E fixtures updated to the ≥ 1 wording. Unit + integration tests
updated to the new semantics (the pruned-below-minimum scenario is
now a folder losing its LAST doc; single-doc folders are pinned as
candidates/pending). Full suite: 2314 passed, app coverage 99%;
ruff + pyright clean; folder-summary E2E stories pass in isolation
(ls_tree_drilldown, sync_summary_visibility, kb_tree,
kb_tree_nav, document_dates, oneshot_llm_retry).
2026-09-14 08:57:57 -04:00
ducoterra 3a81793565 phase: 109_turn_progress_loader
Build and Push Containers / build-and-push-app (push) Successful in 4m39s
Build and Push Containers / build-and-push-db (push) Successful in 35s
All gates green. Final report:

**Phase 109 — final verification pass (all 3 tasks were already complete; no re-implementation needed)**

- Verified D15 (thinking-handler re-open `block.open = true` after `ensureThinkingBlock`; delta keeps closing; "never reopens" narrative removed; restore path still collapses) and D16 (single static `#turn-loader` in composer row, `aria-hidden`, sole `turnLoader.hidden = !inFlight` write in `setUiState`; CSS reuses `typing` keyframes + reduced-motion override after the main rule + provenance; global `[hidden]` guard intact; mock `TURN_PROGRESS_TRIGGER` repro marker; `app/` untouched — pure UI phase).
- Unit pins: `tests/unit/test_frontend_turn_loader.py` (9 tests) — green via full run.
- `uv run pytest tests/e2e/test_turn_progress_loader.py -v --no-cov` → **3 passed** (isolation, DB up): loader visible send→tool-gap→final-thinking, hidden after done; re-opened scratchpad carries `marker-thought-42`/`marker-final-thought-42`; `#send-status` carries state text, back to empty idle; reduced-motion context stills dots (computed `animation-name: none`) with loader still visible.
- Regressions in isolation: `test_thinking_display` 5 passed, `test_stop_generation` 3 passed, `test_big_read_progress` 4 passed, `test_loading_feedback` 5 passed.
- `uv run pytest` → **2314 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90%); `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings.
- All terminal paths (done→idle L2554, stream error→error L2543, timeout→error L2247) funnel through `setUiState` — never-stale by construction.
- No defects found; no code changes made in this pass. Commit + phase-dir move left to the harness per pipeline rules (working tree carries all changes).
- Next pending phase: `108_history_wire_check` (still in `todo/`; out of scope here).
2026-09-14 01:27:16 -04:00
ducoterra fbbd98d734 phase: 108_history_wire_check
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 12s
Final verification pass complete — everything independently re-verified green. Report:

**Phase 108 — History wire check (final verification pass; verdict: NO BUG, tests-only per D13)**

- Verified all 3 tasks complete in `.agents/phases/complete/108_history_wire_check/`; re-ran every pin and gate myself
- Layer 1 (unit): 2 owner-shape pins in `tests/unit/test_history.py` (2-turn keep-whole + A4 thinking mapping) — pass
- Layer 2 (integration): `test_endpoint_two_turn_history_reaches_the_llm` — captured LLM request = `[system, user Q1, assistant R1, user Q2]`, `history_msgs=2` — pass
- Layer 3 (E2E): `tests/e2e/test_history_wire_check.py` — owner's verbatim 3-message scenario, echo byte-exact (T2: `2 prior messages`+R1 tail; T3: `4 prior messages`+R2 tail) — pass
- `VERDICT.md` present in phase dir: layer outcomes + "no bug — model behavior/user error" + pin names

**Gates (exact results)**
- `uv run pytest --cov=app --cov-report=term-missing` → 2305 passed, TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_history_wire_check.py -v --no-cov` → 2 passed (isolation, DB up)
- `uv run pytest tests/e2e/test_llm_history.py -v --no-cov` (phase-74 regression) → 3 passed
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors

**Completion criteria:** all six met — no defects found; diff scoped to `tests/**`, `VERDICT.md`, `.agents/phases/**`, `.agents/reports/**` (no `app/`/`frontend/` changes, consistent with no-bug verdict). Per executor rules, no git commit made — left for the harness.

**Next pending phase:** `109_turn_progress_loader`
2026-09-13 23:57:54 -04:00
ducoterra 6bcee831ec phase: 107_git_full_history_dates
Build and Push Containers / build-and-push-app (push) Successful in 1m49s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All green — verification complete. Final report:

**Phase 107 — final verification pass (all tasks already in `complete/`)**

- **Verified:** `clone_or_pull` full-history fix (no `--depth` clone; `--is-shallow-repository` probe → one-time `git fetch --unshallow` self-heal → `git pull --ff-only`; D12 fail-loud) in `scripts/git_sync.py`; E2E suite `tests/e2e/test_git_source_dates.py` present; all five stale shallow/tip-date narratives corrected (no residual claims repo-wide; `docs_push.py` `--depth` fetches untouched as designed).
- **Unit+integration:** `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py tests/integration/test_import_docs_git.py tests/integration/test_sync_api.py -v --no-cov` → **66 passed** (D10 tip-date test replaced by `test_url_clone_yields_true_per_file_dates` + `test_existing_shallow_checkout_self_heals`; unit pins: clone argv w/o `--depth`, probe→unshallow→pull order, unshallow-failure → `GitSyncError`).
- **New E2E in isolation:** `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` → **4 passed** (old file 2020-06-15 ≠ tip file 2024-06-15 in `GET /api/docs`, Sources `Created`/`Updated` columns, viewer `Created` badge ISO title; folder `updated_at` subtree maxes).
- **Phase-106 regressions in isolation:** `test_document_dates.py` → 6 passed; `test_sync_button.py` → 3 passed; `test_git_sources_admin.py` → 6 passed.
- **Full gates:** `uv run pytest` → **2302 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90%); `uv run ruff check . && uv run pyright` → **clean, 0 errors**.
- **Completion criteria:** 1 ✅ non-shallow URL clone + true per-file dates · 2 ✅ self-heal / plain-pull / fail-loud · 3 ✅ both suites green, bug-pin replaced · 4 ✅ E2E green in isolation · 5 ✅ all regressions + full suite + coverage + lint · 6 ✅ narratives corrected · 7 — no commit made (harness override: changes left in working tree; task files already in `complete/`).
- **No defects found; no deviations.** Next pending phase: **108_history_wire_check**.
2026-09-13 23:21:39 -04:00
ducoterra addbd4ca08 add phase 107
Build and Push Containers / build-and-push-app (push) Successful in 34s
Build and Push Containers / build-and-push-db (push) Successful in 26s
2026-09-13 22:28:04 -04:00
ducoterra e2d08a95a9 chore(agent): phase roadmap from TODO.md, 2 phases (108 history-wire check, 109 turn progress loader) 2026-09-13 22:27:05 -04:00
ducoterra ee3efb28c9 phase: 106_document_dates
Build and Push Containers / build-and-push-app (push) Successful in 4m35s
Build and Push Containers / build-and-push-db (push) Successful in 14s
Everything is verified green. Final report:

**Phase 106 — Document dates (final verification pass; all 10 tasks already complete)**

- Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed.
- `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up)
- 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation**
- `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings**

**Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions.

- **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring).
- **Next pending phase:** none — `todo/` holds only this phase.
2026-09-13 19:28:05 -04:00
ducoterra cec819743d add phase 106 - document dates
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 14s
2026-09-13 12:18:01 -04:00
ducoterra 8476dc1e07 phase: 99_kb_tree_table_and_back_nav
Build and Push Containers / build-and-push-app (push) Successful in 1m55s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification is complete and green. Final report:

**Phase 99 — final verification pass (all 3 tasks already in `complete/`)**

- Verified the one-line Description clamp: `.kb-desc-cell` flex wrapper + `.kb-desc-text` ellipsis triad + `nowrap` column + in-cell Edit button + hover `title` + untouched level block (source diffs match D1; unit pins present)
- Verified the back-button breadcrumb nav: `applyTarget(target, push)` state-only `pushState`, no-duplicate gate, popstate adopt/reset, `bor:view-refresh` alignment before `loadTree()`, anonymous gate listener-free, router.js untouched (D2)
- `uv run pytest --cov=app`: **2180 passed**, coverage **99%** (>90% gate)
- `uv run ruff check .`: clean; `uv run pyright`: **0 errors, 0 warnings**
- `uv run pytest tests/e2e/test_kb_tree_nav.py -v --no-cov` (isolated, DB up): **4/4 passed** (back-walk, jump-then-back, fresh-nav/active-reclick, measured clamp incl. row-height parity ±4px, computed styles, full text in DOM/title/level block)
- Regression suites isolated: `test_kb_tree.py` 8✓, `test_ls_tree_drilldown.py` 3✓, `test_navbar_refresh.py` 7✓, plus `test_sync_summary_visibility.py` 3✓ (marker pin updated to `kb-desc-text kb-summary-pending` — the sanctioned class-pair change) and `test_edit_summaries.py` 4✓
- Criteria 1–4: **met** (each as above). Criterion 5 (atomic commit): left to the harness per executor rules — no `git add`/`commit` run; all changes left in the working tree
- No defects found in prior phases; no deviations. Stray `__pycache__/test_zz_smoke_tree_ui.*.pyc` is a leftover (no `.py` on disk) — inert, not touched
- Next pending phase: **none** — `todo/` will be empty once the harness moves this phase
2026-09-13 01:57:40 -04:00
ducoterra f665a83b1a phase: 98_sync_summary_visibility
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All verification complete. Final report:

**Phase 98 — Sync summary visibility: final verification pass** (all 5 tasks already complete; implementation verified against the design, no defects found, no code changes needed)

- **Implementation checked:** `SyncStatus` phase machine (4 new keys, terminal-keep counts), `on_progress` hook in `generate_folder_summaries`, `summary_pending` on `KbTreeSource`/`KbTreeFolder` + D3 rule in `build_kb_tree`, phase-aware sync labels + pending UI in `sources.js`, `.kb-summary-pending` CSS — all match decisions D1–D5.
- **Unit + integration:** `uv run pytest` → 2184 tests, 0 failed/errors (exit 0)
- **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **99%** on `app/` (criterion >90% ✓; `app/api/sync.py` and `app/rag/folder_summaries.py` at 100%)
- **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- **Phase E2E (isolation):** `uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov` → **3 passed** (phase machine, live label, pending markers + gap-fill self-heal)
- **Regression suites (each isolated, `--no-cov`):** test_kb_tree ✓, test_ls_tree_drilldown 3 ✓, test_sync_button 3 ✓, test_sync_upload_progress 4 ✓, test_oneshot_llm_retry 2 ✓, test_local_directory_sources 3 ✓
- **Completion criteria:** all 7 verified green — status phase fields + terminal semantics; `Writing KB overview…`/`Summarizing folders… (n/m)` labels (title + aria-live); pending set == `missing_folder_summaries` (integration cross-check pinned at `test_docs_api.py:428`); CLI/`ls` byte-identity (no changes to those paths, pins green); suite/coverage/lint gates; dedicated + regression E2E. Commit left to the harness per protocol (no `git add`/`commit` run).
- **Decisions/deviations:** none — no fixes were required this pass.
- **Next pending phase:** `99_kb_tree_table_and_back_nav`.
2026-09-13 00:23:05 -04:00
ducoterra 909c96c7bc minor fix to prevent null bytes from breaking sync
Build and Push Containers / build-and-push-app (push) Successful in 1m54s
Build and Push Containers / build-and-push-db (push) Successful in 19s
2026-09-12 22:12:09 -04:00
ducoterra d731169b8b phase: 105_hidden_folders_toggle
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All completion criteria verified. Everything is green.

**Phase 105 final verification pass — all criteria verified**

- Verified the full implementation in the working tree: `git_sources.include_hidden` column + alembic `0019` (dev DB at head, column present), `iter_importable_files`/`import_sources` flag support with `str(root)`-keyed map used by both walk and progress pre-walk, `GitSourcePatchIn` rename with optional fields, sync/CLI pipeline wiring (OR-collision), and the per-row "Hidden" checkbox + tag + error line on the Sources page
- Unit + integration: `uv run pytest` → exit 0 (2148 tests collected, all pass; this sandbox occasionally swallows pytest's final status line — exit codes verified)
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (3879 stmts, 15 miss) — >90% gate ✓
- Dedicated E2E: `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` → **6 passed in 23.20s** (DB up, in isolation)
- Regression E2E in isolation: `test_source_ignore_paths` 6 passed, `test_git_sources_admin` 6 passed, `test_local_directory_sources` 3 passed, `test_sync_button` 3 passed, `test_smoke` 3 passed
- Lint/types: `uv run ruff check .` + `uv run pyright` → clean (0 errors/warnings)

**Completion criteria:** (1) checkbox persists via PATCH 200 → "hidden on" tag + GET round-trips `include_hidden: true`; failure path reverts box + `role="alert"` canned message ✓; (2) flag OFF byte-identical (only `visible.md` indexed), ON indexes `.hidden/note.md` into the KB catalog, `EXCLUDED_DIRS` excluded both states ✓; (3) A2: flag OFF → `detail.pruned==1`, doc gone from catalog ✓; (4) PATCH bool-only/list-only/both/neither no-op, phase-89 fixed 422s unchanged, 404, anonymous 403 (incl. bool-only body) ✓; (5) env-fallback rows render no checkbox, WCAG-clean (aria-label, keyboard focus, visible label, text tag) ✓; (6) full gate green ✓; (7) commit left to the harness per instructions (no `git add`/`commit` run; phase files untouched).

**Deviations:** none — no defects found; no code changes were needed on this pass.

**Next pending phase:** `.agents/phases/todo/98_sync_summary_visibility`.
2026-09-12 21:26:30 -04:00
ducoterra ecc921098a phase: 104_chip_sizing_question_cap
Build and Push Containers / build-and-push-app (push) Successful in 1m48s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All completion criteria verified. Final report:

**Phase 104 — final verification pass: all green**
- Verified (no code changes needed): single-line ellipsized `.suggestion-chip` CSS + deleted `.maybe-try` override, `renderChips` full-text `title` + `aria-label`-when-clipped, `maxlength="4000"` + `#char-count` counter + `handleSend` over-cap guard, unit pins file, schemas boundary pins, dedicated E2E suite.
- E2E (isolation): `uv run pytest tests/e2e/test_chip_sizing_question_cap.py -v --no-cov` → **6 passed**; regressions: `test_suggestion_chips.py` 8 passed, `test_pinned_composer.py` 4 passed, `test_responsive_polish.py` 7 passed, `test_chat_history.py` 5 passed.
- `uv run pytest` → **2102 passed**; `--cov=app` → **99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- Criteria: chip E2E (single-line, clipped, title+aria-label full text) ✅; paste caps at exactly 4,000, send streams, counter hides ✅; programmatic 5,000-char fill → banner, no turn, text kept ✅; 4,000/4,001 boundary pinned + HTML maxlength == JS constant cross-file pin ✅.
- Diff scope: `frontend/`, new unit file, `tests/unit/test_schemas.py`, new E2E file, phase files — **no `app/` diff, no migration, no `shared.js` diff**.
- Deviations: 4 regression test files touched — 2 genuine DOM-pin conflicts from the new `#char-count` child (explicitly anticipated by the overview) + 3 documented **pre-existing E2E flake fixes** (smooth-scroll race, tab-walk heuristic, 10 ms timeout), each verified pre-existing on the pre-phase-104 tree.
- No commit made (harness commits per the execution protocol override).
- Next pending phase: `98_sync_summary_visibility`.
2026-09-12 19:45:00 -04:00
ducoterra 1f1c01c9f7 phase: 103_suggestions_session_openers
Build and Push Containers / build-and-push-app (push) Successful in 2m32s
Build and Push Containers / build-and-push-db (push) Successful in 12s
Phase 103 final verification pass — all green.

**Verified (all 3 tasks already in `complete/`; no code changes needed):**
- `opening_questions` in `app/api/suggestions.py` — forward walk, one opener per chat (first non-blank user msg, A3), reads raw `messages` not `title` (A4), phase-80 order/dedup/cap/seed contracts; `last_questions` name gone from `app/`+`tests/`
- Docs updated: `app/config.py` seed docstring, `.env.example` `BOR_SUGGESTIONS`, `README.md` — "session openers" wording
- Diff scope correct: only the 6 expected files + phase-file moves; `app/rag/suggestions.py` and `frontend/` untouched

**Test / lint / coverage results:**
- `uv run pytest tests/integration/test_suggestions_api.py -v` → 12 passed
- `uv run pytest tests/e2e/test_suggestion_chips.py -v --no-cov` → 8 passed in isolation (opener-only core pin included)
- `test_responsive_polish.py` → 7 passed; `test_chat_persistence.py` → 4 passed (both isolated, no edits)
- `uv run pytest --cov=app --cov-report=term-missing` → 2086 passed, TOTAL 99% (>90%); `app/api/suggestions.py` 100%
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings

**Completion criteria:** all 7 ✅ (follow-ups-never-surface pin; cap-across-chats pin; seed/dedup/case/partial/A3/401 pins; E2E suites isolated; deflection chips unchanged; full suite + lint; commit + dir move left to harness per executor rules).

**Deviations:** none — no defects found; nothing changed in this pass.
**Next pending phase:** `98_sync_summary_visibility` (numeric order in `todo/`).
2026-09-12 16:37:50 -04:00
ducoterra 3b2dea5685 phase: 102_extensionless_filenames
Build and Push Containers / build-and-push-app (push) Successful in 1m38s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification complete — every gate green, no defects found in previously completed work.

**Phase 102 final verification pass — report**

Verified (all three task files present in `complete/`; working-tree implementation matches D1–D5 design):
- `match_extension` choke point in `app/rag/importer.py` (walk + `formats` counter), `doc_format` name-token badge in `app/api/docs.py`, config/`.env.example` docs, fixture `tests/fixtures/extensionless_kb/`, integration + E2E suites — all present and correct
- Completion criteria: end-to-end sync (✓ integration + E2E), case matrix incl. `mydockerfile`/`Dockerfile.dev`/`.dockerfile` exclusions (✓ unit), `formats=dockerfile:1` not `unknown` (✓ log-line assertion), badge `dockerfile`/`containerfile` + `text` fallback + suffixed unchanged (✓ unit/integration/E2E), prune-on-token-removal (✓ `pruned==2`), suffixed-path rule byte-identical (✓ single-line swap, existing cases untouched)

Test / lint results (exact commands):
- `uv run pytest --cov=app --cov-report=term-missing` → 2084 passed, **99%** coverage (>90% gate)
- `uv run pytest tests/e2e/test_extensionless_import.py -v --no-cov` → 2 passed, isolated, DB up
- Regressions isolated: `test_import_documents` 3✓, `test_import_extensions_env` 2✓, `test_quadlet_jinja_import` 4✓, `test_document_viewer` 7✓, `test_kb_tree` 8✓
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings

Notable: commit intentionally not made (harness commits the phase); 102's task files already sit in `complete/`, overview stays in `todo/` for the harness.
Next pending phases: 98, 99, 103, 104, 105 (numeric next after 102: `103_suggestions_session_openers`).
2026-09-12 15:56:43 -04:00
ducoterra 4dbac1660a phase: 101_tokens_page_overhaul
Build and Push Containers / build-and-push-app (push) Successful in 1m49s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All gates verified green — no defects found, nothing to fix. Final report:

## Phase 101 — final verification pass (all tasks already complete; verified, not re-implemented)

**Verified against the working tree** (task files 01–04 present in `complete/`):
- Active/revoked table split (Status column gone, section hidden while 0 revoked, per-table search with query state surviving re-renders) in `frontend/index.html` + `tokens.js`
- `generateToken()` refuses blank/whitespace names (`Give the token a name first.`, no request); `|| "token"` fallback deleted
- Atomic rotation: `regenerate_token` service + `POST /api/tokens/{id}/regenerate` (201 new-plaintext-once / 404 / 409 / router-wide 403) + `TokenListItem.revoked_at` (D5)
- Regenerate two-step confirm UI + CSS (`.token-regenerate`, neutral hover, no new hue); A4 pins intact

**Test / lint / coverage results:**
- `uv run pytest` → **2065 passed**
- `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%** (>90% ✓)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- `uv run pytest tests/e2e/test_tokens_page.py -v --no-cov` → **4 passed** (isolation, DB up)
- Regression, each in isolation: `test_api_tokens.py` **9 passed**, `test_admin_auth.py` **6 passed**, `test_shared_header.py` **6 passed**, `test_theme_semantic_completion.py` **8 passed** (its revoked-pill pin was correctly re-scoped to the revoked table in this phase)

**Completion criteria:** 1 ✓ split+search (E2E 1–2) · 2 ✓ required name (E2E 3 + source pin) · 3 ✓ rotation end-to-end, old token refused at gate (E2E 4 + API 404/409 pinned) · 4 ✓ A4 holds (list carries no plaintext/hashes) · 5 ✓ suite/coverage/lint green · 6 ✓ E2E + regressions green in isolation · 7 commit left to the harness per executor rules (all changes uncommitted in the working tree)

**Deviations:** none. Next pending phase: `98_sync_summary_visibility`.
2026-09-12 15:16:02 -04:00
ducoterra 58e9d94cff phase: 100_page_width_consistency
Build and Push Containers / build-and-push-app (push) Successful in 4m3s
Build and Push Containers / build-and-push-db (push) Successful in 18s
**Phase 100 — final verification pass: all green.**

- Verified the shipped CSS contract directly: `--chat-column: 72rem` in `:root`; 0 literal `max-width: 46rem`; no `@media (min-width: 1500px)` block; exactly 4 token-capped reading columns; tuning/theme/doc-edit shells cap-free, structurally `.sources-shell`; `mock_llm.py` diff is comment-only.
- Defect found & fixed (phase-93 suite): `test_theme_semantic_completion.py::test_reset_removes_tag_byte_identical` raced theme.js's post-PUT refetch — it asserted the `#bor-theme` tag was gone right after the result line, but tag removal lands in the reconcile after the re-fetch (failed ~1 in 5 runs after `test_chat_rag`, reproduced). Fixed with an auto-waiting `expect(...).to_have_count(0)` (settled-state wait, the file's existing house pattern). 6/6 clean on the repro loop after.
- Tests/lint/coverage: `uv run pytest --cov=app --cov-report=term-missing` → 2052 passed, **99%** on `app/`; `uv run ruff check .` + `uv run pyright` → 0 errors.
- E2E in isolation (all passed): `test_wide_desktop_column.py` 3 (the phase suite — chat==tuning==theme==RAG ±4px at 1280 & 1920, ≈1152px; shared ≈1152px; standalone doc ≈1112px; modal unchanged ≈1100px; 360px overflow-free), `test_ui_customization` 4, `test_admin_theme_tab` 5, `test_document_viewer` 7, `test_save_share_ux` 5, `test_sticky_navbar` 3, `test_markdown_tables` 6, `test_responsive_polish` 7, `test_chat_rag` 3, `test_theme_semantic_completion` 8.
- Completion criteria: ① measured 72rem everywhere (≥~1200px, ±4px) + full-width below + 360px clean — **PASS** (E2E); ② zero 46rem rules / no 1500px block / four token selectors — **PASS** (grep + unit pins); ③ B4 byte-identical no-op + mobile squeeze — **PASS** (theme + responsive suites green); ④ full suite / coverage / lint — **PASS**; ⑤ atomic commit — left to the harness per executor protocol (all changes in the working tree, uncommitted).
- Deviations: none from phase decisions; only change in this pass is the race fix above (test-only, behavior unchanged).
- Next pending phase: `98_sync_summary_visibility` (numeric order in `todo/`; also pending: 99, 101–105).
2026-09-12 13:37:19 -04:00
ducoterra a2ca2f905f chore(agent): phase 105 (hidden-folders toggle) from TODO.md, clear the list 2026-09-12 11:55:03 -04:00
ducoterra ffa6bda3e5 add phases for fixing suggestion chips
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-12 10:52:51 -04:00
ducoterra 17dd3bfac1 docs(plan): remove hardcoded phase/migration references to prevent staleness
Build and Push Containers / build-and-push-app (push) Successful in 14s
Build and Push Containers / build-and-push-db (push) Successful in 16s
Refactor §12 roadmap to point to ls commands and 00_phase.md files
instead of listing specific phase numbers and descriptions.

Remove ~20 phase-number references from anchors, revision notes,
tool surface, feedback, and sync sections — keep only the one
stable reference (phase 03 convention).

Replace migration ranges with 'ls alembic/versions/'.
Replace hardcoded retry values with 'configurable (defaults: ...)'.
Status header now points to §12 instead of listing counts.
2026-09-12 10:22:45 -04:00
ducoterra 800bea769a docs(phases): author todo queue phases 98-102
98 sync summary visibility (status phases + pending markers), 99 catalog
one-line clamp + breadcrumb back-nav, 100 72rem page-width consistency,
101 tokens page overhaul, 102 extensionless filename import.
2026-09-12 10:15:22 -04:00
ducoterra ad7585d474 phase: 97_kb_tree_catalog
Build and Push Containers / build-and-push-app (push) Successful in 2m11s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All completion criteria verified — everything is green, no defects found. Final report:

## Phase 97 final verification pass — ALL GREEN

**Verified (no code changes needed):**
- `GET /api/docs/tree` (admin), `build_kb_tree` pure builder, `PATCH /api/folders/summary`, migration 0018 (`manually_edited`, head confirmed), generator skip/keep + `kept_manual` stat, RAG tree UI + edit affordance in `sources.js`/`index.html`/`styles.css`
- `tests/e2e/test_kb_tree.py`: 8 passed — top level, drill source/folder, edit round-trip, clear, manual-desc-survives-sync, reload fallback, anonymous gate
- Integration: tree shape/order/403/empty/indexed-only + PATCH update/create/root/clear/404/403/no-LLM + stat-walk equivalence (in `test_docs_api.py`); 3-field `folder_summaries=` import token preserved

**Gates (exact commands):**
- `uv run pytest --cov=app --cov-report=term-missing` → **2053 passed**, TOTAL coverage **99%** (>90% ✓)
- `uv run ruff check . && uv run pyright` → **All checks passed / 0 errors**
- `uv run pytest tests/e2e/test_kb_tree.py -v --no-cov` → **8 passed** in isolation
- 30 story/RAG-view E2E suites run **one per process**: all passed, incl. `test_ls_tree_drilldown` (agent `ls` byte-identical ✓), `test_import_documents`, `test_edit_summaries`, `test_admin_auth`, `test_kb_overview`

**Completion criteria:** tree view ✓ · edit round-trip + clear ✓ · manual persists/clear resets ✓ · `ls` unchanged ✓ · pytest/coverage/lint ✓ · E2E isolation ✓ · commit — left to harness per protocol (working tree untouched, `git add/commit` not run)

**Deviations:** none. **Next pending phase:** none — `todo/` contains only 97 (96 already committed).
2026-09-11 22:48:02 -04:00
ducoterra a49be80b8e phase: 96_oneshot_resilience
Build and Push Containers / build-and-push-app (push) Successful in 1m34s
Build and Push Containers / build-and-push-db (push) Successful in 10s
All checks complete. Final report:

**Phase 96 (one-shot resilience) — final verification pass, all green** (all 4 task files already in `complete/`; verified the working-tree implementation against the design)

- `LLMClient.chat()` empty-content retry (D1–D3) via `_chat_once` + `_EmptyContentError` (carries `finish_reason`), under `BOR_LLM_RETRIES`/`BOR_LLM_RETRY_DELAY` — verified in diff
- `missing_folder_summaries()` + `generate_folder_summaries(only_missing=…)` — verified; `folder_summary_table_empty` deleted, both sync gates switched to the gap probe
- `.env.example` comments updated (chat-turn stream + one-shot summary calls)

**Test / lint / coverage results**
- `uv run pytest --cov=app --cov-report=term-missing` → **1988 passed**, coverage **99%** (gate >90%)
- `uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov` → **2 passed** (isolated)
- Regressions, each isolated: `test_ls_tree_drilldown` 3 passed, `test_sync_button` 3 passed, `test_local_directory_sources` 3 passed, `test_llm_retry` 4 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:** retry-then-recover unit-pinned ✓ · exhaustion + `BOR_LLM_RETRIES=0` byte-identical ✓ · streaming path untouched ✓ · gap-fill both sync paths, other rows byte-identical incl. `updated_at` ✓ · no-gap zero-burn ✓ · phase E2E green ✓ · regression E2Es green ✓ · full suite + >90% + lint/types ✓ · no completed-phase behavior change (full suite green) ✓. Commit left to the harness per executor rules (working tree, 16 files).

**Deviations:** none. **Next pending phase:** `97_kb_tree_catalog`.
2026-09-11 13:16:20 -04:00
ducoterra bcaef800c5 phase: 95_read_truncation_cap
Build and Push Containers / build-and-push-app (push) Successful in 1m38s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All gates are green. Final verification is complete — no fixes were needed; the phase is fully implemented and passing.

**Phase 95 final verification report**

- Verified all 3 tasks already implemented (task files in `complete/`): read cap + notice + `ToolResultPiece` (task 01), `tool_result` SSE + live/saved/shared marker (task 02), E2E suite (task 03).
- Code checks passed: `BOR_READ_MAX_CHARS` (default 128 000, validator), pinned `READ_TRUNCATION_NOTICE` + `TRUNCATION_MARKER`, byte-identical at/under-cap read, holder `read_truncations` (no `tool_calls` impact), piece order (after `tool`, before next round), `ChatToolResultEvent`, `ToolCall` back-compat fields, `app.js` live/restore handlers, `shared.js`, `.truncated-note` CSS, `.env.example` entry, mock-LLM echo script.
- **Tests:** `uv run pytest` → 1966 passed; `uv run pytest --cov=app --cov-report=term-missing` → all green, TOTAL **99%** (>90% gate).
- **E2E:** `uv run pytest tests/e2e/test_read_truncation_cap.py -v --no-cov` → **3 passed** (frame order + live marker + LLM notice via echo; save→shared fidelity; short-read control).
- **Regression (isolated):** `test_agent_document_tools` 4 ✓, `test_chat_history` 5 ✓, `test_share_chat` 4 ✓, `test_big_read_progress` 4 ✓, `test_stop_generation` 3 ✓.
- **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings.

**Completion criteria:** ① over-cap read → first-cap-chars + marker + pinned notice — ✓ (unit-pinned: at-cap/cap+1/notice tests); ② user marker live/saved/shared — ✓ (E2E + frontend tests); ③ at/under cap byte-identical, no frame — ✓ (unit + control E2E); ④ top-2 `<documents>` retrieval untouched — ✓ (`app/rag/retriever.py` unmodified vs HEAD); ⑤ suite green, >90% coverage, ruff+pyright clean — ✓; ⑥ no completed-phase behavior change — ✓ (all gates green; commit left to harness per pass rules).

- No defects found; no changes made this pass. Next pending phase: none in `todo/` (96 is the next free number).
2026-09-11 03:42:51 -04:00
ducoterra d4943b4822 phase: 94_ls_tree_drilldown
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 25s
All green. Verification complete.

**Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)**

- Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal
- Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths
- Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met
- `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched)
- Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed
- Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol)

**Next pending phase:** `95_read_truncation_cap`
2026-09-11 00:59:35 -04:00
ducoterra 9188be259b phase: 93_theme_semantic_completion
Build and Push Containers / build-and-push-app (push) Successful in 1m56s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All verification is complete. Final report:

**Phase 93 — Theme semantic completion: FINAL VERIFICATION PASS — ALL GREEN**

- Verified full implementation in tree: migration `0016` (8 nullable semantic columns, applied at head), 17-var `BUILTIN_COLORS`/`COLOR_FIELDS`/`effective_settings`, API validation, `#view-theme` State-colors fieldset (17 pickers), `theme.js` FIELDS/PAIRS (5→8), `.page-head` surface panel (6 shell views + doc-edit + shared.html; login card / document sticky header audited as already-surfaced), mock_llm `content: None` fix
- Fixed 2 pre-existing defects (both fail identically on baseline `d4f38ad`, proven via worktree A/B): `test_nav_rename_sources` — expected nav tail missing the phase-91 "Theme" link; `test_stale_ui_copy` — now truncates `saved_chats` before/after (house `test_suggestion_chips` pattern) so the seed-chip contract is deterministic on the shared dev DB (owner's 22 saved chats triggered phase-80 last-3-questions)
- Tests: `uv run pytest --cov=app --cov-report=term-missing` → **1868 passed, app/ 99%** (>90% ✓); `uv run ruff check .` → clean; `uv run pyright` → **0 errors**
- E2E: dedicated `uv run pytest tests/e2e/test_theme_semantic_completion.py -v --no-cov` → **8/8 in isolation** (all-gray 17-color theme: zero residual color on saved-result/Stale/Revoked/Local/tool-call elements, text labels intact, gray heads non-transparent, pre-paint tag, Reset → byte-identical no-tag); 15 theme/header/nav/responsive suites green in isolation; full 85-file combined run: only the 2 fixed pre-existing failures + 1 combined-run artifact (`test_sync_upload_progress`, green in isolation)
- Completion criteria: (1) monochrome E2E ✓ (2) default byte-identical, no `#bor-theme` tag ✓ (3) all page heads on solid surface ✓ (4) suite/coverage/lint/E2E green ✓ (5) phases 01–92 no behavior change ✓ (6) commit left to harness per protocol
- Notable: cleaned stray uvicorn leftovers from prior implementation pass (owner's `--reload` dev server untouched); no deviations from the phase design
- Next pending phase: `94_ls_tree_drilldown`
2026-09-10 16:43:08 -04:00
ducoterra d4f38ad3ce add PLAN.md
Build and Push Containers / build-and-push-app (push) Successful in 39s
Build and Push Containers / build-and-push-db (push) Successful in 25s
2026-09-10 13:08:55 -04:00
ducoterra bf308eb795 chore(agent): phase 93-95 roadmap from TODO.md — theme completion, ls tree drill-down, read truncation cap
Convert the three unchecked TODO.md items into an executable phase
roadmap (Protocol B, appended after phase 92):

- 93_theme_semantic_completion (TODO L3): the ok/err/accent state
  families become Theme-tab-controlled (B3 revised, owner permission
  2026-09-10) + surface panels behind every page head
- 94_ls_tree_drilldown (TODO L4): ls becomes a source -> folder ->
  file tree with sync-time lite-model folder summaries; controlled
  tool-calling battery as the accuracy/performance gate
- 95_read_truncation_cap (TODO L5): read capped at BOR_READ_MAX_CHARS
  (128k chars ~= 32k tokens, spec'd on the 128k-token minimum context),
  LLM-visible truncation notice pointing at grep, new tool_result SSE
  event (A15 extension) + the visible UI marker

Owner decisions (B3 / A7 scope / A15) are recorded in the phase files;
.agents/PLAN.md is being redone separately per the owner.
2026-09-10 11:37:25 -04:00
ducoterra dac4a3eec0 docs(agent): restore master plan at .agents/PLAN.md with locked decisions through phase 92
Re-lands the master plan (removed from VCS in 281f355 as .agent/PLAN.md)
at .agents/PLAN.md — the path AGENTS.md and the codebase's PLAN §…
references use. Rebuilt from the original text plus every
owner-permitted revision recorded in the completed phases: A1–A17
anchors with the full revision log (auth, SSE thinking/tool/retry
events, agent tools, history, SPA shell), new A18–A20 (docs push,
deploy caching, security headers), current data model (11 tables),
locked persona + <tools> copy, current UI/UX contracts (46rem/92rem
column, theme tab, never-stale feedback, no autoscroll), the full
per-turn log line, and the phase 01–92 roadmap with the open TODO.md
items as next up.
2026-09-10 09:28:58 -04:00
ducoterra df91c6316c phase: 92_theme_save_and_coverage
Build and Push Containers / build-and-push-app (push) Successful in 1m47s
Build and Push Containers / build-and-push-db (push) Successful in 11s
**Phase 92 final verification pass — all green.** This pass re-verified the completed tasks (all 5 task files already in `complete/`) against every completion criterion; no defects found, nothing to fix.

- Verified: 9th identity var `grid_line` end-to-end (migration `0015` at head, model/`theming.py`/schemas/API, 422 + built-in→NULL tests present); `styles.css` zero hardcoded literals outside `:root` + derived `--brand-*` vars; 9th picker in theme form; wordmark themed; `theme.js` save/reset/re-show/mount live-sync; dedicated E2E suite + phase-91 suite updated.
- `uv run pytest --cov=app --cov-report=term-missing` → **1845 passed, exit 0, TOTAL 99%** (>90%)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- `uv run pytest tests/e2e/test_theme_save_and_coverage.py -v --no-cov` → **3 passed** (save-live, reset-live, whole-site)
- `uv run pytest tests/e2e/test_admin_theme_tab.py -v --no-cov` → **5 passed**
- Criteria: (1) Save/Reset repaint open page, no nav, SPA-nav survives, pre-paint intact ✅; (2) both `rg` gates green (only `:root` + documented `#fff` Stop label; zero SVG hex attrs), grid/selection/hovers/wash/wordmark E2E-proven ✅; (3) no-op contract live-checked: row-less `/` = no tag + exact A1 CSP, grid-only row = 9-var tag in `COLOR_FIELDS` order + sha256 CSP, with-row ≡ row-less bytes ✅; (4) full suite/coverage/lint/both E2E ✅; (5) commit left to the harness per instructions.
- Deviations (previously made, probe-verified, kept): live repaint uses CSSOM `<html>` overrides because Chromium blocks `<style>` textContent mutations under the locked sha256-only CSP (tag text still mirrors the next load; `<html>` style exact-saved after Save, empty after Reset); wordmark themed via 3 `.brand-mark` CSS rules instead of inline styles (task 03's inline attrs were CSP-blocked — fixed during task 04).
- Next pending phase: none — `todo/` contains only `92_theme_save_and_coverage`.
2026-09-10 00:23:08 -04:00
ducoterra d22d260b8b phase: 91_admin_theme_tab
Build and Push Containers / build-and-push-app (push) Successful in 5m43s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification is complete — this pass needed no code changes. Final report:

**Phase 91 — Admin Theme tab: final verification pass (all 6 tasks already in `complete/`)**

- Verified pre-paint theming end-to-end: `ui_settings` store + resolver, admin `GET/PUT /api/ui-settings`, `CachingMiddleware` inline-`<style id="bor-theme">` injection before `</head>` (incl. `/shared/<token>` prefix branch, unit-pinned), CSP sha256 exemption for the inline tag, Theme tab shell + `theme.js` editor, CSS-file theming fully retired.
- No defects found; zero changes made — working tree left exactly as the task executors left it.
- Tests: `uv run pytest --cov=app` → 1841 passed, 0 failed (TOTAL coverage **99%**; theming/ui_settings/caching all 100%); `uv run pytest tests/e2e/test_admin_theme_tab.py -v --no-cov` → **5 passed** in isolation.
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings.
- Criteria: (1) unset deployment byte-identical, no `#bor-theme` anywhere — ✓ (unit no-op test + E2E reset byte-compare); `rg "BOR_THEME|themes/"` → single hit is the permitted doc-history comment in `frontend/index.html`. (2) admin-only gate + 403s for anonymous and token users — ✓ (E2E test 3). (3) saved theme inline before `</head>` on every page incl. `/shared/<token>`, computed `--brand` on first paint for admin + anonymous — ✓ (E2E test 2 + unit). (4) reset → byte-identical; 5 contrast pairs warn <4.5:1, non-blocking — ✓ (E2E tests 4–5). (5) suite green, >90% coverage, lint clean — ✓. (6) commit deferred to harness per rules.
- Notable: `.agents/PLAN.md` is absent from the repo — the phase overview's Design section was used as the binding spec; no deviation resulted.
- Next pending phase: **none** — 91 is the last phase in `todo/`.
2026-09-09 17:22:24 -04:00
ducoterra 3095c4c577 phase: 90_upload_no_scan
All verification is complete and green. Final report:

**Phase 90 (upload_no_scan) — final verification pass: all criteria verified, no fixes needed**

- Verified (not re-implemented): all 3 tasks already complete; working tree carries the phase's uncommitted changes
- Upload pipeline: `_run_upload` stops after row upsert; success = `{"message": "uploaded"}`, null/0/0 progress; `UploadOut` removed from `app/schemas.py`; gates/unpack/swap/failed states intact
- Copy: button "Upload", bare "Processing…", result line "Uploaded \<name\> — press Sync sources to import it."; hint + caption re-pointed at Sync; no "Upload &" remnants in `frontend/`
- Tests: `pytest tests/unit/test_git_sources.py tests/integration/test_git_sources_upload.py tests/unit/test_frontend_sync_upload.py` → 56 passed; E2E phase suite `tests/e2e/test_upload_no_scan.py` → 3 passed (zero docs after upload; ignore edit honored by Sync — 2 added, `notes/skipme.md` excluded; re-upload in-place, still zero docs); affected suites `test_archive_upload_sources.py` + `test_sync_upload_progress.py` + `test_source_removal_cleanup.py` → 15 passed
- Full suite: `uv run pytest --cov=app --cov-report=term-missing` → **1808 passed, 99% coverage** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors

**Completion criteria:**
1. Upload settles "ready for sync", zero docs, row + ignore editor — ✅ (E2E test 1)
2. Button "Upload", no "Upload &" copy — ✅ (`rg` empty in frontend/, app/, tests/)
3. RAG-page "Sync sources" imports upload honoring edited ignores — ✅ (isolated E2E test 2)
4. Suite green, >90% coverage, lint clean — ✅ (99%, ruff+pyright clean)
5. Atomic commit — ✅ left to harness per executor rules (no git add/commit run; tree staged-ready)

**Deviations:** `regenerate_overview` still appears once in `git_sources.py` — inside `delete_git_source` (phase-69 locked total-removal behavior, not the upload leg; upload leg is scan-free). `.agents/PLAN.md` does not exist in this repo; locked decisions A1–A4 were enforced from the phase files.

**Next pending phase:** `91_admin_theme_tab` (in `todo/`).
2026-09-09 14:08:18 -04:00
ducoterra 0c8a7b9974 chore(agent): phase 90-91 roadmap from TODO.md — upload without scan + admin theme tab 2026-09-09 08:50:14 -04:00
ducoterra 8c706259e9 phase: 89_source_ignore_paths
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification complete — TODO.md was already cleared in the roadmap commit; the two extra unit-test diffs are necessary fake-signature adaptations for the new keywords. Everything is green, no fixes were needed.

## Phase 89 — final verification pass: ALL GREEN

**Verified (all 6 task files present in `complete/`):**
- `git_sources.ignore_paths` JSONB column + migration 0013; `alembic downgrade -1 && upgrade head` round-trips (head `0013`)
- Importer: `normalize_ignore_path`/`is_ignored`/`_ignore_for_root`, `ignore` in walk + progress pre-walk, `ignore_by_root` in `import_sources`
- API: GET/POST carry list; admin-only `PATCH` (replace, 404/422 fixed details, anonymous 403)
- Pipelines wired: `_run_sync`, `_run_upload` re-upload, `scripts/import_docs.py`
- Sources-page box: dialog, §7.4 save lifecycle, `N ignored` tag, a11y; env rows get no box

**Test/lint results:**
- `uv run pytest` → 1808 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
- `uv run pytest tests/e2e/test_source_ignore_paths.py -v --no-cov` → 6 passed (isolated, DB up)
- Regressions in isolation: `test_git_sources_admin` 6, `test_archive_upload_sources` 5, `test_sync_button` 3, `test_smoke` 3 — all passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:** box→PATCH 200→count+GET round-trip ✅ · sync excludes `ignore/` (no docs/chunks/embeddings/summaries) + prunes newly-ignored (pruned==2) ✅ · no-mid-path rule E2E ✅ · PATCH 404/422/replace/clear/403 ✅ · full gate green ✅ · commit + phase move left to harness per rules.

**Deviations:** none blocking — E2E pins `files == 4` (overview's "5" was an off-by-one vs its own 6-file tree, documented in-test); `tests/unit/test_importer.py` + `test_sync_button.py` test-double fakes extended for the new keywords (needed for the suite to stay green).

**Next pending phase:** none — `todo/` holds only this phase.
2026-09-09 01:45:42 -04:00
ducoterra 0495e4e7e4 chore(agent): phase 89 roadmap from TODO.md — per-source ignore paths for imports 2026-09-08 22:55:58 -04:00
ducoterra 1f0e4c6bb9 fix(ui): bind the shared header controls on explicit init, not at module import
Build and Push Containers / build-and-push-app (push) Successful in 1m53s
Build and Push Containers / build-and-push-db (push) Successful in 11s
header.js's control bindings (sign-out, the mobile hamburger, the
SINGLE New chat button) ran at module import. The Containerfile stage-1
build inlines header.js into every bundle that imports it (the shell's
app.js, token-gate.js and the router's lazy views), so the shell page
registered the #nav-toggle click handler twice, and two toggle handlers
cancel each other — one tap = open + close = the mobile menu dead in
the deployed image only. The dev tree's single ESM instance (and every
test that runs against it) never showed it; a lazy view load adding a
THIRD copy made the menu work again, which is why the failure looked
state-dependent (chat cold boot dead, /sources.html alive).

- header.js: the three bindings move into an exported
  bindSharedHeaderControls(), guarded by a marker on <body> (NOT module
  state — every bundle copy has its own function instance), so later
  bundle copies and repeated inits (the token gate's mid-page header
  re-boot) are no-ops; header.js is now side-effect-free at top level,
  which also lets esbuild tree-shake the dead copies out of the bundles
  that do not need them (the token-gate bundle no longer carries the
  binding code at all)
- app.js / login.js / shared.js / document.js: call
  bindSharedHeaderControls() once at module top — import-time parity,
  unconditional (no async boot path to miss); doc-edit.js ships no
  header controls and calls nothing
- unit: tests/unit/test_header_bindings_once_per_document.py pins the
  contract — the init export, the document-level idempotency marker,
  all three bindings inside the init, NO top-level addEventListener
  remaining, and exactly one module-top call in each header-carrying
  page script; stale import-time docstrings in the legacy header pins
  updated to the new contract

Verified: full unit + integration suite (1746 passed), the hamburger /
pinned-composer / smoke E2E stories green in isolation, ruff + pyright
clean. Containerfile-equivalent esbuild 0.25.5 rebuild probed in
Chromium: exactly ONE #nav-toggle click listener on chat cold boot,
/sources.html and login.html, and a touch tap opens the menu in all
three states (pre-fix production: two listeners on cold boot = dead,
three on sources = alive).
2026-09-08 22:31:45 -04:00
ducoterra 4d287155c0 phase: 88_mobile_chat_hamburger_boot
Build and Push Containers / build-and-push-app (push) Successful in 2m16s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All completion criteria verified green. Final state confirmed: phase stays in `todo/` per A4 (owner device gate), task files in `complete/`, no `app/` changes, no commits made (harness commits).

## Phase 88 — final verification pass report

**Verified (all 6 tasks already complete):** `touch-action: manipulation` on mobile `#nav-toggle`; `#view-chat.chat-booted` double-rAF sticky handover (CSS gate + app.js boot IIFE); `body.nav-menu-open` marker in `setNavMenu` + ≤640px `visibility: hidden` rule; router boot contract (`wasMounted && !opts.boot` + `boot: true`); new unit module (10 pins) + 4-test real-touch E2E story.

**Test / lint / coverage:**
- `uv run pytest --cov=app --cov-report=term-missing` → 1741 passed, **99%** on `app/` (>90% ✓)
- `tests/e2e/test_mobile_chat_hamburger_boot.py` 4/4, `test_mobile_hamburger_nav.py` 8/8, `test_pinned_composer.py` 4/4, `test_smoke.py` 3/3 — each in isolation, `--no-cov` ✓
- `uv run ruff check .` clean; `uv run pyright` 0 errors ✓

**Completion criteria:** (1) 360px cold-boot touch-action + sticky handover + tap-opens-menu-with-cluster-hidden (E2E 1–2) ✓; (2) boot fires no `bor:view-refresh`, re-show fires exactly one, `/sources.html` regression (E2E 3–4) ✓; (3) full gate green ✓; (4) diff limited to 4 assets + 2 new tests + phase files, 0 changes in `app/` ✓; (5) commit deferred to harness per executor rules ✓; (6) owner device re-verification **pending** (A4 — gates the archive; no Owner report recorded yet).

**Deviations (both documented in-tree):** unit pins updated in `test_frontend_router.py`/`test_hamburger_nav.py` (their exact-text pins collided with the mandated new guard/marker text — without them the suite goes red); `boot: true` count pinned at 1 not 2 (codebase has one boot call site, no `history.state` branch — verified against git HEAD).

**Next pending phase:** none in `todo/` — pipeline awaits the owner's on-device report (archive, or `?dbg=nav` instrumentation follow-up if the menu is still dead).
2026-09-08 16:02:45 -04:00
ducoterra 10fd367962 fix(chat): render typing-indicator elapsed hint as horizontal text
Build and Push Containers / build-and-push-app (push) Successful in 1m43s
Build and Push Containers / build-and-push-db (push) Successful in 15s
The phase-87 ".typing-elapsed" dot-geometry reset (specificity 0,1,0)
lost every shared declaration to the ".typing span" dot rule
(0,1,1): the hint rendered as an 8x8px bouncing dot and the "Ns" text
wrapped one character per line below the bubble (overflow-wrap:
anywhere on .bubble). Phase 87's e2e checked text values only, so the
squish shipped unseen.

- retarget the reset at ".typing span.typing-elapsed" (0,2,1) so it
  actually wins; center the dots while the hint line is taller
- unit: pin the reset's selector context (specificity regression guard)
- e2e: layout pin on the live hint — no dot animation, not an 8px box,
  horizontal single-line bounding box
- before/after verification screenshots in
  .agents/reports/87_big_read_progress/

Verified: unit 41 passed, phase-87 e2e 4 passed (isolated), ruff +
pyright clean.
2026-09-08 10:03:10 -04:00
ducoterra 5abe8871e3 fix(chat): raise pre-token guard from 120 s to 300 s
Build and Push Containers / build-and-push-app (push) Successful in 1m48s
Build and Push Containers / build-and-push-db (push) Successful in 15s
The client-side TURN_TIMEOUT_MS was the binding constraint: turns with
slow prompt processing (no first SSE frame within 120 s of visible time)
errored with the 'stuck' copy even though nginx (300 s) and
BOR_LLM_TIMEOUT (300 s) would have let them run. Raise the guard to
300 s so the upstream timeouts are reachable, and re-pin the tests:
the unit constant pins and the fake-clock E2E timeline (295 s hidden
+ 290 s after the re-arm = 585 s: past the original 300 s deadline,
short of the re-armed 595 s deadline).

Verified: tests/unit (full, 100% pass), tests/e2e/test_hidden_tab_stream.py
and tests/e2e/test_loading_feedback.py in isolation.
2026-09-08 09:37:33 -04:00
ducoterra 7cfe58fb21 phase: 87_big_read_progress
Build and Push Containers / build-and-push-app (push) Successful in 2m48s
Build and Push Containers / build-and-push-db (push) Successful in 19s
All criteria verified — no defects found, nothing to fix. Final report:

**Phase 87 — big read progress: final verification pass (all tasks already complete in `complete/`)**
- Verified implementation vs. overview: `TOOL_LINE_ELAPSED_AFTER_MS = 5_000`, single live `armToolLineClock` site, settle on thinking/retry/delta, `stopToolLineClock` in `setUiState`, visible `.typing-elapsed` (aria kept byte-identical), CSS AA pairing — all match design; `app/` byte-identical (0 changes)
- **Tests/lint (exact outcomes):**
  - `uv run pytest --cov=app --cov-report=term` → 1732 passed, coverage **99%** (>90% ✓)
  - `uv run pytest tests/e2e/test_big_read_progress.py -v --no-cov` → **4 passed** (ticking suffix, visible hint, settle, no-timer restore)
  - `test_thinking_display.py` → 5 passed · `test_agent_document_tools.py` → 4 passed · `test_smoke.py` → 3 passed (all isolated)
  - 3 pinned frontend suites + new unit pins → 62 passed · `uv run ruff check . && uv run pyright` → clean, 0 errors
- **Completion criteria:** E2E pins 1–4 ✓ · guard/state-machine byte-identical ✓ (diff is additive only) · diff scope limited to `app.js`, `styles.css`, 2 new test files, phase files; nothing in `app/` ✓
- **Notable:** no deviations; commit + `00_phase.md` move left to the harness per executor rules (task files already in `complete/`)
- **Next pending phase:** none — `todo/` contains only this phase (87 is the last)
2026-09-08 05:51:23 -04:00
ducoterra 0f6b9ff7e6 phase: 86_history_page_width
All criteria verified. Final report:

**Phase 86 final verification pass — all green, no defects found.**

- **Verified (previously implemented):** `position: relative` on shared `.table-wrap` (the one-rule A3 fix, +12 lines in `frontend/assets/styles.css`); unit pin `tests/unit/test_table_wrap_stacking.py` (3 tests); E2E story `tests/e2e/test_history_page_width.py` (5 tests: History 626→≤375, Tokens 618→≤375, SPA switch with window-sentinel, RAG regression, desktop 1280×800)
- **Tests/lint (this pass):** `uv run pytest` → 1720 passed · `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% floor) · E2E in isolation: `test_history_page_width.py` 5 passed, `test_chat_history.py` 5 passed, `test_api_tokens.py` 9 passed, `test_smoke.py` 3 passed · `uv run ruff check . && uv run pyright` → clean, 0 errors
- **Completion criteria:** ① 375px direct+SPA `scrollWidth ≤ innerWidth` on History/Tokens — PASS (E2E pins) ② in-card scroll preserved + full-width + a11y spans in DOM — PASS ③ Sources + desktop regression — PASS ④ full suite/coverage/E2E/lint gates — PASS ⑤ `git diff --stat` limited to `styles.css` (+12) + new test files + phase files, no `app/`/markup/JS — PASS ⑥ commit/move — left to harness per executor rules (working tree intact, `todo/` dir removal already reflected)
- **Notable:** red→green (CSS reverted → 626px failure) and manual live check already recorded in `.agents/reports/86_history_page_width/`; pre-existing untracked `.agents/remediation_plan.md` (Sep 7 security audit) untouched
- **Next pending phase:** `87_big_read_progress`
2026-09-08 02:38:45 -04:00
ducoterra 412a560348 phase: 85_mobile_menu_gate_overlap
All verification complete. Final report:

**Phase 85 final verification pass — all green** (tasks 01–03 already complete; independently re-verified from scratch)
- Verified fix in tree: `styles.css` `.auth-gate` z-index 500→15 + rewritten stacking comment (no `z-index: 500` left); `index.html`/`document.html` comment-only; `token-gate.js` docstring-only (logic byte-identical); `test_mobile_hamburger_nav.py` real-click conversion + new `test_anonymous_toggle_tappable_with_gate_up`; new `tests/unit/test_gate_header_stacking.py` (3 pins); `test_api_tokens.py` untouched
- `uv run pytest` → 1717 passed, 1 warning (exit 0)
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%)
- E2E in isolation: `test_mobile_hamburger_nav.py` **8 passed**; `test_api_tokens.py` **9 passed** (unchanged); `test_smoke.py` **3 passed**; `test_document_viewer.py` **7 passed**
- `uv run ruff check . && uv run pyright` → clean / 0 errors
- Live probe (375×812, anonymous, fresh server): on `/` and `/history.html` `elementFromPoint` at toggle → SVG `path`, never `#auth-gate`; real click opens menu (`aria-expanded=true`); exactly one visible nav link (Chat) + sign-in copy; Chat link topmost over gate; `#main` inert — criteria 1, 2, 3 confirmed directly
- Criteria: (1) real-click menu on / + /history ✅ (2) dropdown above gate ✅ (3) anonymous contents + #main inert ✅ (4) admin byte-identical (phase-46 tests green) ✅ (5) doc-viewer gate under bar (CSS pin + doc-viewer suite) ✅ (6) full gate green ✅ (7) diff scoped to the 5 code files + new unit test + phase files, nothing in `app/` ✅ (8) commit + todo→complete move left to harness per executor rules (task files already in `complete/`)
- Note: `.agents/remediation_plan.md` (untracked) is a pre-existing artifact of the earlier security audit — left untouched; a missing "N passed" line seen mid-pass was my own `-q`+addopts `-q` = `-qq` quirk, not a defect
- Next pending phase: `86_history_page_width`
2026-09-08 01:56:37 -04:00
ducoterra f4150421bb phase: 84_docs_push_error_sanitization
**Phase 84 — final verification pass: all green, no defects found**

- Verified implementation: `app/core/errors.py` (verbatim lift of sync masker), `app/api/sync.py` alias import, docs-push 502 `detail=sanitize_error(str(exc))`, all five `llm.py` error sites sanitized; new/extended test pins in place
- Tests: `uv run pytest` → **1714 passed, 0 failed**; targeted pins (new unit ×2 + integration ×1, existing 502 pin) → 13 passed; sync/git-sources regression → 67 passed
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` → **99%** (`app/core/errors.py` 100%, `app/rag/llm.py` 100%) — >90% met
- E2E isolation: `uv run pytest tests/e2e/test_smoke.py -v --no-cov` → **3 passed**
- Lint/types: `uv run ruff check .` → clean; `uv run pyright` → **0 errors**
- Criteria: 502 masks `*****@`/never token + row untouched ✅; LLM base-URL masked, credential-free strings byte-identical ✅; `_CREDS_RE` only in `app/core/errors.py` (working-tree grep) ✅; full gate green ✅; `git diff --stat` limited to the 4 app files + 2 modified test files + 3 phase task files (untracked: new module, new unit test, complete/ dir, reports, audit plan) ✅
- Commit/phase move left to the harness per instructions (task files already in `complete/`)
- No deviations; nothing to fix
- Next pending phase: **85_mobile_menu_gate_overlap**
2026-09-08 01:11:45 -04:00
ducoterra fa189dede7 phase: 83_chat_save_payload_limits
All verification criteria are confirmed green. Final report:

**Phase 83 — final verification pass (all tasks already complete; verified, nothing to fix)**

- Verified `app/schemas.py` carries every A1 cap (text/thinking 32 000, sources 20, suggestions 50×200, tools 50, SourceRef 120/1000/500, ToolCall 100/2000, messages ≤200) — boundary-only, `app/api/chats.py` untouched
- Verified all test pins exist and pass: 32 unit boundary tests (both edges of every cap + round-trip), 4 oversized-422 integration pins (text/201-msgs/21-sources/PUT, each with nothing-stored/row-unchanged assertions), A3 SSE done-event pin at column maxima, dedicated E2E (anon 40 000-char POST → 422 + small save → 201)
- `uv run pytest tests/unit/test_schemas.py -v` — 32 passed
- `uv run pytest tests/integration/test_chats_api.py` — exit 0; A3 pin — passed
- `uv run pytest tests/e2e/test_chat_save_payload_limits.py -v --no-cov` — 2 passed (isolation); `uv run pytest tests/e2e/test_chat_history.py -v --no-cov` — 5 passed
- `uv run pytest` — exit 0 (~1 704 tests, 0 fail/skip); `uv run pytest --cov=app` — TOTAL 99%, `app/schemas.py` 100% (>90% ✓)
- `uv run ruff check . && uv run pyright` — clean (0 errors)
- `git diff --stat` — only `app/schemas.py`, 3 test files (+`test_chat_api.py` A3 pin, sanctioned by task 02), phase files; no `app/api/chats.py`/`alembic`/`frontend`/`pyproject`/`uv.lock` diff ✓
- All completion criteria met; commit + phase move left to the harness per pipeline rules (changes stay in working tree)
- Deviation note: list caps use `max_length` instead of `max_items` — identical in pydantic 2.13 (`max_items` is a deprecated alias); both boundaries behaviorally pinned
- Next pending phase: `84_docs_push_error_sanitization`
2026-09-08 00:33:21 -04:00
ducoterra e29d68d9f0 phase: 82_security_headers
All completion criteria verified green — no defects found, nothing to fix. Final report:

**Phase 82 (security headers) — final verification pass: all green**

- Verified prior-run implementation: `app/core/security_headers.py` (pure-ASGI, header-only, exact A1 CSP), registration in `app/main.py` after `configure_caching` (outermost), unit/integration/E2E suites.
- Deviation confirmed sound: `data:`-URI favicon (blocked by locked CSP) → static `frontend/assets/favicon.svg` in 5 templates + Containerfile `cp`; SVG element byte-identical to the old data-URI (verified programmatically); serves 200 with all three headers.
- Curl check (server booted like e2e conftest, log: `/tmp/curl_security_headers_final.log`): `/`, `/api/health`, `/assets/styles.css`, `/nope` (404) → all three headers, CSP exactly `default-src 'self'; base-uri 'none'; frame-ancestors 'none'`.
- `uv run pytest tests/unit/test_security_headers.py tests/integration/test_security_headers.py -v --no-cov` → 13 passed (incl. SSE byte-identity pin).
- `uv run pytest tests/e2e/test_security_headers.py -v --no-cov` (isolated) → 2 passed (headers + zero CSP violations + painted page).
- SSE tripwire `uv run pytest tests/e2e/test_chat_rag.py -v --no-cov` → 3 passed.
- `uv run pytest --cov=app --cov-report=term-missing` → 1665 passed, app/ 99% (>90%); `uv run ruff check . && uv run pyright` → clean (0 errors).
- `git diff --stat` limited to phase-82 files + the two documented deviations (favicon set, `tests/unit/__init__.py`); no `pyproject.toml`/`uv.lock`/JS diffs.
- Commit + phase-dir move left to the harness per pipeline rules (not executed by me).

Next pending phase: `83_chat_save_payload_limits`.
2026-09-07 23:54:41 -04:00
ducoterra 42a4222949 phase: 81_login_rate_limit
All gates verified. The phase is complete — everything was already implemented by prior task runs; this pass verified each completion criterion end-to-end and found no defects.

**Phase 81 final verification report** (tasks 01–03 all present in `complete/`; no code changes needed this pass)

- Verified: `app/core/rate_limit.py` (stdlib sliding window, fail-open) + both login routes' 429 pre-check/record/reset wiring + all unit/integration pins present (11th-429, blocked-success-still-429, clean-counter reset, shared counter, autouse `clean_rate_limit` fixture documented).
- Live check (task 03): dev server + 11 rapid wrong logins → `401 ×10, 429`; 429 body carries `retry-after: 900` + generic detail; server restarted (per-process counter cleared by design).
- `uv run pytest tests/unit/test_rate_limit.py -v --no-cov` → 10 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 1652 passed, **TOTAL 99%** (>90%; rate_limit.py 100%, auth.py 100%)
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov` (isolation) → 3 passed
- `uv run ruff check . && uv run pyright` → All checks passed / 0 errors, 0 warnings
- Completion criteria: all met, except commit + phase-dir move — per harness rules I left all changes uncommitted in the working tree (harness commits atomically and moves the phase).
- Diff scope: exactly `app/core/rate_limit.py`, `app/api/auth.py`, `tests/unit/test_rate_limit.py`, `tests/integration/test_auth_api.py` + phase files; `pyproject.toml` / `uv.lock` / `frontend/` untouched.
- Deviations: none in code; commit/move deferred to harness as instructed.
- Next pending phase: `82_security_headers`.
2026-09-07 23:08:46 -04:00
ducoterra 894637108c add configurable llm timeout
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-07 22:01:53 -04:00
ducoterra e2bed52751 chore(agent): phase roadmap from TODO.md, 3 phases (85-87)
TODO.md L3-L5 converted to executable phases (protocol B - append):
- 85_mobile_menu_gate_overlap (L3): the phase-79 token gate (z 500, fixed
  full-viewport) sits above the sticky header (z 20), so an
  unauthenticated visitor's tap on the mobile hamburger hits the gate
  overlay and the menu is unreachable until login. Reproduced:
  elementFromPoint at the toggle resolves to #auth-gate on every shell
  view; real clicks are intercepted. Fix: gate at z 15 (below the
  header + its mobile dropdown), #main stays inert-locked; the
  phase-46 E2E's programmatic-click workaround becomes a real click +
  a new TODO-regression pin.
- 86_history_page_width (L4): at 375px the History page panned ~250px
  into a blank region (document scrollWidth 626) although the table
  scrolled fine inside its card. Root cause: the .visually-hidden
  Actions header span is position:absolute with no positioned ancestor,
  so its 1px box (at the 640px table's right edge) leaks into the
  document's scrollable overflow. Fix: position:relative on the shared
  .table-wrap card (the identical Tokens-view defect, measured 618, is
  fixed by the same rule; RAG is clean and pinned). New E2E story.
- 87_big_read_progress (L5): after a tool read the UI sat on a static
  'Reading <path>' line while the model prefilled the big context -
  the turn looked frozen. Frontend-only: a ticking '(Ns)' suffix on
  the latest tool line after 5s of frame silence (settle on the next
  frame, live-only - restored lines stay timer-free) + the existing
  10s aria-only typing clock promoted to a visible 'Ns' hint. New
  source-level unit pins + slow-proxy E2E story (deterministic >=6s
  gaps via the mock tool flow).

TODO.md cleared (items now live in .agents/phases/todo/). The pre-existing
uncommitted 81-84 phases + remediation_plan.md are a separate workstream
and are NOT part of this commit.
2026-09-07 21:12:03 -04:00
ducoterra 7baca3d289 update README
Build and Push Containers / build-and-push-app (push) Successful in 17s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-07 14:26:26 -04:00
ducoterra 2174caff33 fix button widths on mobile
Build and Push Containers / build-and-push-app (push) Successful in 1m39s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-07 14:18:45 -04:00
ducoterra f664aafeed fix(ui): give the mobile refresh controls labels and full width
At <=640px the RAG "Sync sources" pill and the History "Refresh"
pill squeezed down to tiny icon-only buttons — hard to discover and
tap on a phone. They are now full-width labeled pills:

- the RAG page-head row wraps so the Sync pill drops below the
  "Knowledge base" title at full width; the History page-head
  already wrapped the pill below its title block
- the Sync label's min(16rem, 40vw) cap lifts on mobile (min-width: 0
  engages the ellipsis) so the live-file text truncates against the
  full width instead of the 40vw cap
- the Refresh glyph joins its visible label (it stays hidden on
  desktop, where the label carries the pill)

This matches the established mobile full-width pill language (New
chat / Share / stale-ban Regenerate). The three unit tests that
pinned the old icon-only CSS are updated to pin the new behavior.
2026-09-07 14:16:41 -04:00
ducoterra 7fce6572d0 feat: phases 77–80 — navbar view refresh, static background, API tokens, history suggestion chips
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Single consolidated commit for four completed, validated phases (77, 78,
79, 80). The pipeline run left all work uncommitted because the harness
commits only with PHASE_COMMIT=1 while child executors are forbidden from
committing; the phases themselves all passed validation and moved to
.agents/phases/complete/.

Phase 77 — navbar view refresh
- router.js dispatches bor:view-refresh on re-show / active re-click /
  popstate (gated on wasMounted; first show and boot exempt)
- History / RAG / Sources / Tuning re-fetch on refresh (admin branch);
  Chat deliberately excluded (stream survival)
- History "Refresh" button (admin-only, in-flight disable + status line)
- New story suite tests/e2e/test_navbar_refresh.py (7 tests)

Phase 78 — static background
- Removed the animated glow layers; static 44px grid over the flat --bg
  canvas; default and reduced-motion renders byte-identical
- Updated background/theme E2E suites; removed bg-glow test pins

Phase 79 — API tokens
- api_tokens model + migration 0012; hash-only token service
- Admin tokens API + Tokens admin view; POST /api/token-auth;
  live-revoking require_user on chat / suggestions / document content
- Frontend token gate with localStorage cache; anonymous E2E suites
  migrated to token login
- New story suite tests/e2e/test_api_tokens.py (9 tests)

Phase 80 — history suggestion chips
- last_questions() endpoint with SEED fallback; startNewChat() refetch
- Seed-semantics docs (config.py, .env.example, README)
- Integration state matrix + E2E suite rewritten to the 4 chip states

Also included: phase-76 report artifacts and the repo restore-test-db
skill (previously untracked), scripts/* ruff fixes from phase 77.

Final gate state (phase 80 final pass, covers everything above):
- uv run pytest --cov=app → 1637 passed, 0 failed, app/ coverage 99%
- uv run ruff check . && uv run pyright → clean, 0 errors
- Per-phase story E2E suites green in isolation
2026-09-07 12:39:01 -04:00
ducoterra 495d042a98 chore(agent): phase roadmap from TODO.md — 4 phases (77–80)
Build and Push Containers / build-and-push-app (push) Successful in 1m54s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Protocol B append: navbar refresh + History refresh button (77, TODO L3),
static background — glow layers removed (78, TODO L4), admin-issued API
tokens with the in-app gate + browser caching, only shared chats stay
anonymous (79, TODO L5), onboarding chips as the last 3 questions asked
with the env seed only before the first (80, TODO L6).

TODO.md cleared — its items now live in .agents/phases/todo/.
Owner-confirmed assumptions recorded in each phase overview
(A1–A7, chat 2026-09-06).
2026-09-06 23:54:11 -04:00
ducoterra b78afc08f2 docs(bench): add chat model results to CSV benchmark
Added 6 rows for chat model results (lite + turbo, fixture + derived) to benchmarks/model_benchmarks.csv.
2026-09-06 21:59:06 -04:00
ducoterra f221b40fce feat(agent): add CSV benchmark recorder + summary/embedding test scripts and skills
New files:
- scripts/model_benchmark.py — shared CSV recorder for all model tests
- scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination)
- scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed)
- .agents/skills/test-summary-model/SKILL.md — skill for testing summary models
- .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models
- benchmarks/README.md — schema documentation

Updated:
- .agents/skills/test-chat-model/SKILL.md — now also records to CSV

All three scripts write to benchmarks/model_benchmarks.csv with one row
per run per check. The CSV accumulates results across runs for comparison.
2026-09-06 21:58:35 -04:00
ducoterra 70ba8710f3 docs(agent): record the turbo sanity check on the controlled fixture battery
2026-09-06 fixture runs: contract 100 %, executed 100 %, wall ~113 s (2 runs). Derived battery: FAIL only on usage floor (5/10 tool-turns) — answers seeded questions from context, which is ideal grounded behavior. Wall time ~2.8× lite (113 s vs 40 s). Model is clean.
2026-09-06 21:49:57 -04:00
ducoterra bf64c0d7e4 docs(agent): record the lite comparison on the controlled fixture battery
2026-09-06 fixture runs: contract 92–93 %, executed 64–75 %, wall ~40.5 s (2 runs). Derived battery: FAIL, 36 % executed (38.3 s). Same pattern — copy-invariant re-read habit blocks the ≥90 % executed bar under current ALREADY_IN_CONTEXT refusal semantics. Model is working correctly; the bottleneck is the app's dedupe refusal, not the model.
2026-09-06 21:42:14 -04:00
ducoterra ffa919b8bf fix(chat): keep in-flight answers alive across in-app view switches
Root cause (owner repro, verified in a real browser 2026-09-06): the
five navbar views (Chat, RAG, Sources, Tuning, History) were separate
HTML documents, so a navbar click was a REAL cross-document navigation
— the chat page unloaded, the in-flight SSE fetch was aborted, and the
phase-48 teardown (app/api/chat.py `finally`, "chat: turn cancelled")
stopped the model. Observed: send question -> click RAG mid-stream ->
click Chat -> the answer never finished: no `query_log` row, and on
return a dangling question with no brain record (the pre-token pagehide
partial persist skips because `acc` is empty).

Phase-48 LOCKED-DECISION REFINEMENT (owner-confirmed 2026-09-06,
flagged per AGENTS.md rule 3, not silently deviated): "real navigation
cancels the fetch" now means LEAVING THE APP — tab close,
external/other-document navigation, the Stop button. In-app navbar
switches are client-side view switches and no longer cancel.

Fix — Option A (SPA shell), chosen over B (Service Worker owns the
stream) and C (server-side turn registry + resume):
- frontend/index.html is the shell: ONE `<main id="main">` holds the
  five `<section class="view">` blocks; hidden views carry BOTH
  `hidden` and `inert` (WCAG — no focus/keyboard traversal). The
  shared header, the single `doc-modal-*` skeleton, and the
  `#app-version` footer each exist exactly once; the per-view copies
  from the four folded pages are dropped.
- New frontend/assets/router.js (vanilla module — no framework, no
  bundler, No-CDN rule intact): lazy-imports a view module on FIRST
  show only (mount-once, hide-forever — the chat view's in-flight SSE
  reader persists across switches; that persistence IS the fix);
  intercepts same-shell navbar links with preventDefault +
  history.pushState (never a document load); handles popstate; single
  writer of `.nav-link` active state (is-active + aria-current),
  document.title, and the per-view meta description (values carried
  over from the old pages' heads, brand-resolved at write time).
- Each folded page's JS becomes `export async function mount(root)` —
  root-scoped queries; `initSharedHeader()` dropped (the header boots
  once in the shell via the chat module; the admin flag comes from the
  same cached `fetchIsAdmin()` promise — zero extra requests).
- app/main.py: a small list-driven route factory serves the shell for
  /tuning.html, /sources.html, /git-sources.html, /history.html —
  registered AFTER the API routers and BEFORE the static catch-all
  (routes-first). The phase-33 caching middleware applies no-cache +
  `?v=` rewriting unchanged; app/core/caching.py needed NO change
  (the view paths did not change — pinned by the integration tests).
- The four old view .html files are DELETED (one source of truth);
  deep links to the old URLs keep working (the router picks the view
  from the pathname); `/?chat=<id>` is unaffected; the Containerfile
  bundles router.js (inlining the lazy view modules) and drops the
  folded page files.
- app/schemas.py: HistoryTurn.text cap 4000 -> 32000 — the shell
  keeps long saved answers in the chat, and the old cap (stricter than
  the 24_000-char total history budget) 422-rejected any second turn
  in such a chat (found by the phase-42 E2E suite on the shell).

Boundaries: login.html, shared.html, doc-edit.html, document.html
REMAIN separate documents (flow pages, not navbar tabs); a mid-stream
navigation to doc-edit/document.html still cancels per phase 48
(follow-up candidate, out of scope). The SSE API is unchanged. Real
departures still cancel the turn — phase 48 intact (pinned by
tests/e2e/test_stop_generation.py, unchanged, and by the new suite's
real-departure control).

Tests:
- Phase-20 suite REWRITTEN to the new semantics
  (tests/e2e/test_sources_midstream_bug.py): a navbar switch no longer
  cancels — the stream survives the switch and the FULL answer
  settles; the pagehide partial persist REMAINS for real departures
  (the partial's exact shape — first streamed chunk prefix, no done
  metadata — is still pinned there).
- NEW story suite tests/e2e/test_nav_switch_keeps_stream.py (mock
  LLM): the owner repro (send -> RAG mid-stream -> Chat: window
  sentinel survives = same document, FULL answer, exactly one brain
  turn in bor.chat.v1, exactly one settled query_log row, auto-saved
  row matches) + the same mid-stream switch against the other three
  views + the real-departure-still-cancels control + the no-switch
  baseline.
- tests/unit/test_frontend_router.py: source-level pins of the router
  invariants (click interceptor targets ONLY same-shell view paths,
  pushState-only switches, mount-once guard, hidden+inert pair,
  single-writer active state/title); shell-route integration tests
  (each folded path serves the shell with no-cache + `?v=` body; a
  non-view path still 404s); the file-reading unit pins re-pointed at
  the shell (the four view files are gone — the shell is the source
  of truth).

Verification (this commit): full suite green — 1565 unit+integration
tests, app/ coverage 99% (>90% floor); ruff + pyright clean; the
phase's E2E suites green in isolation (house protocol, AGENTS.md rule
9). Owner repro verified in a real browser against the real LLM
(dev server :8010, headful Chromium): "tell me about everquest" ->
RAG mid-stream -> Chat — the answer completed with one brain bubble
and no error banner, `query_log` gained exactly one settled row
(deflected=True: the dev KB holds no EverQuest docs — the settle, not
the topic, is the proof), zero "chat: turn cancelled" lines for that
turn; the control (real navigation to /shared.html mid-stream) still
cancelled (no settled row, the cancel line logged, the partial
persisted on return). Screenshots: .agents/screenshots/76_manual_*.

Phase 76 (76_spa_nav_shell) complete — moved to
.agents/phases/complete/.
2026-09-06 06:31:31 -04:00
ducoterra 7e567bddf3 prepping for SPA conversion
Build and Push Containers / build-and-push-app (push) Successful in 14s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-05 21:50:56 -04:00
ducoterra 0e4651c779 feat(docs): save the whole chat session as a doc
Build and Push Containers / build-and-push-app (push) Successful in 1m46s
Build and Push Containers / build-and-push-db (push) Successful in 12s
Phase 75 (TODO.md L4): "Save as doc" now drafts a document from the
ENTIRE chat session — every question and answer up to the click, in
order — instead of only the clicked bubble's answer; the existing
doc-edit screen's free-form body editing is how the user edits out
anything they don't want to keep from previous replies (no new UI
surface).

Task 01 (frontend):
- app.js buildSessionTranscript(): walks the bor.chat.v1 conversation
  record in order — a numbered section per user turn ("## N.
  <question, raw>" + blank line + the raw answer text; more answers
  join under the same heading), sections blank-line separated, all
  trailing whitespace collapsed to one final newline. Only the raw
  persisted text travels (m.who + m.text — no thinking blocks, no
  source chips, no tune metadata); a brain record before the first
  user record is skipped; a heading-only section marks a user turn
  whose answer never landed (A6, owner-confirmed 2026-09-08).
- saveAsDoc(btn): the draft body is buildSessionTranscript(); the
  dead single-bubble markdown parameter is dropped (the button's
  appendSaveAsDocButton signature is unchanged — one button per
  bubble). Title/path/double-click guard/hand-off are unchanged
  (defaultDocTitle: the last question, whitespace-collapsed,
  <=120 chars; docs/<slug>.md).
- Unit: the app.js source pins move to the transcript shape (whole
  session, no thinking, no dead parameter).

Task 02 (E2E):
- tests/e2e/test_save_doc_session.py (bare-repo fixture, the
  phase-59 convention — git as source of truth): three DISTINCT
  on-topic turns in one session (turn 1 carries the phase-17
  "think out loud" trigger so its record has a thinking block the
  transcript must exclude) -> save on the LAST bubble -> the
  prefilled body is ## 1./## 2./## 3. in order, byte-exact against
  the deterministic mock, thinking-free -> edit the whole
  section-2 block out of the body -> push -> git show
  bor-docs:<path> equals the EDITED body byte-for-byte (section 2's
  question and answer provably absent; sections 1 and 3 byte-exact;
  the UI's sha prefix is git rev-parse bor-docs). Second test:
  the button on the FIRST bubble still drafts the whole session
  (A6 — the transcript is the session at click time, title stays
  the last question); canceling leaves the branch tip untouched.
- tests/e2e/test_response_to_docs.py: the phase-59 single-turn body
  expectation moves to the transcript shape ("## 1. <question>" +
  the answer's markdown) — the rest of the suite unchanged.

Also lands the phase-74 file moves (00_phase.md /
03_mock_marker_e2e.md -> complete/) and the phase reports — the
house convention of committing .agents/ with the phase.
2026-09-05 16:59:31 -04:00
ducoterra 055c0b5d85 feat(rag): pass chat history with prior thinking to the LLM
Build and Push Containers / build-and-push-app (push) Successful in 1m39s
Build and Push Containers / build-and-push-db (push) Successful in 11s
Phase 74 (TODO.md L4): a follow-up question now reaches the model WITH
the conversation so far — every prior user/brain turn and the prior
thinking blocks on brain turns (preserve-thinking) — while
POST /api/chat stays stateless (A10): the client provides the history
in the request body and the server stores nothing new.

Server (task 01):
- ChatRequest.history: optional list[HistoryTurn] (who: user|brain,
  text, optional thinking) — absent/empty keeps the request
  byte-identical to pre-phase-74 (the two-message [system, user]
  request; the kill-switch semantics are pinned in the integration
  suite).
- app.rag.prompts.history_to_messages: pure mapper — walks the turns
  newest-first against the settings budgets (history_max_turns=40 /
  history_max_chars=24000, BOR_HISTORY_MAX_TURNS /
  BOR_HISTORY_MAX_CHARS); a capped turn is dropped WHOLE (never cut
  mid-answer); the kept window is returned oldest-first; brain turns
  carry their thinking as reasoning_content (A4) only when
  non-empty.
- Both branches feed it: the deflected path splices it between the
  system prompt and the current user message (the phase-71 recovery
  still rebuilds from messages[1:]), the grounded agent receives
  run_agent(..., history=hist); llm.py's message params widen to
  list[dict[str, Any]] (string-only messages stay byte-identical on
  the wire — the SDK passes message dicts through verbatim).
- The per-turn log line (PLAN §9) gains history_msgs=N after
  kb_chars=N.
- Pins: tests/unit/test_history.py (mapper: mapping, reasoning
  gating, both budgets, drop-whole, ordering, empty default),
  tests/unit/test_config.py (the two settings + env overrides),
  tests/unit/test_agent.py (the history splice + the default),
  tests/integration/test_chat_api.py (deflected AND grounded forward
  the history incl. reasoning_content, no-history byte-identity, 422
  pins, the log field).

Client (task 02):
- runTurn — the single funnel for fresh send / phase-49 retry /
  phase-53 stale-regen — sends history = the conversation record
  minus the current question, with thinking only on brain records
  that streamed one (undefined drops the key from the JSON, the
  record's convention); the question is never duplicated into the
  history.

Wire proof (task 03):
- The mock's echo my history marker (HISTORY_TRIGGER) answers with
  the deterministic history echo — history: N prior messages; last
  answer tail: <last 24 chars>; thinking: yes|no — checked BEFORE
  the DEFLECT_MODE branch (like TABLE_TRIGGER), so it fires on both
  turn branches whatever the gate says; the module docstring records
  the user/assistant-only history invariant that keeps every
  existing (tool-result-classified) marker flow unaffected.
- tests/e2e/test_llm_history.py (isolated): a grounded follow-up and
  a deflected follow-up both receive history: 2 prior messages +
  thinking: yes + the byte-exact tail of turn 1's answer (derived
  from the persisted bor.chat.v1 record — the same array the client
  maps into the body); a cold start receives history: 0 prior
  messages / last answer tail: none / thinking: no.
- Regressions green in isolation: chat_rag, chat_history (phase 50),
  agent_document_tools, harness_aligned_tools, stop_generation,
  retry_answer, response_to_docs.
2026-09-05 16:04:40 -04:00
ducoterra a16130c71d fix(chat): keep generating while the tab is hidden
Root cause (task 01): none of C1-C3 - in Chromium 151 (real mode) a
merely-hidden tab neither stops the stream (frames arrive at full rate;
turn completes) nor fires pagehide on tab switch; C1's double-record
path was proven latent via a synthetic pagehide (trigger is
browser-dependent, e.g. Safari) and C2 (the 120s pre-token guard) was
confirmed to fire while hidden.

- C1: the pagehide partial-persist is correlated with the turn's settle
  (leavePartialIndex) - the done/stop settle REPLACES it in place
  (identity-guarded rememberBrainTurn in-place mode), so bor.chat.v1
  and the auto-saved saved_chats row keep exactly ONE brain turn per
  question; a real navigation never runs a settle, so the leave-save
  is unchanged.
- C2: the visibility re-arm gives the still-armed pre-token guard a
  fresh TURN_TIMEOUT_MS when the tab returns to visible - hidden time
  no longer counts toward the 120s guard.
- Phase-48 teardown contract untouched: Stop / tab close / real
  navigation still cancel the fetch and stop the model.
- Unit pins: tests/unit/test_frontend_hidden_tab.py (the app.js
  mechanisms without a browser).
- E2E pins: tests/e2e/test_hidden_tab_stream.py - synthetic pagehide
  mid-stream completes exactly once with one brain turn (localStorage
  + auto-saved row), reload restores one bubble, no-event baseline,
  and the fake-clock pre-token guard re-arm (discriminating: fails
  with the re-arm disabled).
2026-09-05 14:34:33 -04:00
ducoterra 45c3fa2863 chore(agent): phase roadmap from TODO.md (phases 73-75), clear the file
Convert the two unchecked TODO items into executable phases (Protocol B,
appended after the 72 completed phases):

- 73_hidden_tab_stream (TODO L3): a merely-hidden browser tab must never
  stop a generating answer; repro/root-cause decision tree + the pagehide
  partial-correlation fix + the hidden-tab E2E pin.
- 74_llm_chat_history (TODO L4, history): client-provided history in
  POST /api/chat (stateless, A10) mapped through both the deflected and
  grounded agent paths, prior thinking blocks preserved via
  reasoning_content, capped oldest-first; mock echo marker + E2E.
- 75_save_doc_full_session (TODO L4, save-as-doc): the Save-as-doc draft
  body becomes the full session transcript; edit-out happens in the
  existing doc-edit body; multi-turn git-verified E2E.

Owner-confirmed assumptions A1-A7 are recorded as ASSUMPTION lines in the
task files. TODO.md is cleared (items now live in .agents/phases/todo/).
2026-09-05 12:14:21 -04:00
ducoterra dbf2af26c6 refactor(agents): migrate .agent/ planning tree to .agents/
Standardize on the .agents/ directory (shared with project skills):
phases/, user_stories/, reports/, screenshots/, validate.sh, and
phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves
history; runtime artifacts move alongside).

Updates every reference in AGENTS.md, README.md, .gitignore, app
docstrings, and test story headers. Historical KB content in data/
and the runtime pipeline.log transcript are left untouched.
2026-09-05 10:57:07 -04:00
ducoterra 766702c750 finally getting accurate answers
Build and Push Containers / build-and-push-app (push) Successful in 1m46s
Build and Push Containers / build-and-push-db (push) Successful in 12s
2026-09-05 10:26:39 -04:00
ducoterra bb2803bebd feat(skills): add test-chat-model skill — add a chat model and run the controlled tool-calling battery
Codifies the 2026-09-05 turbo comparison workflow as a project skill under
.agents/skills/: switch BOR_LLM_CHAT_MODEL in .env, run the fixture gate
(twice, for variance) + the locked derived gate with per-turn wall timing,
interpret the two metrics against the reference model rates (re-read habit:
lite ~100%, turbo ~12%; usage-floor MISS as test artifact; caps as real
regression), record the verdicts byte-exact in TOOL_CALLING_TESTING.md, and
commit the doc. Rules baked in: never touch the battery/thresholds/fixtures,
never edit app code, never commit .env.
2026-09-05 00:13:21 -04:00
ducoterra ce896ab8ab docs(agent): record the turbo comparison on the controlled fixture battery
turbo (2026-09-05, same fixture KB): fixture gate PASS 100%/100% on both
metrics, two runs (wall 105-135s vs lite 43-55s); the redundant re-read
of seeded documents that capped lite's executed ratio at 58-73% is
model-specific (turbo re-read rate ~12% vs ~100% in-sample), corroborating
section 7's framing. Locked derived battery: turbo fails only the >=6/10
tool-turn usage floor (it answers seeded read-target questions from
context instead of making the refusable read call) - accuracy on all
emitted calls still 100%/100%.
2026-09-05 00:09:55 -04:00
ducoterra 988ff78526 fix(agent): teach the document-identity contract on ls/read/grep refusals — end the post-harness tool-loop rambling
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Build and Push Containers / build-and-push-db (push) Successful in 14s
Phase 72 (72_teaching_refusals) — completed under the 2026-09-04 controlled
methodology (owner directive: stop clearing/re-importing the homelab KB per
iteration; measure tool-calling accuracy on a controlled fixture KB, target
>90%).

Real-model gate verdicts (live, configured chat model 'lite', fixture KB):
- Controlled fixture battery (the new methodology's pass condition —
  contract accuracy >= 90%): PASS, 4 consecutive runs:
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/11 executed (73%) contract 11/11 (100%) 2026-09-04 (wall 43.4s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 8/13 executed (62%) contract 12/13 (92%) 2026-09-04 (wall 50.6s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 7/11 executed (64%) contract 11/11 (100%) 2026-09-04 (wall 46.8s)
  gate: lite PASS turns=10 answered=10 caps=0 tool-turns=10 calls 9/15 executed (60%) contract 14/15 (93%) 2026-09-04 (wall 54.8s)
- Locked derived battery (phase-72 task 05, executed >= 90% bar, run
  unchanged on the same fixture KB):
  gate: lite FAIL turns=10 answered=10 caps=0 tool-turns=10 calls 5/15 executed (33%) contract 12/15 (80%) 2026-09-04 (wall 47.7s)
  The teaching works — every bare-path trap self-corrects in exactly one
  round, zero cap hits, zero repeat loops, 10/10 answered. The locked
  executed bar is blocked by ALREADY_IN_CONTEXT dedupe refusals on the
  corrected re-reads (the trap question seeds its target, so the correct
  combined-form read is refused for redundancy) — a copy-invariant model
  behavior (five copy variants, 0/15 re-reads flipped, 2026-09-03 -> 04)
  and an app-semantics decision for the owner (TOOL_CALLING_TESTING.md
  sections 5 and 7), not a copy lever.

Copy changes this phase owns (unit pins updated to follow):
- app/rag/agent.py: ls teaching refusals (path-like scope -> document-path
  line; unknown source -> no-source line with the source-name
  parenthetical), read/grep 'did you mean source/path?' teaching
  (find_path_candidates: exact or suffix path match, catalog order, cap 3),
  ALREADY_IN_CONTEXT naming the correct action (answer from the text
  already in the prompt), read tool description front-loaded with the
  do-not-read rule (the 2026-09-04 controlled telemetry: the re-read is
  the only remaining refusal class; contract accuracy 92-100% across runs)
- app/rag/prompts.py: TOOLS_SECTION states the document-identity contract
  up front (ls path = source name; read/grep = combined source/path
  including the source name; do-not-read for <documents> documents placed
  next to the read teaching; one-call-per-reply and never-repeat rules)
- tests: refusal pins (unit + integration), new dedicated E2E suite
  tests/e2e/test_tool_path_teaching.py (mock misuse flow, green in
  isolation), regression suites green in isolation (harness_aligned_tools,
  agent_document_tools, agent_unlimited_tools, search_tool, chat_rag).

Gates: uv run pytest green (1501); coverage TOTAL 99% (>90%); ruff +
pyright clean. Carries the still-uncommitted phase-71 todo/ -> complete/
move and both phases' .agent/reports/ (AGENTS.md 8).
2026-09-04 13:11:07 -04:00
ducoterra 7909bdb8da test(agent): controlled fixture KB + one-command fast loop for tool-calling iterations
The phase-72 iteration loop cleared the database, git-cloned the homelab repo, re-imported 38-51 documents and re-embedded per run — many minutes per iteration against a different KB every time (owner directive 2026-09-04: stop importing the homelab repo on every test run). Replace it with:

- tests/fixtures/agent_kb/: 8 hand-written markdown docs (sources 'deployments'/'homelab') whose specifics (rack7, 10.77.42.0/24, VLAN 130, rbm-8842, 17 2 * * *, obsidian-bor:2026.7.14, 18765, 18443, ...) no model can guess; read targets carry non-topical filenames so their questions do not lexically seed them (the read must actually happen)
- tests/fixtures/test_kb.dump.sql: data-only snapshot (TRUNCATE + INSERTs incl. embeddings, self-contained git_sources rows, static KB overview) — verified by round-trip checksum at build time
- scripts/load_test_kb.py: one-off rebuild (real pipeline + embeddings, ~2s) that also prints the per-question retrieval report (all 10 battery questions must be grounded)
- scripts/restore_test_kb.py: sub-second one-transaction restore (no git clone, no re-embedding)
- scripts/agent_realmodel_check.py: the gate gains --restore / --mode fixture (curated 10-question battery with one unambiguously correct tool behavior per question) / --turns N (12s micro-loop) / --concurrency / per-turn + total wall timing, and a second accuracy metric (contract accuracy: well-formed calls targeting resolvable entities) alongside the phase-72 locked executed ratio — the re-read of a seeded doc is a copy-invariant model behavior (5 variants, 0/15 flipped) that the dedupe refusal counts as a failure
- TOOL_CALLING_TESTING.md: the human-readable methodology (fast loop, design rules, metrics, copy levers + tried-and-reverted table, current standing, open design question)

Measured: restore 0.03s; micro-loop ~12s; full loop ~43-55s; concurrency 2/3 gives no gain (endpoint serializes).
2026-09-04 13:10:15 -04:00
ducoterra 575d6c88d0 feat(agent): strip raw tool-scaffolding from streamed answers — deterministic filter with one bounded recovery 2026-09-03 13:39:15 -04:00
ducoterra 801639efcc feat(agent): align the document tools with the harness-trained shape — ls, read(path), grep(pattern, path?) 2026-09-03 11:17:47 -04:00
ducoterra 16f1cfbcaf add additional phases and launch.json
Build and Push Containers / build-and-push-app (push) Successful in 2m6s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-02 18:59:09 -04:00
ducoterra 0bf96f22e1 fix(agent): make read_document robust to combined source/path arguments
The model treated the combined 'source/path' string (as printed in
search result lines, read-result headers and refusals) as the
document's identity and passed it as 'source' — e.g.
source='homelab/active/container_caddy/caddy.md' instead of
source='homelab', path='active/container_caddy/caddy.md'.

- Rewrite the read_document description with the split rule (source =
  before the FIRST '/', path = after it) and a worked example; share
  the source/path parameter descriptions between read_document and
  search_documents; map search result lines back onto the split.
- New _resolve_document: on a lookup miss with a '/' in source, retry
  at the first slash (source names are directory basenames and can
  never contain '/'), plus a continuation candidate for a split at a
  later slash; a self-corrected combined form for an already-in-context
  document is still rejected as ALREADY_IN_CONTEXT.
- A slash-carrying source that matches nothing gets an educational
  refusal naming the corrected arguments instead of the generic line
  that repeated the combined form.

Verified live against aipi (lite) + the imported homelab KB: A/B on
the exact failure scenario (5 runs each, right after a
combined-source search result) — old descriptions 5/5 combined, new
descriptions 5/5 clean; two live UI turns (Playwright) produced only
clean split arguments, including a multi-hop read of
install_caddy_deskwork.yaml that landed in done.sources. Full suite:
1376 passed, app coverage 99% (agent.py 100%), ruff + pyright clean,
agent/search E2E green in isolation.
2026-09-02 17:42:57 -04:00
ducoterra 137d5fa1a5 feat(sources): removing a source deletes its files and index entries behind a confirmation modal
Build and Push Containers / build-and-push-app (push) Successful in 1m29s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-02 15:55:33 -04:00
ducoterra 265e736b3d add phase 68 complete 2026-09-02 13:33:45 -04:00
ducoterra 189aa92182 fix tool descriptions to prevent incorrect tool calls
Build and Push Containers / build-and-push-app (push) Successful in 1m32s
Build and Push Containers / build-and-push-db (push) Successful in 10s
2026-09-02 13:32:58 -04:00
ducoterra 8cf3a827ee feat(agent): search_documents tool — the model can grep the indexed documents for an exact string
Build and Push Containers / build-and-push-db (push) Canceled after 0s
Build and Push Containers / build-and-push-app (push) Canceled after 1m11s
2026-09-02 12:04:34 -04:00
ducoterra 88293ed02f feat(rag): retry a failed LLM request before the first token lands — BOR_LLM_RETRIES/BOR_LLM_RETRY_DELAY with a live 'retrying' status 2026-09-02 10:52:38 -04:00
ducoterra f04ddbe1f8 fix(web): history tab copy — every chat saves automatically, there is no Save button
Build and Push Containers / build-and-push-app (push) Successful in 1m35s
Build and Push Containers / build-and-push-db (push) Successful in 10s
meta description -> locked (A3) auto-save string (history.html L6).
page-sub -> locked (A3) string, the <strong>Save</strong> emphasis retired with the button (L106-109).
empty row -> locked (A3) string; colspan=6, hidden, and row id untouched (L163).
h1, the anonymous gate section, and history.js are byte-identical — state language verified accurate.
2026-09-02 01:29:42 -04:00
ducoterra 8a1f99cb38 feat(web): move the chat action cluster to the pinned bottom and align the button sets
- task 01: relocate the .chat-actions row (New chat + Share, comments byte-identical with a Phase 65 note) from the top of the column to the bottom of .chat-shell, directly above the composer
- task 02 (owner-locked A1): wrap the row + #composer in ONE sticky .chat-bottom unit (position: sticky; bottom: env(safe-area-inset-bottom, 0), no z-index) — the pills stay at the bottom of the screen at every scroll position and settle into flow above the footer
- task 03 (owner-locked A2): right-align the bottom row to the column's right edge (justify-content: flex-end), mirroring the right-aligned Save-as-doc corner; the five action pills share one 44px / 999px-pill geometry
- task 04: dedicated Playwright suite tests/e2e/test_bottom_chat_actions.py (resting geometry, the A1 pin across the sticky range, A2 alignment + DOM order + mobile stack + 360px overflow bound + 44px touch targets, New chat / Share click-through) — green in isolation
- task 05: regression matrix green in isolation (pinned_composer 4, save_share_ux 5, chat_persistence 4, share_chat 4, chat_history 5, smoke 3); full gate green — unit + integration pass, app/ coverage 99% (>90%), ruff + pyright clean
2026-09-02 01:08:18 -04:00
ducoterra 4677d86f49 feat(sources): real-time file progress for sync and upload — background upload with success toast 2026-09-01 23:51:43 -04:00
ducoterra cddc84c7db chore(agent): phase roadmap from TODO.md — 67_llm_retry + 68_search_tool 2026-09-01 20:15:08 -04:00
ducoterra 66419bf652 chore(agent): phase roadmap from TODO.md, 2 phases
Phase 65 (TODO.md L3): move the New chat + Share cluster to the pinned
bottom of the chat column (sticky .chat-bottom unit with the composer)
and right-align the row to the Save-as-doc action corner.
Phase 66 (TODO.md L4): History tab copy — every chat saves
automatically; retire the Save-button references.
2026-09-01 19:53:56 -04:00
ducoterra a6a1bf7143 chore(agent): phase roadmap from TODO.md, 1 phase (64_sync_upload_progress) 2026-09-01 19:27:31 -04:00
ducoterra 15a16a8fe0 fix(agent): unambiguous document listing format for LLM parsing
Build and Push Containers / build-and-push-app (push) Successful in 1m34s
Build and Push Containers / build-and-push-db (push) Successful in 10s
2026-09-01 12:44:53 -04:00
ducoterra c738105932 feat(web): customizable placeholder, footer text, and color theme via BOR_* env vars
BOR_INPUT_PLACEHOLDER / BOR_FOOTER_TEXT / BOR_THEME (+ the indigo.css example theme); authoring guide: frontend/assets/themes/README.md, docs: README 'Customizing the look'.
2026-09-01 12:04:06 -04:00
ducoterra baefcde668 fix(web): retire the stale homelab-era copy — neutral, accurate defaults on every page
Fixed: index.html meta description, empty-state sub and composer
placeholder (A1); app/config.py default suggestion chips → the four
neutral A2 defaults (BOR_SUGGESTIONS override unchanged); sources.html
KB page-sub → the current source model (git repos + local dirs +
uploaded archives, Sync pulls/imports); git-sources.html example URL
→ your-repo.git (A3); all 9 footers → neutral default in
span.footer-text (the phase-62 hook); E2E/unit conftests force the
code defaults so a local .env cannot leak corpus copy into tests;
new unit text pins + dedicated E2E suite.

Task 02 verification read-through — no change needed:
- sources.html sync result/error copy (matches the real sync behavior)
- tuning.html page-sub (accurate as written)
- history.html page-sub (accurate as written)
- doc-edit.html page-sub (accurate as written)
- git-sources.html page-sub (accurate as written)
- #sources-gate anonymous copy (accurate as written)
2026-09-01 10:54:50 -04:00
ducoterra 4971e2859d chore(agent): track .agent/ planning tree in git
Build and Push Containers / build-and-push-app (push) Successful in 12s
Build and Push Containers / build-and-push-db (push) Successful in 10s
Remove the blanket .agent/ gitignore so the phase roadmap, user
stories, reports, and PLAN.md are versioned with the code. Only
runtime artifacts (.agent/phase-sessions/, .agent/pipeline.log)
remain ignored. Update AGENTS.md git protocol rule to match.
2026-09-01 10:18:22 -04:00
ducoterra 5fa620fde5 fix(web): keep the navbar stuck to the top — drop the body height cap on the sticky range
Build and Push Containers / build-and-push-app (push) Successful in 2m28s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-09-01 04:20:26 -04:00
ducoterra 725af9fac1 feat(docs): save chat answers as docs — edit screen, commit + push to the .env docs branch 2026-09-01 03:52:03 -04:00
ducoterra 7b7a834a1a feat(web): 2x reading column on wide desktops — 92rem at >=1500px (chat, shared, document view) 2026-09-01 00:39:19 -04:00
ducoterra 140b97ebf3 feat(kb): edit + re-embed document summaries from the viewer (admin) 2026-08-31 23:52:22 -04:00
ducoterra d94f3d5a52 feat(import): user-extensible BOR_IMPORT_EXTENSIONS — any well-formed extension, A9 family stays the default 2026-08-31 22:42:41 -04:00
ducoterra 281f3555c3 remove .agent, it shouldn't be committed. Do not commit it again
Build and Push Containers / build-and-push-app (push) Successful in 1m47s
Build and Push Containers / build-and-push-db (push) Successful in 14s
2026-08-31 08:08:16 -04:00
ducoterra 914097abcf feat(chat): save by default + share anonymously — auto-saved chats, guest-facing Share, success toast, action row 2026-08-31 05:20:25 -04:00
ducoterra c564e317ed fix(web): never 304 a rewritten page — pages drop conditional validators, assets keep them 2026-08-31 01:35:28 -04:00
ducoterra 9518d9d5d1 chore(agent): phase 53 — record task 06 completion move to complete/ 2026-08-31 00:04:17 -04:00
ducoterra 32b7bfd4b3 feat(chat): invalidate saved chats on sources sync — versioned stamps, stale marker, Regenerate against the new index 2026-08-30 23:39:15 -04:00
ducoterra ea8e041189 chore(agent): phase roadmap from TODO.md — 1 phase (55 save/share UX) 2026-08-30 21:13:23 -04:00
ducoterra aba8615177 fix(chat): rest the composer at the viewport bottom — sticky alone left it mid-screen
Phase 52's first pass shipped `position: sticky; bottom` on `.composer` and
called the phase done, but the owner's requirement — "the chat message-input
textarea should be at the bottom of the screen" — still failed in the browser:
on an empty/short chat the input rested just under the empty state (~57% of
the viewport) with a dead band down to the footer.

`position: sticky` can only pull a box UP toward the scrollport's bottom edge;
it can never push a box DOWN to meet it, so on a page that does not overflow
it is a no-op. The old story suite only exercised an overflowing conversation
(one test even asserted the buggy resting position as expected), which is why
the half-fix passed.

- `.messages { flex: 1 1 auto }` — absorbs a short page's free space so the
  composer's resting in-flow position is the bottom of the full-height column
  (body min-height:100dvh -> .app-main flex:1 -> .chat-shell flex:1); basis
  stays `auto`, no height cap, no overflow — the document stays the scroller
- `.composer { bottom: env(safe-area-inset-bottom, 0) }` — the explicit 0
  fallback replaces the env()-only offset, which degraded to `auto` (no pin)
  wherever env() is unsupported
- E2E: `test_empty_chat_composer_sits_in_normal_flow` ->
  `..._at_the_screen_bottom` (chrome-only band below the resting composer);
  the phone suite now checks the resting position as well as the pinned one
- Unit pins: the flex-grow half and the full-height column are pinned, so the
  fix cannot silently regress to sticky-only

Still CSS-only — no DOM change, no JS, no new scroll call site (phase 42
never-auto-scroll contract intact), no z-index.

Verified: 1019 unit/integration tests pass (app/ coverage 99%), ruff and
pyright clean; tests/e2e/test_pinned_composer.py green in isolation (4), plus
the stop/autoscroll/persistence/mobile-nav suites and 14 layout/scroll
neighbours green in isolation.
2026-08-30 16:14:18 -04:00
ducoterra 820753948e feat(chat): pin the composer to the viewport bottom — Stop is always reachable while reading 2026-08-30 14:59:40 -04:00
ducoterra 619bf2187a chore(agent): phase roadmap from TODO.md — 2 phases (52 pinned composer, 53 stale saved chats) 2026-08-30 09:33:50 -04:00
ducoterra 114b115034 feat(chat): share a chat by link — anonymous read-only /shared/<token> page, share/unshare 2026-08-30 01:34:44 -04:00
ducoterra ece93a7c8f feat(chat): save and view chat history — admin-only saved_chats, History page, open-a-chat return 2026-08-29 21:22:25 -04:00
ducoterra 6832957ab0 feat(chat): retry the last answer — redo-in-place Retry button on the latest brain bubble 2026-08-29 18:32:15 -04:00
ducoterra 1a60ecbd8b feat(chat): stop an in-flight answer — Send becomes Stop, the partial is kept and persisted, the model stream is torn down 2026-08-29 17:27:04 -04:00
ducoterra 6bf7f456d4 show sync button for admin
Build and Push Containers / build-and-push-app (push) Successful in 1m38s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-08-28 23:24:23 -04:00
ducoterra 3a404eb161 fix(chat): stop the submit up-hop and keep the thinking pin alive across paragraph breaks
Build and Push Containers / build-and-push-app (push) Successful in 3m55s
Build and Push Containers / build-and-push-db (push) Successful in 13s
- scrollReveal lands at the document bottom (window.scrollTo) instead of
  scrollIntoView({ block: 'end' }): the old alignment sat above the
  in-flow composer, so every Enter hopped the page up by the
  composer+footer height and pushed the composer below the fold.
- The thinking window's pin state is now captured BEFORE the re-render
  (const pinned = block.open && isThinkingNearBottom(textEl)): the
  post-render distance read the new chunk's rendered height, not the
  user's position, so any chunk taller than the 32px band (real-model
  deltas, '\n\n' paragraph breaks) killed the follow at the first
  2-newline gap.
- Mock LLM: new 'think in paragraphs' trigger (scratchpad with real
  blank-line breaks, 60-char frames) — the 12-char mock frames never
  rendered past the band, which is why the bug survived the E2E gates.
- E2E (both verified red against the old code):
  test_submit_does_not_hop_up, test_thinking_window_follows_across_paragraph_breaks.
- Unit source-marker tests updated to the new contracts.
2026-08-28 17:10:02 -04:00
ducoterra 03d26255c6 feat(sources): upload tarball/zipfile archives as sources — unpack, scan, and replace in place
Phase 49 (owner request, chat 2026-08-28: "The git sources page should
remove local directory and should instead accept a tarball or zipfile
upload which it will unpack and scan … reuploading the same tarball
should not create a new folder, but should unpack and overwrite the
previously unpacked content" — design confirmed in the same
conversation):

* POST /api/git-sources/upload (admin-only, require_admin): accepts
  .tar/.tar.gz/.tgz/.zip, streams it with the BOR_UPLOAD_MAX_MB cap
  (bounds BOTH the compressed upload and the total extracted bytes —
  zip-bomb guard), safely unpacks (absolute/traversal/symlink/hardlink
  escape and device/FIFO members rejected), and atomically swaps the
  content in over BOR_UPLOAD_DIR/<name>/ (name = filename minus the
  archive suffix — no missing window, a failed upload never touches the
  existing folder/row/KB). The git_sources row is upserted by path
  (kind='local', no duplicates, added_at preserved), the models are
  checked fail-fast (503 sanitized when down — the folder/row stay
  committed and the next sync/re-upload retries idempotently), and the
  source is scanned synchronously in the request (single-source
  import_sources prune=True + change-gated KB overview), answering 200
  with the sync-style counts. One upload at a time (409); the request
  session is released before the scan so a concurrent TRUNCATE cannot
  deadlock against it.
* app/rag/archive_upload.py: ArchiveUploadError, ARCHIVE_SUFFIXES,
  archive_source_name (safe-name derivation), unpack_archive (guarded
  zip/tar extraction with the extracted-byte cap, no partial state),
  swap_in (atomic replace with restore-on-failure) — fully unit-tested.
* app/config.py + .env.example: BOR_UPLOAD_DIR (default
  ~/bor-sources/uploads, deliberately separate from the git checkouts)
  and BOR_UPLOAD_MAX_MB (default 512; a validator fails loud at
  startup on <= 0).
* python-multipart added to the dependencies — FastAPI's required
  multipart parser (an A2 implementation detail, phase locked decision).
* The Sources page: the phase-38 "Add a local directory" form is
  removed; #archive-upload-form takes its place (labeled file input,
  "Upload & scan" button, the §7.4 never-stale lifecycle, inline
  role=alert error, role=status count line); hint + table caption
  updated. The POST /api/git-sources kind=local API contract is
  UNCHANGED — a plain directory is still registrable via the API, and
  existing Local rows list/remove/sync exactly as before.
* The phase-38 story E2E (test_local_directory_sources.py) is rewritten
  API-driven — the form it drove is gone; its acceptance stands.
* The story E2E (test_archive_upload_sources.py): the swap,
  upload→scan→list (the deterministic "Uploading…" in-flight state, the
  Local row, /api/docs + the RAG catalog), same-filename re-upload
  (in-place replace, prune, no duplicate row, v2-only folder), the
  422 inline error + recovery (the form is not wedged), and the
  anonymous gate + 403.
* README: the archive-upload section (formats, naming rule, in-place
  replace, both new settings), the local-directory form removal noted,
  config reference rows for BOR_UPLOAD_DIR / BOR_UPLOAD_MAX_MB.

Gates: unit+integration green, app/ coverage 99%, the story E2E green
in isolation, the regression suites (git sources admin, local
directory sources, sync button, import documents, nav rename, smoke,
shared header) green in isolation, ruff + pyright clean.

Note: per this phase's file-level staging, frontend/assets/styles.css
also carries the small same-day in-flight owner rework already in the
working tree (the .sign-in-mobile companion rule for the phase-48
mobile sign-in copy); the phase-49 change is the upload form's block.
2026-08-28 15:57:59 -04:00
ducoterra 872a07cee7 feat(ui): rename nav items — "Sources" becomes "RAG", "Git sources" becomes "Sources"
Owner request (2026-08-28): the two admin-only nav items read like the
same thing, so they are relabeled — the document-catalog link
(#nav-sources, /sources.html) becomes "RAG" and the source-manager link
(#nav-git-sources, /git-sources.html) becomes "Sources".

Phase 48 (48_nav_rename_sources), label-only per the locked decision:
- all six pages (index, sources, git-sources, tuning, document, login):
  the two <a> texts swap; ids, hrefs, hidden defaults, is-active /
  aria-current placement, and nav order (Chat, RAG, Sources, Tuning)
  are byte-unchanged otherwise.
- header.js: comment/docstring label mentions only — the reveal-by-id
  logic is untouched (ship-hidden/reveal contract intact).
- test_git_sources_admin.py / test_mobile_hamburger_nav.py: the two
  suites that asserted the old label text are updated; comment-only
  label fixes in test_shared_header.py / test_nav_consistency.py.
- tests/e2e/test_nav_rename_sources.py: the story E2E (green in
  isolation) — renamed labels + unchanged hrefs/order/markers on all
  six pages, click navigation with the active marker, the anonymous
  ship-hidden contract, and regression guards for the untouched
  controls (#sync-label "Sync sources", viewer #doc-back "Sources").
- All eight surrounding header/nav suites stay green in isolation;
  unit+integration green, app/ coverage 99% (frontend-only change),
  ruff + pyright clean.

Note: per this phase file-level staging, the six page files and
header.js also carry the same-day in-flight owner rework that was
already in the working tree when phase 48 ran (mobile sign-in dropdown
copy, sync button ship-hidden on the Sources page); the label rename
itself is the two-text swap on each page.
2026-08-28 12:33:18 -04:00
ducoterra 03bead092c various fixes 2026-08-28 09:42:19 -04:00
ducoterra 5d679f5184 feat(import): index quadlet unit files and jinja templates (A9 revision)
Phase 47 (owner permission 2026-08-27, TODO.md L10–11, roadmap R1): the
full Podman quadlet family (.container, .network, .volume, .image,
.pod, .kube, .swap, .os, .endpoint) and .j2 Jinja templates join the
allowed + default A9 import formats, chunked as plain text (owner
decision — no TOML/Jinja-aware splitter). No env configuration needed:
a default import now indexes them.

- app/config.py: _ALLOWED_IMPORT_EXTENSIONS + the default
  import_extensions CSV gain the ten names (the original seven first);
  the never-widen BOR_IMPORT_EXTENSIONS validator is untouched and
  still rejects truly unknown extensions.
- app/rag/chunker.py: ten _FORMAT_CHUNKERS entries -> chunk_text
  (HARD_MAX_CHARS 1200 honored, unknown-suffix fallback unchanged);
  docstring/comments cite the A9 revision 2026-08-27.
- tests/fixtures/docs/homelab/: quadlet/compose.container (realistic
  quadlet TOML, >1500 chars, [Unit]/[Service]/[Container] sections,
  RESE-QUADLET-SENTINEL-77aa), quadlet/lan.network,
  quadlet/cache.volume, templates/deploy.j2 (for/set/if Jinja
  constructs + RESE-JINJA-SENTINEL-33dd). Every suite that seeds the
  fixture tree updates its 9 -> 13 document-count constants.
- tests/unit/test_config.py: allowed set carries all seventeen formats,
  default CSV + dotted import_extension_set include the ten, the
  validator accepts the new names and still rejects unknowns.
- tests/unit/test_chunker.py: dispatch parity with chunk_text for every
  new suffix (parametrized), the .container fixture chunks >=2 under
  the cap with the sentinel surviving, the .j2 fixture keeps {{ }}
  verbatim, the unknown-suffix fallback is unchanged.
- tests/unit/test_importer.py: a default-extensions walk over a temp
  tree indexes exactly the ten new files (unknown/hidden/excluded
  filtered), the original seven still walk, stem-title fallback holds.
- tests/integration/test_import_quadlet_jinja.py (new): import_sources
  over a temp tree with .container/.volume/.j2 -> documents + chunks
  rows with stem titles; delta re-import updates only the changed .j2
  doc; prune drops the deleted .volume doc with cascade.
- tests/e2e/test_quadlet_jinja_import.py (new, story suite, mock-only,
  isolation): GET /api/docs (admin session) lists the four new-format
  docs with non-zero chunk counts and stem titles; the Sources table
  renders a row + .doc-link per file; the phase-26 modal shows the
  .container TOML ([Container] section + sentinel) with stem title and
  the container format badge; a RESE-JINJA-SENTINEL-33dd question
  FTS-matches the .j2 chunk -> honest-positive (A8: LOW requires zero
  FTS hits) — the bubble is not .is-deflected and a source chip names
  templates/deploy.j2.
- README.md + .env.example: the extended default format set (A9
  revised 2026-08-27, plain-text chunking, narrow-only rule intact).
- .agent/PLAN.md: the A9 revision (owner-locked R1) — A9 row status,
  the revision note under the anchors table, and the §5 chunking-policy
  + §11 workflow lines. The only PLAN edit this phase.

Gates: uv run pytest 795 passed; app/ coverage TOTAL 99% (>90%);
ruff check + pyright clean; story E2E 4/4 in isolation (DB up);
regression E2E suites test_import_documents (3) / test_sync_button
(3) / test_git_sources_admin (6) green in isolation.

Also records the 47_quadlet_jinja_import task-file moves (01–03)
todo/ -> complete/.
2026-08-28 07:02:24 -04:00
ducoterra 6be692d999 feat(header): hamburger dropdown nav on mobile (owner permission)
TODO.md L9 (owner permission 2026-08-27, roadmap A5): "The navbar on
mobile is way too squished. Make it a hamburger dropdown menu with a
nice animation." At <=640px the nav links leave the bar — a 44px
#nav-toggle opens #app-nav as an animated (180ms slide+fade)
edge-to-edge dropdown with comfortable rows and the auth visibility
contract intact inside the menu; at >640px the bar is byte-identical
to pre-phase-46 (hamburger absent, inline pills as before).

- frontend/*.html (all six pages): the shared bar gains the
  #nav-toggle button (type=button, aria-expanded=false,
  aria-controls="app-nav", aria-label="Menu", aria-hidden 3-line
  SVG icon) immediately before the nav, and the nav gains
  id="app-nav" — one <nav>, no duplicated links, so the whoami reveal
  works inside the menu unchanged (phase-34 same-bar contract intact).
- frontend/assets/styles.css: .nav-toggle is display:none outside media
  queries (desktop untouched); the <=640px block adds the 44px toggle
  (+hover in the .steering-toggle:hover family, sized 20px icon), turns
  .app-nav into the dropdown (absolute top:100% edge-to-edge under the
  sticky header, surface + hairline + --shadow-lg, z-index 21 =
  header+1, closed state invisible + non-interactive with the 180ms
  opacity/transform/visibility-delayed pair, .is-open the only
  opener), and comfortable 1rem/0.75rem menu rows — superseding the
  phase-34/35 pill-squeeze rules for .nav-link/.app-nav (the 900px
  tablet block, action pills, and 58px bar height untouched). The
  reduced-motion block stills BOTH the closed and .is-open states: the
  .is-open rule (0,2,0) out-specifies a bare .app-nav (0,1,0), so the
  override must name both — verified live in Chromium (task 03).
- frontend/assets/header.js: ONE module-owned binding (import-time,
  null-safe like the sign-out binding): click toggles .is-open +
  aria-expanded in sync, a delegated nav-link click closes, Esc closes
  and refocuses the toggle, and matchMedia("(max-width: 640px)")
  change drops the state on resize back to desktop. The binding
  touches only the container — ship-hidden whoami links stay hidden.
- tests/unit/test_hamburger_nav.py (new): the markup/CSS/JS contract
  pins (six identical toggles in the shared row, desktop byte-
  identical, dropdown + .is-open + 180ms + reduced-motion rules, the
  superseded squeeze rules gone, the one-binding behavior).
- tests/e2e/test_shared_header.py: assert_shared_bar gains mobile=True
  (at <=640px the bar shows the hamburger + the closed nav; the
  per-role menu contents are pinned by the story suite).
- tests/e2e/test_mobile_hamburger_nav.py (new, story suite, 375x812):
  toggle is a visible >=44px target, menu closed (opacity 0 /
  visibility hidden), no horizontal overflow; anonymous menu shows
  exactly "Chat" (admin-only links stay hidden inside); admin menu
  shows all four links (whoami reveal inside the menu); a link click
  navigates + the arrival page ships closed; Esc closes and refocuses
  the toggle (outside click does NOT close — accepted: the locked
  close set is Esc + link + resize, no backdrop); the 180ms
  opacity/transform pair is live and reducedMotion:reduce stills both
  states with open/close still working; 1280x800 regression — toggle
  display:none, all four inline links inside the header band.

Gates: unit+integration 773 passed; app/ coverage TOTAL 99%
(unchanged — frontend-only phase); story E2E 7 passed in isolation
(mock LLM, DB up); regression suites test_nav_consistency (6) /
test_header_consistency (3) / test_shared_header (6) /
test_responsive_polish (7) / test_tuning_nav_link (4) all pass in
isolation; ruff check + pyright clean. A11 honored: no CDN, no new
assets.

Also records the 46_mobile_hamburger_nav todo/ -> complete/ move.
2026-08-28 06:07:02 -04:00
ducoterra b855d0aef9 feat(rag): unbounded agent tool calls behind a round cap (owner revision)
Phase 45 (owner permission 2026-08-27, TODO.md L8: "allow the LLM
to make as many tool calls as it wants"): the phase-37 per-turn tool
budgets (BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS, default 1 each)
and their exhaustion refusals are removed — a grounded turn now offers
list_documents / read_document for the whole turn (re-lists included),
bounded only by the round cap:

- app/config.py: agent_max_rounds (BOR_AGENT_MAX_ROUNDS, default 10,
  negative rejected) replaces agent_list_calls / agent_read_calls;
  .env.example + README document the single knob; app/rag/prompts.py
  docstrings follow.
- app/rag/agent.py: the loop runs tools until the model answers or
  rounds >= max_rounds, at which point it forces one final no-tools
  answer (the cap is the only forced exit); 0 = no tools — exactly one
  tools=None request, byte-identical to the pre-phase-37 path (the
  kill switch). Rejected calls (unknown tool / missing args /
  already-in-context / unknown path) still consume a round, so
  pathological rejected-call streams are bounded by the cap. The
  per-call log line is now tool/args/round=N/M; the per-turn
  tool_calls=N field and the tool SSE event are unchanged.
- tests/e2e/mock_llm.py: MULTI_READ_TRIGGER ("read two documents") —
  the deterministic list -> read #1 -> read #2 -> forced-answer flow
  (byte-stable "I read <sp1> and <sp2>." line), classified by the
  count of tool-role read results; the phase-37 single-read flow stays
  byte-identical (unit-pinned in tests/unit/test_mock_tool_flow.py).
- tests/e2e/test_agent_unlimited_tools.py (new, story suite,
  mock-only): three tool frames/lines in order (one list, two reads —
  the second read is what the old read budget refused) + the
  both-named non-deflected answer; done.sources + chips = retrieval
  doc + both reads, deduped; no budget refusal rendered; the
  single-read marker flow regression (exactly one read, single tool
  pair).
- .agent/PLAN.md: the phase-45 SSE revision note (owner-locked, R2) —
  the only PLAN edit this phase; the phase-37 note's budget clause is
  marked removed.

Unit/integration rewrites (test_agent.py round-cap matrix incl. the
kill switch and rejected-call spam, test_config.py, test_chat_api.py
agent_max_rounds=0 fixtures) landed with the server core so every gate
stays green.

uv run pytest: 756 passed, app/ coverage 99%; ruff + pyright clean;
story E2E 4/4 in isolation (ran twice); regression E2E suites
(agent_document_tools unmodified, chat_rag, smoke) green in isolation.

Also records the 45_agent_unlimited_tools todo/ -> complete/ task-file
moves (00/01/02 pending in the working tree, task 03 moves on success).
2026-08-28 04:50:56 -04:00
ducoterra bc70ce36e0 feat(chat): render markdown tables in answers, viewer, and thinking
GFM pipe tables in the shared renderer (TODO.md L6): a table-protection
pass in frontend/assets/markdown.js (fences -> tables -> escape order)
pulls each header+separator+body block out as a placeholder, renders
cells escape-first with the same inline transforms, and reinserts a
semantic <table class="md-table"> inside a horizontal-overflow
.md-table-wrap — so a pipe table in a chat answer, the document
viewer/modal, and the thinking block all render the same semantic
table. Fences win over tables; lone pipes stay text.

- styles.css: .md-table palette rules (PLAN §7.2 tokens, no motion);
  min-width: max-content so a WIDE table keeps its natural width and
  the wrapper is the real scroller (width:100% alone wrapped the wide
  table's cells — proven by the new E2E).
- mock_llm.py: TABLE_TRIGGER ("show me a table") -> byte-stable
  TABLE_ANSWER (3-column table, <img onerror> XSS probe line, wide
  5-column table), checked before DEFLECT_MODE like SUMMARY_MODE.
- tests/fixtures/docs/homelab/tables.md: 3x3 pipe table + pipe-heavy
  fenced block (viewer/fence subject); the shared fixture set grows
  8 -> 9 docs, so every suite pinning the count (added/formats/
  stat-docs/EXPECTED_ROWS) is updated accordingly.
- tests/e2e/test_markdown_tables.py (new, story suite): chat table
  shape + non-deflection, wide-table wrapper scroll (no page
  overflow), XSS probe inert, viewer modal table, fence-not-a-table,
  lone pipe stays text.
- tests/e2e/test_agent_document_tools.py: fix a pre-existing flake —
  the "Calling tool…" label window is ~0.4 s at the mock's 0.1 s
  tool-frame pacing, and a polling expect could stride over it
  (failed 3 of 5 runs on the committed baseline). The pre-submit
  MutationObserver record is the deterministic source of truth; the
  racy to_have_text gate is gone.

uv run pytest: 738 passed, app/ coverage 99% (TOTAL unchanged);
ruff + pyright clean; story E2E 6/6 in isolation; regression E2E
suites (chat_rag, document_viewer, document_summaries, smoke) green.
2026-08-28 03:35:50 -04:00
ducoterra 27b7cb96d5 feat(chat): thinking window scrolls again, follows the tail only while pinned
TODO.md L7: "Add scrolling back to the thinking block, but have it
autoscroll while thinking content is generating." Owner direction
2026-08-27 (roadmap A2) reverses the phase-21 no-scroll choice
(2026-08-24): details.thinking .thinking-text is user-scrollable again
(overflow-y: auto — the 320px clip stays, owner-confirmed), and the
phase-17 per-chunk bottom-pin is GATED: the window follows the live
tail only while the user is pinned near its bottom (THINKING_NEAR_
BOTTOM_PX = 32); scrolling up pauses the follow, returning to the
bottom re-arms it on the next chunk (the gate re-runs on every frame).

- frontend/assets/styles.css: .thinking-text overflow-y: hidden ->
  auto; the phase-21 owner-choice comment is replaced with the
  2026-08-27 direction; max-height: 320px and every other declaration
  in the rule byte-identical.
- frontend/assets/app.js: export const THINKING_NEAR_BOTTOM_PX = 32 +
  isThinkingNearBottom(textEl) (scrollHeight - scrollTop -
  clientHeight <= band); the thinking-handler pin becomes
  `if (block.open && isThinkingNearBottom(textEl))` — a scrolled-up
  reader is never re-pinned and a closed (restored) block is never
  pinned; everything else in the handler (and phase 42's no page
  scroll) untouched.
- tests/unit/test_thinking_scroll.py (new, replaces the deleted
  tests/unit/test_thinking_no_scroll.py): pins the CSS contract (auto
  + 320px + owner-direction comment, no hidden/scroll left), the
  exported 32px band, the gate math, the gated pin (no unconditional
  `if (block.open)` remains), and the surviving collapsed-restore pin.
- tests/e2e/test_thinking_scroll.py (new, mock-only, replaces the
  deleted tests/e2e/test_thinking_no_scroll.py — its pins asserted the
  reversed phase-21 behavior, so both phase-21 files are deleted in
  this commit): user scroll restored on the frozen 4s-hesitation tail
  (wheel is 1:1; click+Home keyboard — the plain div is not
  keyboard-focusable by design, tabindex is test scaffolding; the
  literal drag holds the user's position — headless Chromium's
  overlay scrollbars are not grabbable by synthetic mouse events,
  documented in the suite), follow-while-pinned at the 2nd-to-last
  and last chunk (±1px) with the last chunk's text inside the visible
  rectangle, no re-pin over ≥5 mid-stream chunks after a
  half-window scroll-up, re-pin on the next chunk after returning to
  the bottom, the CSS contract, plus the phase-11 (long answer: page
  scrolls, bubble overflow untouched) and phase-17 (restored
  collapsed block with full text) regressions.
- tests/unit/test_chat_persistence.py: the CSS pin flips with the
  contract (auto in, hidden out — owner direction 2026-08-27).
- tests/unit/test_frontend_scroll.py: the "page-level band constant is
  gone" pin now excludes the phase-43 window-level
  THINKING_NEAR_BOTTOM_PX (a different band — the window's, not the
  page's).

E2E (isolation): test_thinking_scroll 7/7 (twice); regressions
test_thinking_display 5/5, test_chat_persistence 4/4,
test_no_reply_autoscroll 5/5, test_smoke 3/3; unit+integration 725
passed, app/ coverage 99% (unchanged — frontend-only phase);
ruff + pyright clean.
2026-08-28 01:36:32 -04:00
ducoterra 7c6763319b fix(chat): stop autoscrolling while a reply streams (owner direction)
TODO.md L5: "Get rid of the chat reply autoscroll, it's breaking things
like making it impossible for the user to scroll while a reply
generates." Owner direction 2026-08-27 (roadmap A1) revises the
phase-18 follow-the-bottom choice: the page NEVER auto-scrolls while a
turn streams. Kept (owner decision): the submit reveal (the user's own
message) and the one-shot phase-14 restore landing.

- frontend/assets/app.js: delete NEAR_BOTTOM_PX + isNearBottom;
  scrollReveal becomes the one unconditional scrollIntoView (still
  smooth, still "auto" under prefers-reduced-motion via SCROLL);
  addMessage(who, html, scroll = false) carries an explicit scroll
  intent — only the submit (", true") and the two restore landings
  scroll. The thinking/tool/delta handlers and the typing indicator
  drop their page-scroll calls; the thinking block's INTERNAL
  bottom-pin (textEl.scrollTop, phase 17 — reworked separately in
  phase 43) and the turn-end focus({ preventScroll: true }) survive.
- tests/unit/test_frontend_scroll.py: rewritten pin for the new
  contract — phase-18 gate absent, helper unconditional, explicit
  intent at submit/restore, no page-scroll call in the streaming
  handlers, typing bubble scroll-free, SCROLL reduced-motion intact.
- tests/unit/test_chat_persistence.py: restore-landing pin updated to
  the new signature (the old forced "auto" is gone; the landing
  rides the default SCROLL — noted at the call site).
- tests/e2e/test_no_reply_autoscroll.py (new, replaces the deleted
  test_follow_bottom_scroll.py): no autoscroll across >=10 samples
  (1px tolerance) during a long answer and during the thinking stream;
  submit-from-the-top still reveals the user message; the restore
  landing lands one-shot on the latest message and stays; long answer
  + sources and the collapsed thinking block persist and restore.

E2E (isolation): test_no_reply_autoscroll.py 5/5; regressions
test_chat_rag 3/3, test_thinking_display 5/5,
test_chat_persistence 4/4, test_long_answers 2/2, test_smoke 3/3;
unit+integration 723 passed, app/ coverage 99%; ruff + pyright clean.
2026-08-28 00:29:25 -04:00
ducoterra 6cf1df9bf2 feat(sync): fail fast with a modal when a model is unavailable
TODO.md L4: with a dead model endpoint the sync discovered it only
mid-import, after slow clones — and a tooltip on the button is not a
readable error.

- app/rag/llm.py: ModelUnavailableError + check_models(llm) — a tiny
  pre-sync probe (one short embedding + one 1-token-scale completion)
  that fails naming the unavailable model (embed first, then the
  summary model); the sync sanitizer still masks credentials.
- app/api/sync.py: the probe is step 1 of _run_sync — before source
  resolution and before any clone_or_pull; a model failure is just
  another 'failed' state (no new endpoint, A10/A12 untouched).
- frontend/assets/header.js: applySyncFailure now also opens the
  module-owned error modal (every page carrying #sync-btn, zero
  page-markup changes): lazily built backdrop + role=alertdialog
  panel, error text via textContent, close via button / Esc /
  backdrop, focus in-and-out to #sync-btn (with a body→#sync-btn
  fallback — the run's disabled button drops focus to <body>).
- frontend/assets/styles.css: the modal on the phase-08 error palette
  (z-index above the header, .is-open open/close, reduced-motion
  stilling, 44px close target).
- Tests: probe unit tests (both up / embed down / summary down /
  custom model names), sync integration (fail-fast before any clone,
  probe-before-effective_sources ordering, credential masking,
  healthy regression), the phase-41 source pins, and the story E2E
  (two module apps on distinct ports — dead endpoint on a closed
  loopback port vs session mock: ≤10 s fail-fast + modal contract,
  all three dismissal paths with focus out to #sync-btn, button
  title/.is-error + Sources banner untouched, healthy phase-32
  lifecycle regression to 'Synced HH:MM').

E2E (isolation): test_sync_model_down.py 4/4, test_sync_button.py
3/3, test_git_sources_admin.py 6/6, test_local_directory_sources.py
3/3; unit+integration 721 passed, app/ coverage 99%; ruff + pyright
clean.
2026-08-27 23:44:35 -04:00
ducoterra 6f9e033117 fix(header): ship the tuning toggle hidden — no anonymous flash
#steering-toggle (the header 'Tuning' button) shipped visible in all
six pages and was only removed after /api/whoami resolved, so
anonymous visitors saw it flash for the whole round-trip (TODO.md L3).
It now ships hidden on every page and initSharedHeader unhides it only
for admin — the same ship-hidden / reveal-for-admin contract as the
admin-only nav links; the anonymous end-state (removed from the DOM,
phase-16 'absent, not hidden') is unchanged.

Adds the story E2E suite (MutationObserver proves zero visible frames
for anonymous on every page, admin reveal + panel + count badge,
nav-contract regression) and the source-level unit pins. Also fixes
test_steering.py's BASE_SCRIPT_COUNT (2 → 3: brand.js + markdown.js +
app.js, since phase 39).
2026-08-27 22:38:58 -04:00
ducoterra 02c76ad328 chore(agent): phase roadmap from TODO.md — 8 phases (40–47), 24 tasks
Converts the 9 TODO items into an executable phase roadmap (Protocol B,
appended after phase 39):

- 40 tuning toggle anonymous flash (TODO L3)
- 41 sync fail-fast + modal when a model is down (TODO L4)
- 42 no reply autoscroll (TODO L5)
- 43 thinking scroll back — user scroll + gated autoscroll (TODO L7)
- 44 markdown tables (TODO L6)
- 45 agent unlimited tool calls behind BOR_AGENT_MAX_ROUNDS (TODO L8)
- 46 mobile hamburger nav (TODO L9)
- 47 quadlet + jinja import formats, A9 revision (TODO L10–L11)

Each phase carries a user story, a dedicated Playwright E2E suite plan,
and owner-locked decisions (R1 A9 format extension, R2 phase-37 budget
revision, A1–A5 scope decisions) confirmed 2026-08-27.

Also records the completed phases 30–39 todo/ -> complete/ moves that
were pending in the working tree. TODO.md is cleared (items now live in
.agent/phases/todo/).
2026-08-27 18:25:53 -04:00
ducoterra 492d8275e7 add git to containerfile
Build and Push Containers / build-and-push-app (push) Successful in 2m17s
Build and Push Containers / build-and-push-db (push) Successful in 15s
2026-08-27 10:32:12 -04:00
ducoterra beaca4fe97 add db build and push
Build and Push Containers / build-and-push-app (push) Successful in 16s
Build and Push Containers / build-and-push-db (push) Successful in 10s
2026-08-27 10:00:21 -04:00
2689 changed files with 274315 additions and 13824 deletions
-499
View File
@@ -1,499 +0,0 @@
# Brain of Reese — Master Plan
> **Status:** Phase 1–3 complete (scaffolded, designed, decomposed).
> **Rule:** Every agent reads this file first. Decisions marked `LOCKED` in the
> Anchors table are settled — do not re-litigate them in a phase.
> **Revisions (2026-08-21, owner permission):** A7/A8/A9 revised (multi-format
> ingestion, hybrid FTS+vector retrieval, re-tuned honesty gate); dark tech
> theme (Phase 08); clickable document viewer (Phase 10); thinking display
> (Phase 17, owner permission 2026-08-23); follow-the-bottom scroll
> (Phase 18, owner choice 2026-08-23); shared header (Phase 19);
> whole-document context (Phase 24 — A7's 24k context cap removed,
> owner permission 2026-08-24). See roadmap §12.
---
## 1. Mission
A **knowledge base chatbot** that embeds the `~/Homelab` and `~/Deployments`
projects into a Postgres vector database and lets anyone ask *Reese* (the
bot) questions about them.
**Product feel:** a chippy, upbeat assistant that is optimistic about the
user's ability ("you've got this") and **radically honest** — if retrieval
didn't surface anything relevant it says *"I haven't done anything like
that"* and offers alternatives instead of hallucinating.
### In scope (v1)
- Chat UI (mobile-friendly, well-styled, no auth, no CDN).
- RAG over text knowledge files — `md, markdown, txt, yaml, yml, json, py`
by default (A9, revised 2026-08-21) — from `~/Homelab` + `~/Deployments`
(and any future directory the importer is pointed at).
- Self-hosted models via `https://aipi.reeseapps.com/v1` — `turbo` (chat),
`embed` (embeddings, **768 dims — verified**).
- Postgres 17 + pgvector, cosine similarity, chunk→document mapping so the
LLM receives the **entire relevant document** as context.
- Idempotent import/update script, documented in the README.
- Ample server logging + explicit UI loading/progress feedback (never a
stale submit button).
### Out of scope (v1)
- Auth / multi-user (API is stateless under `/api` so it can be added later).
- Binary / non-text content, file uploads, caching layer, message persistence.
- Real-time document watching (manual re-import for now).
---
## 2. Architectural Anchors (LOCKED DECISIONS)
| # | Component | Decision | Rationale | Status |
|---|-----------|----------|-----------|--------|
| A1 | Runtime | Python 3.12+, `uv` for all package management | Fast, reproducible envs; one language for API + tooling | LOCKED |
| A2 | Web framework | FastAPI + Pydantic v2 + Uvicorn | Async, typed, SSE-friendly for LLM streaming, free OpenAPI docs | LOCKED |
| A3 | Database | **PostgreSQL 17** (`docker.io/postgres:17`, pgvector compiled in via `db/Containerfile`) with **cosine** (`<=>`) search | One system for relational + vectors; pgvector is mature; official base image kept per project standard | LOCKED |
| A4 | Orchestration | `compose.yaml`, started with **`podman compose up -d`** | Matches Reese's toolchain | LOCKED |
| A5 | LLM backend | OpenAI-compatible `https://aipi.reeseapps.com/v1`; models **`turbo`** (chat) & **`embed`** (embeddings); `openai` async client | Self-hosted, offline from cloud; no new model management | LOCKED |
| A6 | Embedding dim | **768** (verified 2026-08-21 against live endpoint via `scripts/llm_probe.py`); configured by `BOR_EMBEDDING_DIM` | User recalled 768 — probe confirmed; dimension is fixed at table creation, so mismatch must fail loudly at import time | LOCKED |
| A7 | Retrieval→context | **Hybrid:** cosine top-30 + Postgres FTS top-30 (OR tsquery, `ts_rank`) fused with **RRF (k=60)** → map to parent documents ranked by best fused chunk score → feed the **full text of top-N=2 documents** (deduped) to the LLM | Owner permission 2026-08-21: pure-cosine top-4 missed real docs (gitlab case — best chunk ranked 7th behind vendored-cache junk; score compression 0.41–0.84); the lexical signal finds name-your-tool questions; whole-document context contract preserved. A7 revised 2026-08-24 — matched documents never truncated (owner: "this should never happen"; emergency-valve variant rejected) | LOCKED (revised 2026-08-24) |
| A8 | Honesty gate | **Deflection mode** (LLM must open with a variant of *"I haven't done anything like that"* and offer 2–3 alternative questions) when best cosine < `BOR_RELEVANCE_THRESHOLD` **and** no candidate chunk FTS-matches the question; threshold re-tuned for the `embed` model's compressed score range (default **0.62**, calibrated via `scripts/eval_retrieval.py`; the E2E mock uses its own 0.30 calibration via the app fixture) | Owner permission 2026-08-21: at 0.30 the gate never discriminated (measured corpus range 0.41–0.84); the FTS-OR keeps name-your-tool questions honest-positive; deflection product behavior unchanged | LOCKED (revised 2026-08-21) |
| A9 | Content scope | Text formats **`md, markdown, txt, yaml, yml, json, py`** (default, `BOR_IMPORT_EXTENSIONS`), **hidden (dot) directories skipped by default**, plus the exclusion list (`node_modules`, `__pycache__`, `.pytest_cache`, `dist`, `build`, …) | Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (`.esphome/.espressif/**`, …) that outranked real content | LOCKED (revised 2026-08-21) |
| A10 | Auth | **None in v1**; all endpoints stateless under `/api` | Per user (auth later); statelessness keeps the future migration cheap | LOCKED |
| A11 | Frontend | Vanilla HTML/CSS/JS in git; **no CDN** — everything served by FastAPI `StaticFiles`; minified by esbuild in the `Containerfile` build stage; system font stack | No external deps at runtime; tiny, auditable surface; mobile-friendly by construction | LOCKED |
| A12 | Aux services | **None in v1** (no Valkey, no SeaweedFS) | No sessions/auth (no store), no uploads (no object storage); add later only if a need appears | LOCKED |
| A13 | Migrations | Alembic + SQLAlchemy 2.0 (sync) + psycopg 3 | Standard, reversible, reviewable schema history | LOCKED |
| A14 | Debugging | `debugpy` **only when `DEBUGPY=1`** (env var read directly, not via settings); listen `0.0.0.0:5678` (override `DEBUGPY_PORT`), non-blocking, attach-on-demand; **not imported at all when off** | Zero overhead by default per project standard; attach-on-demand keeps production runs clean | LOCKED |
| A15 | Chat transport | **SSE streaming** from `POST /api/chat` (deltas + final `done` event with metadata) | Local LLM latency is 10–30s; live token stream + explicit completion event power the UI's feedback states | LOCKED |
| A16 | Testing | Per phase: unit + integration (pytest, **coverage >90%** on `app/`) + **one dedicated Playwright E2E file per user story**, run in isolation; E2E uses a deterministic mock LLM by default (`E2E_REAL_LLM=1` opts into live aipi) | One story, one phase, one E2E gate — the pipeline's core invariant | LOCKED |
| A17 | Git | Conventional Commits, **always `--no-gpg-sign`**, repo-local `commit.gpgsign=false`; one atomic commit per completed phase | Subsequent agents may lack the GPG key | LOCKED |
> **A10 revision (phase 16, owner permission 2026-08-22):** single-admin
> signed-cookie auth — public: chat / documents / suggestions / health;
> admin-only: docs catalog + steering. The row above keeps the original v1
> decision text; the public API surface stays stateless (the signed
> session cookie is the only session state) — recorded as a revision,
> not a silent deviation.
>
> **A10 UI revision (phase 19, owner permission 2026-08-23):** the
> "Sources" nav link is hidden from anonymous users on all pages — the
> soft-gate page and the API split above are unchanged.
>
> **A7 revision (phase 24, owner permission 2026-08-24):** the
> `[…truncated…]` cap on document context is removed —
> `select_documents` always returns the full top-N texts;
> `BOR_MAX_CONTEXT_CHARS` is gone. The steering section
> (`BOR_STEERING_MAX_CHARS`, phase 15) keeps its budget and the shared
> marker.
---
## 3. High-Level Architecture
```
┌────────────────────────────────────────────┐
│ Podman Compose │
Browser │ ┌──────────────────────────────────────┐ │
┌──────────┐ HTTP │ │ brain-of-reese/app (FastAPI) │ │
│ index.html│◄──────┼─►│ • static frontend (no CDN) │ │
│ app.js │ SSE │ │ • /api/chat /api/suggestions │ │
└──────────┘ │ │ • /api/health /api/docs │ │
│ │ • RAG pipeline (embed→retrieve→gen) │ │
│ └──────┬──────────────────┬───────────┘ │
│ │ SQL (psycopg) │ OpenAI-compat│
│ ┌──────▼──────┐ ┌───────▼────────────┐ │
│ │ db: │ └─────────┬──────────┘ │
│ │ postgres:17 │ │ │
│ │ + pgvector │ │ │
│ └─────────────┘ │ │
└──────────────────────────────┼────────────┘
▼
https://aipi.reeseapps.com/v1
(self-hosted: turbo, embed)
Offline tooling (same repo, same venv):
scripts/import_docs.py → walks A9-format dirs, chunks, embeds, upserts
scripts/eval_retrieval.py → ranks hybrid results for a question (tuning)
scripts/llm_probe.py → verifies models + embedding dim
```
### Component breakdown
| Component | Responsibility | Lives in |
|-----------|----------------|----------|
| **App (FastAPI)** | Serves frontend + `/api`; RAG pipeline; logging | `app/` |
| **RAG pipeline** | `embed` → pgvector cosine top-K → doc mapping → context assembly → `turbo` (streamed) with persona/honesty prompt | `app/rag/` (added in story phases) |
| **Importer** | Directory walk (A9 formats, hidden dirs skipped, exclusions), sha256 delta detection, format-aware chunking, batched embedding, upsert/prune | `scripts/import_docs.py` (story phase) |
| **DB** | `documents`, `chunks`, `query_log` + `vector` extension | `db/` image, `alembic/` |
| **Frontend** | Chat shell, sources view, loading/feedback states | `frontend/` |
### Chat data flow
```
user question
→ POST /api/chat {message}
→ embed(question) [aipi /v1/embeddings, model=embed]
→ cosine top-30 + FTS top-30 (OR tsquery, ts_rank) [pgvector + PG FTS]
→ RRF fuse (k=60) → docs ranked by best fused chunk score
├─ best cosine >= 0.62 OR fts_hits > 0 → top-2 documents' FULL content
│ → system prompt (persona + HONESTY rules + docs)
│ → turbo, stream=True → SSE deltas
└─ else → DEFLECT_MODE system prompt (weak hits as topics)
→ turbo, stream=True → SSE deltas (honest reply)
→ query_log row (question, score, deflected, sources, latency)
→ final SSE "done" event: {deflected, sources[], suggestions[]}
```
---
## 4. API Design
All endpoints stateless (A10). Errors: standard JSON `{detail: str}`.
| Method | Path | Purpose | Story |
|--------|------|---------|-------|
| GET | `/api/health` | Liveness + db up/down + version | 01 |
| GET | `/api/suggestions` | Onboarding suggestion strings | 01 (05 refines) |
| GET | `/api/docs` | Indexed document list (source, path, title, chunks, indexed_at) | 02 |
| GET | `/api/documents/content?source=…&path=…` | One indexed document's full content (feeds the viewer page) | 10 |
| POST | `/api/chat` | RAG chat turn → **SSE stream** | 03/04 |
### SSE contract (`POST /api/chat`)
```
data: {"type":"thinking","text":"…"}\n\n
data: {"type":"thinking","text":"…"}\n\n
data: {"type":"delta","text":"Hey! "}\n\n
data: {"type":"delta","text":"Good "}\n\n
...
data: {"type":"done","deflected":false,"sources":[{"source":"Homelab","path":"kubernetes.md","title":"Kubernetes Homelab Cluster"}],"suggestions":[]}\n\n
```
Client rules: render deltas as they arrive; render `thinking` text in a
collapsible block above the answer; auto-collapse on the first `delta`;
tolerate interleaved `thinking` events (append — never reopen once the
answer started); the `done` shape is unchanged (thinking never travels on
`done`); on `done` append source chips / suggestion chips and clear the
busy state; on HTTP/stream error show the error banner + retry (never a
stuck button).
> **SSE revision (phase 17, owner permission 2026-08-23):** the contract
> gains one event type — `{"type":"thinking","text":"…"}` — carrying the
> model's reasoning ahead of the `delta` events (the `turbo` model emits
> `delta.reasoning_content` chunks before the first content chunk, verified
> live 2026-08-23; `BOR_STREAM_THINKING=0` suppresses the frames
> server-side). `delta` and `done` shapes are unchanged — a recorded
> extension of A15, not a silent deviation.
>
> **SSE revision (phase 37, owner permission 2026-08-26):** the contract
> gains a second event type — `{"type":"tool","name":"…","argument":…}` —
> carrying the model's document tool calls on grounded turns (phase 37:
> `list_documents` / `read_document`, budgeted by `BOR_AGENT_LIST_CALLS`
> / `BOR_AGENT_READ_CALLS`; `argument` is `"source/path"` for
> `read_document`, null otherwise). Client rule: render each `tool` frame
> as a "calling tool" line/state (task 05); `delta` and `done` shapes are
> unchanged — the read document is reflected in `done.sources` instead
> (deduped) — a recorded extension of A15, not a silent deviation.
---
## 5. Data Model (PostgreSQL 17)
Created by `alembic/versions/0001_initial_schema.py` (idempotent
`CREATE EXTENSION IF NOT EXISTS vector`).
### `documents`
| Column | Type | Notes |
|--------|------|-------|
| id | `UUID` PK | |
| source | `VARCHAR(120)` | source dir basename, e.g. `Homelab` |
| path | `VARCHAR(1000)` | relative to source dir, e.g. `ansible/roles/k3s.md` |
| full_path | `VARCHAR(2000)` | absolute path at import time (diagnostics) |
| title | `VARCHAR(500)` | first markdown H1, else file stem |
| content | `TEXT` | **full markdown — the RAG context** |
| content_hash | `VARCHAR(64)` | sha256 of content — change detection |
| indexed_at | `TIMESTAMPTZ` | |
| — | `UNIQUE (source, path)` | upsert key |
### `chunks`
| Column | Type | Notes |
|--------|------|-------|
| id | `UUID` PK | |
| document_id | `UUID` FK→documents CASCADE | **embedding→document mapping** |
| position | `INT` | 0-based order within the doc |
| content | `TEXT` | chunk text (heading-aware) |
| embedding | `VECTOR(768)` | nullable until embedded (two-phase import) |
| tsv | `TSVECTOR` | **generated** `to_tsvector('english', content) STORED` + GIN index (hybrid retrieval, A7) |
> No vector index in v1: sequential scan is fine at this corpus size
> (~100–500 docs). Revisit with an HNSW index if retrieval latency grows.
### `query_log`
`id UUID PK, question TEXT, top_score FLOAT, fts_hits INT, chunk_hits INT, deflected BOOL, sources TEXT, latency_ms INT, created_at TIMESTAMPTZ`
### Document state transitions
```
unseen ──import──▶ indexed ──hash changed + re-import──▶ reindexed
│
└──file deleted + --prune──▶ removed (chunks cascade)
```
### Chunking policy (markdown-aware)
Split on `## `/`### ` headings into sections; sub-split any section longer
than `BOR_CHUNK_TARGET_CHARS` (2000) at paragraph boundaries with
`BOR_CHUNK_OVERLAP_CHARS` (200) overlap; each chunk keeps its nearest
preceding heading in the text for retrieval quality.
**Format-aware (A9, revised):** `yaml`/`yml` split on top-level keys and
`---` separators (key line kept as anchor); `json` pretty-printed, split on
top-level keys; `py` split on top-level defs/classes (stdlib `ast`);
`txt` on paragraphs; markdown unchanged. Every format honors the 1200-char
hard cap (aipi ~1024-token request limit).
---
## 6. RAG Pipeline & Persona
### Locked system prompt (sent with every chat turn)
```
You are "Brain of Reese" — the digital brain of Reese, a self-hoster and
homelab tinkerer. Personality: chippy, upbeat, warm, and genuinely
optimistic about the user's ability to do things ("you've got this").
Rules:
1. Answer ONLY from the provided document context. Cite which document(s)
you used, by path.
2. Be concrete: names, versions, ports, hosts, schedules — the specifics in
the docs are the value.
3. HONESTY GATE: if <relevance> is "LOW", you must NOT pretend to know.
Start your answer with a variant of: "I haven't done anything like that."
Then offer 2-3 alternative questions about things you DO have notes on.
4. Never invent facts, hosts, or steps that are not in the context.
5. Keep answers tight: short paragraphs, bullets where helpful.
<relevance>{HIGH|LOW}</relevance>
```
- `HIGH` mode appends the full document text under `<documents>…</documents>`.
- `LOW` mode (deflection) appends only the **titles** of the weak hits so the
model can suggest real alternatives (marker used by the E2E mock:
`DEFLECT_MODE` appears in the system prompt).
### Retrieval (hybrid — A7/A8, revised 2026-08-21)
- Embed the question (`embed`, 768-d) → cosine top-30 candidates.
- Lexical: OR tsquery over the question's tokens → FTS top-30 by `ts_rank`.
- **Reciprocal Rank Fusion** (`Σ 1/(k+rank)`, k=60) → distinct parent docs
ranked by best chunk's fused score → top 2 → full content, concatenated
— **never truncated** (A7 revised, phase 24, owner permission
2026-08-24).
- Honesty gate: LOW only when `best cosine < BOR_RELEVANCE_THRESHOLD`
(default 0.62, calibrated against the `embed` model's measured 0.41–0.84
distribution) **and** zero FTS hits among the candidates.
---
## 7. UI/UX Strategy
### 7.1 Layout structure
- **App frame:** sticky header (64px) + `<main>` (flex-grow) + footer.
Container: `max-width: 72rem; margin-inline: auto; padding-inline: 1.25rem`.
- **Shared header (Phase 19, owner permission 2026-08-23):** the bar is a
shared contract across chat / sources / viewer — one bar per page, same
controls (brand + nav [Chat, Sources — admin only] + New Chat + Sign in
/ Sign out on chat & sources; the viewer bar = back + title + the same
actions in `.doc-header-actions`). One shared module
(`frontend/assets/header.js`) toggles the existing controls on each
page, so they can never "disappear" between pages again; the heights
stay pinned at 64px / 58px (phase 12).
- **Chat:** a *centered column capped at 46rem*. This is deliberate: chat is
a vertical conversation — a centered, capped column is the correct pattern
(NOT a layout bug). The 72rem frame + header/footer ensure the column
never reads as a hairline in a sea of whitespace.
- **Sources page:** full-width responsive **table** (min 640px, horizontal
scroll wrapper on small screens) + stat cards in
`grid-template-columns: repeat(auto-fit, minmax(170px, 1fr))`.
No skinny single-column lists anywhere: lists/tables/grids use ≥80–90% of
the container width.
- **Mobile (≤640px):** suggestion chips become a horizontally scrollable row;
composer stays reachable with `safe-area-inset-bottom`; touch targets ≥44px.
### 7.2 Accessibility (WCAG 2.1 AA)
- Semantic landmarks on every page: `<header>`, `<nav aria-label>`,
`<main>`, `<footer>`; skip-link to `#main`.
- Every control labeled: visible `<label>` or `aria-label` (icon-only
buttons always get `aria-label`); form input has a (visually-hidden) label.
- Live regions: message stream `aria-live="polite"`; typing indicator
`role="status"`; banner `role="status"`; errors `role="alert"`.
- **Dark tech theme (Phase 08, 2026-08-21)** — page `#0a0e17`, surface
`#121a2e`; ink `#e8ebf4` on surface ≈14.5:1; ink-soft `#9aa4bd` on
surface ≈6.9:1; **dark ink `#0a0e17` on brand `#6d78f2` ≈5.2:1** (white
on brand ≈3.7:1 — never used for text); brand-ink `#a5b4fc` on
brand-soft `#232b52` ≈6.9:1; deflection `#fbbf24` on `#2b2110` ≈9.5:1
(border `#f59e0b`); error `#fca5a5` on `#2d1318` ≈9.1:1. All computed,
all ≥4.5:1. `prefers-reduced-motion` also stills the Phase-08 background
layer.
- `:focus-visible` outline 3px; `prefers-reduced-motion` respected by the
typing/spinner animations.
### 7.3 No external dependencies
- System font stack only (no font files to bundle, no CDN fonts).
- Zero `<script src="https://…">` / `<link href="https://…">` — enforced by
an integration test (`tests/integration/test_api.py::test_index_html_served_locally`)
and re-checked by every UI phase's verification step.
- Markdown rendering is a ~60-line local function (escape-first, then
transform) — XSS-safe, no library.
### 7.4 Visual feedback standard (the "never stale" contract)
| State | UI |
|-------|----|
| **Idle** | Send button enabled, label "Send". |
| **Thinking (pre-token)** | 3-dot typing bubble + button disabled with spinner, label "Thinking…". |
| **Thinking (model reasoning)** | Collapsible `.thinking` block streams open (replaces the typing dots as the live indicator), auto-collapses on the first answer token, toggleable afterwards, persisted with the message (phase 14); 120s guard clears on the first `thinking` *or* `delta` event. |
| **Streaming** | Deltas append live into the brain bubble; button stays busy. |
| **Done (answer)** | Source chips under the bubble (mono, path-based); button re-enabled. |
| **Done (deflected)** | Amber-bordered bubble + "Maybe try:" suggestion chips. |
| **Error** | Red banner (`role="alert"`) with retry hint; button re-enabled. |
| **KB offline** | Amber banner at top of chat ("start Postgres…"); chat disabled with explanation. |
| **Guard** | 120s client-side timeout → error state (a button can never sit "stuck" forever). |
| **Scroll (follow-the-bottom, phase 18)** | The page auto-scrolls only while the user is pinned to the bottom (≤200px band, `NEAR_BOTTOM_PX` — the composer zone; submitting reveals the user's message through the same gate, which holds in real use); scrolling up holds the viewport for the rest of the turn (thinking and answer alike); restore lands one-shot on the latest message. |
> The **Thinking (model reasoning)** row is a phase-17 addition (owner
> permission 2026-08-23) — see the §4 SSE revision.
>
> The **Scroll** row is a phase-18 addition (owner choice 2026-08-23 —
> option 1: follow-the-bottom, no "↓ new content" pill).
### 7.5 Component inventory (ids used by tests)
`#messages` (stream), `#empty-state`, `#suggestions`, `.suggestion-chip`,
`#composer`, `#message-input`, `#send-btn` / `#send-label`, `#typing-indicator`,
`.msg.user/.msg.brain .bubble`, `.source-chip`, `.msg.brain.is-deflected`,
`#kb-banner`, `#app-version`; sources: `#stat-docs`, `#stat-chunks`,
`#stat-last`, `#docs-table`, `#docs-tbody`, `#sources-empty`; viewer
(Phase 10): `/document.html`, `#doc-title`, `#doc-meta`, `#doc-content`,
`.doc-raw`, `.format-badge`, `#doc-not-found`, `.doc-link` (Sources table
path links); thinking (phase 17, owner permission 2026-08-23):
`.thinking`, `.thinking-text` (collapsible thinking block; plain
`<summary>`, no id); auth (phase 16, owner permission 2026-08-22):
`#sign-in-link`, `#sign-out-btn`, `#sources-gate`; shared header (phase
19, owner permission 2026-08-23): `#nav-sources` (Sources nav link,
hidden for anonymous), `#new-chat-btn` + `#sign-in-link` + `#sign-out-btn`
on the sources and viewer pages (ids shared with chat),
`.doc-header-actions` (viewer).
---
## 8. Debugging (debugpy protocol)
- `DEBUGPY` unset/`0` → **`debugpy` is never imported** (verified by unit test).
- `DEBUGPY=1` → listener on `0.0.0.0:${DEBUGPY_PORT:-5678}`, **non-blocking**,
app continues; IDE attaches on demand.
- Entry point: `app/core/debugging.py::configure_debugging()` called at the top
of `app/main.py` module import — so `uv run uvicorn app.main:app`,
`python -m scripts.…`, and tests all honor it.
- VS Code: `"type": "debugpy", "request": "attach", "connect": {"host": "localhost", "port": 5678}`.
---
## 9. Observability
- **App logs:** single-line `timestamp LEVEL logger :: message` on stdout;
uvicorn access logs on. INFO by default (`BOR_LOG_LEVEL`).
- **Per-chat-turn log line (required):**
`question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N kb_chars=N threshold=… deflected=… sources=… thinking_chars=… tool_calls=N total_ms=…`
(`thinking_chars=` counts the turn's reasoning chars — phase 17, owner
permission 2026-08-23 — and is counted even when `BOR_STREAM_THINKING=0`
suppresses the frames.)
> **Log-line revision (phase 37, owner permission 2026-08-26):** the
> required per-turn line gains `tool_calls=N` after `thinking_chars=` —
> the count of agent tool executions that consumed budget on the turn
> (phase 37's `list_documents` / `read_document`; rejected calls do not
> count, and deflected turns run no tools). `summary_hits=` (phase 30)
> and `kb_chars=` (phase 31) are recorded here as well; `sources=` lists
> the retrieval docs plus any agent-read documents, deduped.
- **Importer logs:** per-file `added|updated|unchanged|pruned` + summary
(counts, embedding batches, total time).
- **`query_log` table:** durable record of every question (score, deflection,
sources, latency) for tuning the threshold and finding gaps in the docs.
---
## 10. Testing Strategy (LOCKED — A16)
| Layer | Tooling | Runs | Gate |
|-------|---------|------|------|
| Unit | pytest | `uv run pytest tests/unit` | pass |
| Integration | pytest + FastAPI TestClient | `uv run pytest tests/integration` | pass |
| Coverage | pytest-cov on `app/` | `uv run pytest --cov=app --cov-report=term-missing` | **>90%** per phase |
| E2E | Playwright (sync API), one file per story | `uv run pytest tests/e2e/test_<story>.py -v --no-cov` | passes **in isolation** |
- **E2E determinism:** `tests/e2e/mock_llm.py` serves a deterministic
OpenAI-compatible API. Embeddings are genuine L2-normalized token-overlap
vectors, so the cosine threshold behaves like production: on-topic
questions retrieve, off-topic questions deflect. `E2E_REAL_LLM=1` switches
the app fixture to live aipi (needs imported KB).
- **E2E prerequisites:** `podman compose up -d db`; Chromium installed via
`uv run playwright install chromium`.
- DB isolation: story E2E fixtures truncate `query_log` (and re-import
fixtures for import-dependent stories) per test module.
---
## 11. Import & Update Workflow (documented in README)
```
# first import (and any future refresh):
uv run python -m scripts.import_docs # defaults: ~/Homelab ~/Deployments
uv run python -m scripts.import_docs --source ~/OtherProject # extra dirs
uv run python -m scripts.import_docs --prune # drop deleted / filtered-out files
uv run python -m scripts.eval_retrieval "How did I install gitlab?"
uv run python -m scripts.llm_probe # sanity: models + dim
```
Behavior: sha256 delta per `(source, path)` — unchanged files are skipped
(no re-embedding); changed files are re-chunked + re-embedded (chunks
replaced atomically); `--prune` removes docs whose files disappeared or no
longer match the format filter. Formats per A9 (revised): `md, markdown,
txt, yaml, yml, json, py` (`BOR_IMPORT_EXTENSIONS`), hidden (dot)
directories skipped, exclusion list applied. `scripts/eval_retrieval.py`
ranks live hybrid results for a question (retrieval tuning).
---
## 12. Roadmap (one story → one phase → one Playwright gate)
| Phase | File | Story | Playwright gate |
|-------|------|-------|-----------------|
| 01 | `01_infrastructure.md` | — (foundation) | `tests/e2e/test_smoke.py` |
| 02 | `02_story_import_documents.md` | `import-documents.md` | `tests/e2e/test_import_documents.py` |
| 03 | `03_story_chat_rag.md` | `chat-rag-answer.md` | `tests/e2e/test_chat_rag.py` |
| 04 | `04_story_honest_deflection.md` | `honest-deflection.md` | `tests/e2e/test_honest_deflection.py` |
| 05 | `05_story_suggestion_chips.md` | `suggestion-chips.md` | `tests/e2e/test_suggestion_chips.py` |
| 06 | `06_story_loading_feedback.md` | `loading-feedback.md` | `tests/e2e/test_loading_feedback.py` |
| 07 | `07_story_responsive_polish.md` | `responsive-polish.md` | `tests/e2e/test_responsive_polish.py` |
| 08 | `08_story_dark_tech_theme.md` | `dark-tech-theme.md` | `tests/e2e/test_dark_tech_theme.py` |
| 09 | `09_story_retrieval_quality.md` | `retrieval-quality.md` | `tests/e2e/test_retrieval_quality.py` |
| 10 | `10_story_document_viewer.md` | `document-viewer.md` | `tests/e2e/test_document_viewer.py` |
| 17 | `17_thinking_display.md` | `thinking-display.md` | `tests/e2e/test_thinking_display.py` |
| 18 | `18_follow_bottom_scroll.md` | `follow-bottom-scroll.md` | `tests/e2e/test_follow_bottom_scroll.py` |
| 19 | `19_shared_header.md` | `shared-header.md` | `tests/e2e/test_shared_header.py` |
| 24 | `24_whole_document_context.md` | `whole-document-context.md` | `tests/e2e/test_whole_document_context.py` |
> Row 17 (thinking display) added 2026-08-23 with owner permission — the
> A15 SSE extension recorded in §4.
>
> Row 18 (follow-the-bottom scroll) added 2026-08-23 with owner choice —
> option 1: follow-the-bottom, no "↓ new content" pill (UI-behavior-only
> change; no anchor revised).
>
> Row 19 (shared header) added 2026-08-23 with owner permission —
> Sign in/Sign out + New Chat on every page via one shared module, and
> the phase-16 "Sources" link UX choice revised: the nav link is hidden
> for anonymous (the soft-gate page and the A10 API split are unchanged).
>
> Row 24 (whole-document context) added 2026-08-24 with owner permission
> — A7's 24k context cap removed (documents are never truncated)
Completion = unit+integration green, coverage >90%, story E2E green in
isolation, UI verification passed, **one `--no-gpg-sign` commit**.
---
## 13. Future (post-v1 hooks, deliberately not built)
- Auth (stateless API makes this a drop-in: sessions → Valkey).
- HNSW index on `chunks.embedding` at scale.
- Conversation persistence (messages tables).
- Watchdog auto-re-import (inotify) — until then the script is the truth.
- More sources: any directory of A9-format files via `--source`.
@@ -1,70 +0,0 @@
# Task 02 — E2E story suite, story file, validation, commit
**Phase:** `21_thinking_no_scroll` · **Source:** `TODO.md` L4
## Objective
The story gate: `tests/e2e/test_thinking_no_scroll.py` proves the window
can't be user-scrolled but always tracks the live tail, plus regressions,
story file, final validation, and the single atomic commit.
## Work
1. `tests/e2e/test_thinking_no_scroll.py` (new — reuse
`test_thinking_display.py`'s mock-LLM streaming scaffolding; the mock
must stream a **long** thinking body, in many chunks, so
`.thinking-text` overflow exceeds its 320px box). Tests:
1. `test_thinking_window_not_user_scrollable` — open the block, wait
until `scrollHeight > clientHeight`; focus `.thinking-text`
(`el.focus()`), dispatch mouse wheel over it
(`page.mouse.wheel(0, -200)` after moving the mouse over the
element) and press `Home`/`ArrowUp`: `scrollTop` must not decrease
(assert `scrollTop` unchanged within 1px between actions).
2. `test_thinking_window_tracks_live_tail` — while chunks stream,
after the 2nd-to-last and last chunk:
`scrollTop === scrollHeight` (within 1px) — the visible window is
the live tail; the **last** chunk's text is within the visible
rectangle (its offsetTop + scrollTop geometry check, or
`elementFromPoint` at the box's bottom).
3. `test_thinking_window_css_contract` — computed style of
`.thinking-text`: `overflow-y === "hidden"`,
`max-height === "320px"`.
4. `test_answer_bubble_still_scrollable` (regression, phase 11) — a
long answer (use the long-answer mock from
`test_long_answers.py`): the answer bubble is still
user-scrollable (scrollTop moves on wheel) and
`overflow-y` is not `hidden` there.
5. `test_restored_collapsed_thinking_unaffected` (regression,
phase 17) — a turn with stored `thinking`, reload: the collapsed
Thinking block renders with its text (existing pin from
`test_thinking_display.py` — replicate, don't duplicate the file).
2. `.agent/user_stories/thinking-no-scroll.md` (new) — story file per
the repo format: goal, the bug report verbatim from `TODO.md` L4, the
owner-confirmed A2 decisions from `00_phase.md`, E2E mapping table.
3. Run the suite **in isolation** (prereq `podman compose up -d db`):
`uv run pytest tests/e2e/test_thinking_no_scroll.py -v --no-cov`.
4. Regressions, in isolation, one command each:
- `uv run pytest tests/e2e/test_thinking_display.py -v --no-cov`
- `uv run pytest tests/e2e/test_long_answers.py -v --no-cov`
5. Final validation: `uv run pytest` green; `uv run pytest --cov=app
--cov-report=term-missing` ≥ today's number (>90% gate);
`uv run ruff check . && uv run pyright` clean.
6. **UI Structure Check** (AGENTS.md rule 5): no new surface; the block
keeps its summary chevron, focus-visible ring, aria-live/label
contract, and the reduced-motion stillness (styles.css ~line 686).
7. Write the phase report (`.agent/reports/21_thinking_no_scroll/`).
8. Commit (one atomic commit) and move the phase:
```bash
git add -A .agent/ frontend/ tests/
git commit --no-gpg-sign -m "fix(ui): thinking window no longer scrolls — live 320px view pinned to the stream tail"
mv .agent/phases/todo/21_thinking_no_scroll .agent/phases/complete/
```
## Testing & Quality
- Story suite green **in isolation**; both regression suites green in
isolation; full unit+integration suite green; `app/` coverage at or
above today's number (>90%); ruff + pyright clean.
## Completion Criteria
- [ ] `test_thinking_no_scroll.py` 5/5 in isolation.
- [ ] Regressions (thinking display, long answers) green in isolation.
- [ ] Story file + phase report exist.
- [ ] One `--no-gpg-sign` commit; phase directory in `complete/`.
@@ -1,60 +0,0 @@
# Task 01 — Diagnose and fix the animated background (styles.css)
**Phase:** `22_background_animation` · **Source:** `TODO.md` L5 —
*"Fix background animation not working, just blinking"*
## Objective
Find why the phase-08 animated background reads as "just blinking" and
fix `styles.css` so the grid drift and the glow breathe are both visibly
and smoothly alive, per the phase-08 design comments (pure CSS, zero JS).
## Work
1. **Reproduce:** `uv run uvicorn app.main:app --reload` (db up), open `/`
in a visible Chromium window (Playwright or the interactive browser),
observe ≥15s. Note exactly what moves and what doesn't.
2. **Diagnose per the `00_phase.md` checklist** (per-layer visibility
toggles, `background-position` samples on `body::before`, mask
inspection, occlusion check against `html`/`body` rules, glow
opacity-swing perception). Record findings + before-screenshot in
`.agent/reports/22_background_animation/` and
`.agent/screenshots/22_background_animation/`.
**ASSUMPTION (to verify, not assume):** the likely culprits, in
order — (a) the masked grid drift is too faint/slow to perceive,
(b) only the glow opacity swing is visible and it reads as a blink,
(c) a later rule occludes the `z-index: -1` layers. Confirm which
one actually fires before touching CSS; the fix must match the found
cause.
3. **Fix in `frontend/assets/styles.css`** (smallest change that makes
the design read):
- grid: raise line alpha and/or the mask's visible radius and/or the
drift speed as needed for a clearly visible, seamless drift
(drift delta must still equal one 44px cell for a seamless loop —
if the speed changes, keep `background-position` 0→44px and only
move the duration);
- glow: if the breathe reads as a blink, narrow the opacity delta
(e.g. 0.8↔1) and/or lengthen the period — it must read as
breathing, not pulsing;
- keep: both layers `position: fixed; inset: 0; z-index: -1;
pointer-events: none`; no `filter: blur`; no JS; palette/contrast
untouched.
4. **After-screenshot** (same viewport, two frames a few seconds apart
showing motion) into the same screenshots dir.
5. `tests/unit/test_background_animation.py` (new — repo source-pin
pattern): pin the **final** `styles.css` values — both
`@keyframes` present, `body::before` → `bg-grid-drift linear
infinite`, `body::after` → `bg-glow-breathe`, both layers
`fixed`/`z-index: -1`/`pointer-events: none`, `html` keeps
`background: var(--bg)`, `body` keeps `background: transparent`.
6. Manual re-verify: the "just blinking" perception is gone — smooth
drift + gentle breathe, no jank, no static frame.
## Testing & Quality
- `uv run pytest tests/unit/test_background_animation.py -v` green.
- `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] Root cause documented (with before/after screenshots) in the
phase report dir.
- [ ] Both layers visibly animate as the phase-08 design describes;
pure CSS, zero JS, no blur.
- [ ] Unit pins green against the final values; lint/types clean.
@@ -1,120 +0,0 @@
# Phase 23 — Containerfile: Build the Whole App Image Again
**Source:** `TODO.md` L6 — *"Fix Containerfile build not working"*
**Story:** `.agent/user_stories/containerfile-build.md` (created by task 02)
**Context:** `Containerfile` (3 stages: node:22-alpine + esbuild
0.25.5 frontend bundle → uv/python deps → slim runtime serving
`/app/static`); `frontend/` (4 pages: `index.html`, `sources.html`,
`document.html`, `login.html`; assets: `styles.css`, `markdown.js`
(classic script), `header.js`/`app.js`/`sources.js`/`document.js`/
`login.js` (ES modules)); `scripts/entrypoint.sh`.
## Verified diagnosis (2026-08-24, this conversion — not a guess)
1. **Root cause of the build failure:** phase 19 switched the page
scripts to `import … from "/assets/header.js"` (an absolute URL).
esbuild resolves that as the *filesystem* path `/assets/header.js`
and the stage-1 bundle dies:
`✘ [ERROR] Could not resolve "/assets/header.js"`
(reproduced with esbuild **0.25.5**, the exact pinned version, on a
copy of `frontend/`).
2. **Secondary gap (image would be broken even if it built):** stage 1
bundles only `app.js` + `sources.js` and copies only `index.html` +
`sources.html`. Missing from the image: `document.html` +
`login.html` (phases 10/16), `document.js` + `login.js`, and
`markdown.js` (classic script loaded by `index.html` +
`document.html`).
3. **Verified fix:** with relative imports (`from "./header.js"`) all
four page scripts bundle cleanly with esbuild 0.25.5.
4. **Latent double-evaluation trap:** all four HTML pages also load
`<script type="module" src="/assets/header.js">` directly while the
page script imports it. In dev the browser dedupes (same module
URL) — but in the image the bundled page script already contains the
header code, so shipping a raw `header.js` too would evaluate the
module **twice** (duplicate sign-out listener, double init). The
direct tags are redundant: the page script's `import` is hoisted and
guarantees `header.js` evaluates before the page script's body calls
`initSharedHeader()`, in dev and in the bundle alike.
## Objective
`podman build -f Containerfile .` succeeds, and the resulting image
serves the **whole app** — all four pages with their bundled, minified,
local-only assets (No CDN rule) — with `header.js` evaluated exactly
once per page.
## Owner-confirmed (2026-08-24, roadmap A4)
1. **Relative imports** (`./header.js`) over an esbuild alias — simpler,
verified working, dev-server behavior unchanged (files are
side-by-side).
2. **Remove the four redundant direct `header.js` script tags** (the
design above) rather than ship a raw `header.js` into the image —
single module evaluation, no duplicate listeners.
3. The image must cover **all four pages + all local assets** they
reference — the integration test (task 02) enforces this coverage so
the gap cannot silently reappear.
## Dependencies
- `19_shared_header` (complete) — introduced the absolute imports (root
cause) and the direct `header.js` tags.
- `10_story_document_viewer` / `16_admin_auth` (complete) — the pages
missing from the image.
- `08_story_dark_tech_theme` (complete) — No CDN rule the image must
honor.
## Tasks
1. `01_fix_containerfile_build.md` — relative imports, tag removal,
stage-1 asset coverage, green `podman build`, image smoke test.
2. `02_integration_test_commit.md` — `tests/integration/
test_containerfile_assets.py` (hermetic coverage pin), regression
suites, story file, final validation, the single atomic commit,
phase move to `complete/`.
## Locked decisions
- **A11 honored** — vanilla JS, no CDN, static serving from FastAPI.
**A16 honored** — integration test for the new build coverage; story
file + report; no Playwright suite required (this phase is
build/infrastructure — the phase gate is the hermetic integration
test + the real `podman build` + image smoke recorded in the report,
plus the dev-server E2E regressions). No anchor changed.
## Testing & Quality
- **Integration (new `tests/integration/test_containerfile_assets.py`,
hermetic — no podman, no network):** every `frontend/*.html` is
copied into stage 1's `/out`; every local `src`/`href` asset
referenced by the four pages is produced by a stage-1 line (esbuild
`--outfile` or `cp`); the four page module scripts are the exact set
esbuild bundles; `markdown.js` is produced; no HTML references
`/assets/header.js` directly (single-evaluation design pin); the
esbuild version stays pinned.
- **Unit:** none (no `app/` changes).
- **Coverage:** the >90% `app/` gate is unaffected, re-run to prove it.
- **Build gate (manual, recorded in the report):** `podman build
-f Containerfile .` green; image smoke (task 01 step 6) results +
log excerpt in `.agent/reports/23_containerfile_build/`.
- **Dev regressions (E2E, isolated):** `test_smoke.py`,
`test_shared_header.py`, `test_chat_persistence.py` (the HTML tag
removal touches dev page load).
- **Lint/types:** `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] Local esbuild 0.25.5 bundles all four page scripts cleanly.
- [ ] `podman build -f Containerfile .` green (log excerpt in the
report).
- [ ] Image smoke: container runs (throwaway Postgres 17 + pgvector);
`GET /`, `/sources.html`, `/document.html`, `/login.html` → 200;
`/assets/app.js` minified and contains the header code;
`/assets/markdown.js` 200; no `http(s)://` asset reference in any
served page (No CDN rule).
- [ ] Dev server unchanged in behavior: the three regression E2E suites
green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app
--cov-report=term-missing` ≥ today's number.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/containerfile-build.md` exists.
- [ ] One `--no-gpg-sign` commit (below);
`.agent/phases/todo/23_containerfile_build/` moved to
`.agent/phases/complete/`.
## Commit
```bash
git add -A .agent/ Containerfile frontend/ tests/ && git commit --no-gpg-sign -m "fix(build): Containerfile builds again — relative module imports, all four pages and shared assets in the image"
```
@@ -1,91 +0,0 @@
# Task 01 — Fix the build: relative imports, tag removal, full stage-1 asset coverage
**Phase:** `23_containerfile_build` · **Source:** `TODO.md` L6 —
*"Fix Containerfile build not working"*
## Objective
Make `podman build -f Containerfile .` succeed and ship the **complete**
frontend in the image: all four pages, all four bundled page modules,
the classic `markdown.js`, and the minified `styles.css` — with
`header.js` evaluated exactly once per page.
## Work
1. **Reproduce the failure** and record it in
`.agent/reports/23_containerfile_build/` (log excerpt):
- fast: `npx -y esbuild@0.25.5` on a copy of `frontend/` → the
`Could not resolve "/assets/header.js"` error (root cause, already
reproduced during conversion);
- authoritative: `podman build -f Containerfile .` → stage 1 fails
at the same line.
2. **`frontend/assets/{app,sources,document,login}.js`** — change the
header import from absolute URL to relative (one line each; the
specifiers are currently `from "/assets/header.js"`):
```js
import { … } from "./header.js";
```
(owner-confirmed A4 — relative over esbuild alias; dev-server
behavior is unchanged since the files are side-by-side and the
module URL resolves to the same file.)
3. **Remove the four redundant direct `header.js` tags** (owner-confirmed
A4-2 — the single-evaluation design from `00_phase.md`):
- `frontend/index.html` (~line 119) —
`<script type="module" src="/assets/header.js"></script>`;
- `frontend/sources.html` (~line 125), `frontend/document.html`
(~line 80), `frontend/login.html` (~line 67) — same tag.
- Update the surrounding HTML comments that describe the
header-before-page-script load order (e.g. index.html ~lines
115–119): the order is now guaranteed by the page script's own
`import` (hoisted, evaluated before the page script body calls
`initSharedHeader()`).
4. **`Containerfile` stage 1** — cover the whole app (keep the pinned
`esbuild@0.25.5` and the existing flags):
```dockerfile
RUN mkdir -p /out/assets \
&& esbuild ./assets/app.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/app.js \
&& esbuild ./assets/sources.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/sources.js \
&& esbuild ./assets/document.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/document.js \
&& esbuild ./assets/login.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/login.js \
&& esbuild ./assets/markdown.js --minify --outfile=/out/assets/markdown.js \
&& esbuild ./assets/styles.css --minify --outfile=/out/assets/styles.css \
&& cp ./index.html ./sources.html ./document.html ./login.html /out/
```
(`markdown.js` is a classic script — minify only, **no** `--bundle`;
it exposes globals used by the pages.)
5. **Verify locally (no podman):** with esbuild 0.25.5, all four module
bundles + the markdown minify succeed on the real `frontend/` (not a
copy — the copy was only for the diagnosis).
6. **`podman build -f Containerfile .`** → green.
7. **Image smoke test** (results + log excerpt into the report dir):
- throwaway Postgres 17 + pgvector (`podman compose up -d db` and
point the container at it, or a one-off container with the same
env as `compose.yaml`);
- run the built image (migrations run via the entrypoint);
- `GET /`, `/sources.html`, `/document.html`, `/login.html` → 200;
- `GET /assets/app.js` → 200, minified (single-line-ish), and
contains the header code (e.g. the `clearChatStorage` function
body); `GET /assets/markdown.js`, `/styles.css`, the other three
page modules → 200;
- No CDN rule: none of the four served pages contain an `http(s)://`
`src`/`href` asset reference.
- Teardown the throwaway containers when done.
8. **Dev-server regression check** (the tag removal touches dev page
load — confirm boot order still holds): `uv run uvicorn
app.main:app --reload`, load all four pages, check the sign-out
binding exists exactly once (DevTools: no duplicate listener — one
`POST /api/logout` per click) and `initSharedHeader()` ran. (The
isolated E2E regressions run in task 02.)
## Testing & Quality
- Steps 5–8 above; `uv run ruff check . && uv run pyright` clean
(no Python changes, but keep the gate green).
## Completion Criteria
- [ ] The recorded build failure is fixed at the root cause (relative
imports) — not masked by an alias/patch.
- [ ] All four direct `header.js` tags removed + comments updated; the
page scripts' `import "./header.js"` is the only header load.
- [ ] Stage 1 produces: 4 HTML pages, 4 bundled modules, minified
`markdown.js`, minified `styles.css`.
- [ ] `podman build` green; image smoke all-200 + No CDN + single
header evaluation; dev-server boot unchanged (step 8).
- [ ] Log/screenshot evidence in `.agent/reports/23_containerfile_build/`.
@@ -1,70 +0,0 @@
# Task 02 — Integration coverage test, story file, validation, commit
**Phase:** `23_containerfile_build` · **Source:** `TODO.md` L6
## Objective
Pin the stage-1 asset coverage so it can't silently rot again (a new
page/script/asset without a matching Containerfile line fails CI), plus
regressions, story file, final validation, and the single atomic commit.
## Work
1. `tests/integration/test_containerfile_assets.py` (new — **hermetic**:
parses `Containerfile` + `frontend/` as text, no podman, no network).
Tests:
1. `test_every_html_page_is_copied_into_stage1` — for each
`frontend/*.html` in the repo, a stage-1 line copies it into
`/out` (regex over the `cp` line; the set must be exactly the
four current pages — a new page added to `frontend/` fails this).
2. `test_every_local_asset_reference_is_produced` — collect every
local `src=`/`href=` under `assets/` or `/assets/` from the four
HTML files; each basename must be produced by a stage-1 line
(an `esbuild … --outfile=/out/assets/<name>` or a `cp` of it).
(This is what catches a missing `markdown.js`-style gap.)
3. `test_page_module_scripts_are_bundled` — the set of `type="module"`
page scripts referenced by the HTML (basenames) equals the set of
scripts esbuild bundles in stage 1 (`app.js`, `sources.js`,
`document.js`, `login.js`).
4. `test_header_module_is_imported_not_directly_loaded` — no HTML
file contains a `<script … src="/assets/header.js">` (or
`assets/header.js`) tag (the single-evaluation design pin,
owner-confirmed A4-2); and each of the four page scripts imports
it relatively (`from "./header.js"`).
5. `test_markdown_js_is_a_produced_classic_script` — `markdown.js`
has a stage-1 minify line **without** `--bundle` (it is a classic
global script) and no `import`/`export` statements at its top
level (source pin of that assumption).
6. `test_esbuild_stays_pinned` — the frontend stage pins a concrete
`esbuild@X.Y.Z` version (no floating version).
2. `.agent/user_stories/containerfile-build.md` (new) — story file per
the repo format: goal, the bug report verbatim from `TODO.md` L6, the
verified diagnosis (root cause + missing-asset gap + double-eval
trap), the owner-confirmed A4 decisions, and the test mapping table.
3. Regressions, in isolation, one command each (prereq
`podman compose up -d db`):
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov`
- `uv run pytest tests/e2e/test_shared_header.py -v --no-cov`
- `uv run pytest tests/e2e/test_chat_persistence.py -v --no-cov`
4. Final validation: `uv run pytest` green (includes the new
integration test); `uv run pytest --cov=app --cov-report=term-missing`
≥ today's number (>90% gate); `uv run ruff check . && uv run pyright`
clean.
5. Finish the phase report (`.agent/reports/23_containerfile_build/` —
build log excerpt, smoke results, regression results).
6. Commit (one atomic commit) and move the phase:
```bash
git add -A .agent/ Containerfile frontend/ tests/
git commit --no-gpg-sign -m "fix(build): Containerfile builds again — relative module imports, all four pages and shared assets in the image"
mv .agent/phases/todo/23_containerfile_build .agent/phases/complete/
```
## Testing & Quality
- New integration suite green within `uv run pytest`; the three
regression E2E suites green in isolation; full suite green; `app/`
coverage at or above today's number (>90%); ruff + pyright clean.
## Completion Criteria
- [ ] `test_containerfile_assets.py` 6/6 within the full suite.
- [ ] Regressions (smoke, shared header, chat persistence) green in
isolation.
- [ ] Story file + phase report (build log + smoke evidence) exist.
- [ ] One `--no-gpg-sign` commit; phase directory in `complete/`.
@@ -1,138 +0,0 @@
# Phase 24 — Whole-Document Context: a matched document is never truncated
**Source:** `TODO.md` L3–L4 — *"Documents are truncated for some reason?
This should never happen"* + *"When the LLM matches a chunk it should get
the entire document placed in its context so it can see the whole thing
before answering the question"*
**Story:** `.agent/user_stories/whole-document-context.md` (created by
task 03)
**Context:** `app/rag/retriever.py::select_documents` (the one and only
place document content is cut — the 24k budget), `app/api/chat.py::plan_turn`
(passes the budget on both HIGH and LOW paths), `app/config.py`
(`max_context_chars`), `app/rag/prompts.py` (the shared `[…truncated…]`
marker — still owned by the steering section), `tests/unit/test_retriever.py`
(pins the current cap), `tests/e2e/mock_llm.py` + `tests/e2e/test_chat_rag.py`
(E2E patterns), `.env.example` + `README.md` (document the knob).
## Verified diagnosis (2026-08-24, this conversion — not a guess)
1. **The importer stores the whole file** — `app/rag/importer.py:199–228`
reads each file into `documents.content` in full (sha256 over the whole
content). No truncation at import time.
2. **Chunking never touches the LLM context** — the 2000-char target /
1200-char hard cap only shapes `chunks` rows (retrieval + embeddings);
the chat prompt is built from `documents.content`.
3. **The viewer serves raw content** — `GET /api/documents/content`
(`app/api/docs.py`) returns `doc.content` unchanged; no truncation there
either. So the owner's "truncated for some reason" is *not* a separate
import/viewer bug.
4. **The one and only truncation point is `select_documents()`**
(`app/rag/retriever.py:252–289`): the top-2 documents' combined text is
capped at `BOR_MAX_CONTEXT_CHARS` (default **24 000**) and the
lowest-ranked overflowing document is truncated in place with
`[…truncated…]`. `plan_turn()` (`app/api/chat.py`) passes the budget on
both the HIGH (grounded) and LOW (deflected) paths. The marker the owner
saw in answers comes from here.
5. **The cap is pinned + documented** — `tests/unit/test_retriever.py`
(`test_combined_content_capped_with_truncation_marker`,
`test_single_doc_over_budget_is_truncated_to_budget`,
`test_under_budget_no_truncation`); the knob is in `.env.example`
(`BOR_MAX_CONTEXT_CHARS=24000`) and `README.md:377`.
## Objective
When hybrid retrieval matches a chunk, the LLM sees the **entire** parent
document — the 24k context budget (and `BOR_MAX_CONTEXT_CHARS`) is removed
from the document path, and a dedicated story E2E proves deterministically
that a >24k document — and the *second* document of a >24k pair (the exact
case the old budget cut) — reaches the model whole.
## Owner-confirmed (2026-08-24, roadmap D1–D5)
1. **D1 — no cap at all (revises LOCKED A7):** `select_documents` returns
the full top-N document texts, always. The `max_context_chars` setting
and `BOR_MAX_CONTEXT_CHARS` env var are removed. If a future KB ever
makes the prompt too large for the model, the existing `LLMError` → SSE
`error` path surfaces it loudly — no silent partial context. The
emergency-valve variant (raised cap + warning log) was **explicitly
rejected**.
2. **D2 — `top_n_docs = 2` unchanged** (the TODO is about truncation, not
about how many documents).
3. **D3 — no viewer/import changes** — both already serve full content
(verified diagnosis above).
4. **D4 — E2E evidence via a deterministic mock tail-echo** (repo pattern,
cf. the phase-15 tuning-note echo); the big documents are seeded
directly in the DB inside the E2E test — `tests/fixtures/docs/` must
not grow, because other suites pin `summary.added == 8`.
5. **D5 — no `query_log` schema change** (no new columns, no migration).
## Dependencies
- `03_story_chat_rag` (complete) — the RAG turn + `plan_turn` this phase
modifies.
- `09_story_retrieval_quality` (complete) — hybrid retrieval +
`select_documents` (A7) whose cap this phase revises.
- `15_steering_notes` (complete) — still owns `[…truncated…]` +
`BOR_STEERING_MAX_CHARS` (shared marker; unchanged).
## Tasks
1. `01_remove_context_cap.md` — remove the 24k budget from
`select_documents`, `plan_turn`, `config`, `.env.example`, `README`;
rewrite the unit tests to pin *no* truncation.
2. `02_whole_doc_e2e_suite.md` — mock tail-echo trigger +
`tests/e2e/test_whole_document_context.py` (whole >24k doc, whole second
doc of a >24k pair, small-doc regression).
3. `03_story_docs_plan_commit.md` — story file, PLAN.md A7/§6/§12 revision
(owner permission 2026-08-24), full validation, one `--no-gpg-sign`
commit, phase move.
## Locked decisions
- **A7 revision (owner permission 2026-08-24, D1):** A7's context clause
becomes *"feed the **full text of top-N=2 documents** (deduped)"* — the
"capped at 24k chars" clause is **removed**; matched parent documents are
**never truncated**. The revision note lands in PLAN.md via task 03
(phase-17/19 precedent — recorded, not silently deviated).
- **Steering unchanged** — `BOR_STEERING_MAX_CHARS` (8 000) still caps the
`<tuning>` section with the same `[…truncated…]` marker (phase-15
behavior byte-identical).
- **A16 honored** — dedicated Playwright story suite run in isolation;
unit + integration green; `app/` coverage >90%.
## Testing & Quality
- **Unit:** `tests/unit/test_retriever.py` — `select_documents` returns
byte-identical full content well past the old 24k budget, no marker;
ranking / dedup / n-cap tests unchanged.
- **Integration:** `uv run pytest tests/integration` green — existing
`test_chat_api.py` prompt tests exercise `plan_turn` through the new
signature (no integration test pins the cap — verified).
- **Coverage:** >90% on `app/` held
(`uv run pytest --cov=app --cov-report=term-missing`).
- **E2E (new, isolated):** `tests/e2e/test_whole_document_context.py` —
the tail sentinel of a 30k-char document (and of the *second* document of
a >24k pair) appears in the rendered answer; `[…truncated…]` never
appears; the small-document grounded path is unchanged.
- **Lint/types:** `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] `select_documents` has no budget parameter and never truncates;
`TRUNCATION_MARKER` remains for the steering section only.
- [ ] `BOR_MAX_CONTEXT_CHARS` gone from `app/config.py`, `.env.example`,
and `README.md`.
- [ ] `uv run pytest` green; `uv run pytest --cov=app
--cov-report=term-missing` ≥ today's number.
- [ ] `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`
green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/whole-document-context.md` exists; PLAN.md
carries the A7 revision + §6 bullet + §12 row 24 (owner permission
2026-08-24).
- [ ] One `--no-gpg-sign` commit (below);
`.agent/phases/todo/24_whole_document_context/` moved to
`.agent/phases/complete/`.
## Commit (task 03 — after the phase dir has moved to `complete/`)
```bash
git add -f .agent/phases/complete/24_whole_document_context/ .agent/user_stories/whole-document-context.md .agent/PLAN.md
git add -A .agent/phases/todo/24_whole_document_context/ app/ README.md .env.example tests/
git commit --no-gpg-sign -m "feat(rag): feed whole matched documents to the LLM — no context truncation (A7 revised)"
```
(The *conversion* commit — this phase dir force-added under `todo/` plus
the cleared `TODO.md` — lands separately when the roadmap is written, per
the phase-20–23 precedent, commit `824914c`.)
@@ -1,62 +0,0 @@
# Task 01 — Remove the 24k document-context budget
**Phase:** `24_whole_document_context` · **Source:** `TODO.md:3–4` — *"Documents are truncated for some reason? This should never happen"* + *"When the LLM matches a chunk it should get the entire document placed in its context so it can see the whole thing before answering the question"*
**Story:** `.agent/user_stories/whole-document-context.md` (created by task 03)
## Objective
`select_documents` returns the full text of the top-N parent documents,
always — the budget parameter, the `max_context_chars` setting, the env var,
and the documentation lines are removed, and the unit tests pin the new
no-truncation contract (owner-confirmed D1: no cap at all — the
emergency-valve variant was explicitly rejected).
## Work
1. `app/rag/retriever.py` — `select_documents(chunks, n=None)`: drop the
`max_chars` parameter and the whole `remaining`/truncation loop; return
the deduped top-N documents with their `content` byte-identical. Update
the module docstring (remove the "capped at ``BOR_MAX_CONTEXT_CHARS``"
sentence; state the A7 revision — whole documents, never truncated,
owner permission 2026-08-24) and the `select_documents` docstring.
`TRUNCATION_MARKER` **stays** exported — `app/rag/prompts.py`
(steering section, phase 15) still imports it.
2. `app/api/chat.py` — `plan_turn`: both `select_documents(…)` call sites
(HIGH and LOW paths) drop the `max_chars=settings.max_context_chars`
argument. No other turn-flow change (deflection, steering, `query_log`,
SSE all unchanged).
3. `app/config.py` — delete `max_context_chars: int = 24_000` and its
comment. (Any leftover `BOR_MAX_CONTEXT_CHARS=…` in an operator's
gitignored `.env` is silently ignored via `extra="ignore"` — no
migration, no startup check needed.)
4. `.env.example` — delete the `BOR_MAX_CONTEXT_CHARS=24000` line.
5. `README.md` — delete the settings-table row
`| BOR_MAX_CONTEXT_CHARS | 24000 | cap on total document text sent to the LLM |`
(line ~377). Leave the steering-notes `[…truncated…]` mention (~line 178)
alone — that budget still exists.
6. `tests/unit/test_retriever.py` — drop the `max_chars=…` argument from
every `select_documents` call; replace the three cap tests
(`test_combined_content_capped_with_truncation_marker`,
`test_single_doc_over_budget_is_truncated_to_budget`,
`test_under_budget_no_truncation`) with
`test_content_never_truncated_even_past_old_budget`: two documents of
20 000 + 15 000 chars (35 000 combined — past the old 24 000 cap) come
back with content **byte-identical** (assert equality against the
originals) and `TRUNCATION_MARKER` absent from both. Keep the module
docstring accurate ("ordering and dedup — no context cap, A7 revised").
## Testing & Quality
- Unit: `uv run pytest tests/unit/test_retriever.py -v` green — ranking,
dedup, n-cap, and the new no-truncation test.
- Integration: `uv run pytest tests/integration` green — the existing
`test_chat_api.py` prompt tests exercise `plan_turn` through the new
signature (no integration test pins the cap — verified during conversion).
- Coverage: `uv run pytest --cov=app --cov-report=term-missing` ≥ today's
number (the >90% gate).
## Completion Criteria
- [ ] `rg "max_context" app/ tests/ .env.example README.md` → **no matches**
(`steering_max_chars` is a different setting and must remain).
- [ ] `uv run pytest tests/unit/test_retriever.py -v` green.
- [ ] `uv run pytest` green; `uv run pytest --cov=app
--cov-report=term-missing` ≥ today's number.
- [ ] No behavior change outside the context budget — deflection, steering,
viewer, and import behave byte-identically to before.
@@ -1,84 +0,0 @@
# Task 02 — Story E2E: whole documents reach the LLM (mock tail-echo)
**Phase:** `24_whole_document_context` · **Source:** `TODO.md:3–4` — *"Documents are truncated for some reason? This should never happen"* + *"When the LLM matches a chunk it should get the entire document placed in its context so it can see the whole thing before answering the question"* (this task makes the no-truncation contract provable end-to-end)
**Story:** `.agent/user_stories/whole-document-context.md` (created by task 03)
## Objective
`tests/e2e/test_whole_document_context.py` proves, with the deterministic
mock, that a matched document **larger than the old 24k cap** — and the
**second** document of a pair whose combined size exceeds it (the exact case
the old budget cut) — reach the LLM whole.
## Work
1. `tests/e2e/mock_llm.py` — add one user-message trigger, same pattern as
`LONG_ANSWER_TRIGGER` and the phase-15 tuning-note echo:
- `END_OF_NOTES_TRIGGER = "show the end of your notes"` — verified
2026-08-24: no existing E2E question or fixture file contains the
phrase, so every other suite is unaffected.
- In `compose_answer`, after the `DEFLECT_MODE` check and before the
generic branch: when the trigger is present in the user message, the
answer quotes the **tail of the context** — e.g.
`f"…and the very end of my notes reads: “{_context(body)[-160:]}” (Deterministic mock answer for E2E.)"`.
Byte-stable across runs, so a sentinel placed at the *end* of a
document appears in the rendered answer **iff the whole document was
in the prompt**. (The tail includes the closing `</documents>` —
harmless for `to_contain_text` sentinel assertions.)
- Keep the mock docstring's trigger list updated.
2. `tests/e2e/test_whole_document_context.py` — new story suite. Reuse the
`page`, `app_url`, `mock_llm`, `db_ready` fixtures and the
TRUNCATE-then-seed pattern from `tests/e2e/test_chat_rag.py` /
`tests/integration/test_document_content.py`. **Seed the big documents
directly via SQLAlchemy** (a `documents` row + 2–3 `chunks` rows, each
chunk with `embedding = embed_text(chunk_text)` imported from
`tests.e2e.mock_llm` — the same deterministic bag-of-words vector the
mock computes, so the question's live mock embedding genuinely overlaps).
**Do not add fixture files** — `tests/fixtures/docs/` must stay at its
current 8 files; other suites pin `summary.added == 8`.
- Content helper: a ~30 000-char document = repeated
"gitlab install playbook" body (tokens shared with the question →
hybrid hit) whose **last line is a unique sentinel**, e.g.
`WHOLE-DOC-TAIL-GITLAB-<test-unique>`.
- `test_whole_document_over_old_cap_reaches_llm` — seed one 30 000-char
document (past the old 24 000 cap); ask
`"Show the end of your notes about the gitlab install playbook, please."`;
assert the rendered brain bubble contains the tail sentinel **and**
`"Deterministic mock answer for E2E"`, contains **no**
`[…truncated…]`, the document's `.source-chip` renders, and the
`query_log` row has `deflected == False`.
- `test_second_document_of_over_cap_pair_reaches_llm` — seed two
documents of ~16 000 chars each (32 000 combined — under the old
budget the lower-ranked one was truncated in place). Doc 1's first
chunk carries the question's key tokens ("gitlab install playbook");
doc 2's first chunk carries a weaker overlap ("playbook", "notes") so
doc 1 ranks first and doc 2 is the **last block inside
`<documents>`** — i.e. the tail-echo surfaces doc 2's sentinel. Assert
doc 2's tail sentinel in the answer. If the ranking ever flips locally,
strengthen doc 1's token overlap (repeat the question phrase in its
first chunk) until two consecutive isolated runs are stable — do not
weaken the assertion.
- `test_small_document_path_unchanged` — regression: re-import the
standard fixtures via the real importer (the `test_chat_rag.py`
`_import_fixtures` pattern), ask the usual on-topic question
(`"How is my Kubernetes cluster set up?"`) → grounded answer +
`kubernetes.md` chip, no `[…truncated…]` (the under-24k path is
byte-identical to before).
- Header comment: story path + run-in-isolation command
(`uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`).
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`
green **in isolation** (DB up: `podman compose up -d db`).
- E2E regressions: `uv run pytest tests/e2e/test_chat_rag.py
tests/e2e/test_honest_deflection.py -v --no-cov` still green (the mock
change is additive — the trigger phrase appears in no existing question).
- Coverage: the >90% `app/` gate is unaffected (tests + mock only) — re-run
`uv run pytest --cov=app --cov-report=term-missing` to prove it.
- Lint/types: `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`
— all three tests green in isolation.
- [ ] `uv run pytest tests/e2e/test_chat_rag.py
tests/e2e/test_honest_deflection.py -v --no-cov` green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `tests/fixtures/docs/` unchanged (still 8 files).
@@ -1,76 +0,0 @@
# Task 03 — Story file, PLAN.md revision, validation, commit
**Phase:** `24_whole_document_context` · **Source:** `TODO.md:3–4` — *"Documents are truncated for some reason? This should never happen"* + *"When the LLM matches a chunk it should get the entire document placed in its context so it can see the whole thing before answering the question"*
**Story:** `.agent/user_stories/whole-document-context.md` (created by this task)
## Objective
Record the user story and the A7 revision (owner permission 2026-08-24),
run the full validation gate, land one atomic commit, and move the phase
to `complete/`.
## Work
1. `.agent/user_stories/whole-document-context.md` — story file in house
style (cf. `.agent/user_stories/sources-midstream.md`):
- Header: `**Phase:** 24_whole_document_context · **E2E:**
tests/e2e/test_whole_document_context.py`.
- Bug report section: the two TODO.md items (L3–L4), verbatim.
- Narrative: *Given* a question whose chunk matches a document,
*when* the turn assembles context, *then* the LLM receives the
**entire** parent document — never a `[…truncated…]`-cut version —
so the answer is grounded in the whole note.
- Owner-confirmed (2026-08-24, roadmap D1–D5) — the five decisions from
the phase overview, verbatim.
- Acceptance criteria: no budget parameter in `select_documents`;
`BOR_MAX_CONTEXT_CHARS` gone from settings/env/README; the story E2E's
three tests green in isolation; steering-note truncation (phase 15)
unchanged; unit+integration green, coverage >90%, one
`--no-gpg-sign` commit.
- Playwright Mapping Rule: the three tests of task 02, one line each.
2. `.agent/PLAN.md` revisions (phase-17/19 precedent — record the owner
permission, never deviate silently):
- §2, A7 row: replace *"feed the **full text of top-N=2 documents**
(deduped, capped at 24k chars)"* with *"feed the **full text of
top-N=2 documents** (deduped)"* and append the rationale addendum:
"A7 revised 2026-08-24 — matched documents never truncated (owner:
'this should never happen'; emergency-valve variant rejected)".
- §2, revision-note block under the anchors table: add
**A7 revision (phase 24, owner permission 2026-08-24):** the
`[…truncated…]` cap on document context is removed —
`select_documents` always returns the full top-N texts;
`BOR_MAX_CONTEXT_CHARS` is gone. The steering section
(`BOR_STEERING_MAX_CHARS`, phase 15) keeps its budget and the shared
marker.
- §6, retrieval bullet: *"top 2 → full content, concatenated, truncated
to `BOR_MAX_CONTEXT_CHARS` (24k) with a `[…truncated…]` marker"* →
*"top 2 → full content, concatenated — **never truncated** (A7
revised, phase 24, owner permission 2026-08-24)"*.
- §12 roadmap table: add row 24 —
`| 24 | 24_whole_document_context.md | whole-document-context.md | tests/e2e/test_whole_document_context.py |`
plus a footnote: "Row 24 added 2026-08-24 with owner permission —
A7's 24k context cap removed (documents are never truncated)".
3. Final validation (AGENTS.md §9 — all must be green before the commit):
- `uv run pytest` (unit + integration)
- `uv run pytest --cov=app --cov-report=term-missing` (>90% gate)
- `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov`
(in isolation)
- `uv run ruff check . && uv run pyright`
4. Commit + phase move (AGENTS.md §8; use the commit block in
`00_phase.md` — it assumes the move happens first):
`mv .agent/phases/todo/24_whole_document_context
.agent/phases/complete/`, then the two `git add` lines + one
`--no-gpg-sign` commit.
## Testing & Quality
- All AGENTS.md §9 gates: unit + integration green, `app/` coverage >90%,
story E2E green in isolation, lint + types clean.
## Completion Criteria
- [ ] `.agent/user_stories/whole-document-context.md` exists (story,
D1–D5, acceptance criteria, Playwright mapping).
- [ ] `.agent/PLAN.md` carries the A7 row + revision note (§2), the §6
bullet, and the §12 row 24 + footnote — every one marked
"owner permission 2026-08-24".
- [ ] All validation commands green (work step 3).
- [ ] One `--no-gpg-sign` commit;
`.agent/phases/complete/24_whole_document_context/` exists and
`.agent/phases/todo/24_whole_document_context/` is gone.
@@ -1,128 +0,0 @@
# Task 01 — Still Background: Static Grid + Three Opacity-Only Glow Fades
**Phase:** `25_background_no_motion` · **Story:** `.agent/user_stories/background-no-motion.md`
## Objective
Redesign the background block of `frontend/assets/styles.css` to the
owner's spec (no movement; different bright spots slowly fading in and
out), and pin the new contract at source level.
## Work
0. **Capture the "before" evidence FIRST** (pre-change): with the DB up
(`podman compose up -d db`), boot the app the same way
`tests/e2e/conftest.py`'s `app_server` fixture does (uvicorn
`app.main:app` on a free port, wait for it to answer), and with a small
throwaway Playwright script screenshot the `/` page at 1280×800 →
`.agent/screenshots/25_background_no_motion/before.png`, then again
~4s later → `before_4s.png` (the pair shows the down-right jitter +
the uniform brightness pulse). Kill the server when done.
1. `frontend/assets/styles.css` — **grid layer `body::before`:** remove
the `animation: bg-grid-drift 60s linear infinite;` declaration and
delete the whole `@keyframes bg-grid-drift { … }` block. Keep the 44px
cells, the 60% `--line` alpha 1px lines, and the widened radial mask —
the grid remains as a **static texture**. Update the block's comment:
drift removed (owner 2026-08-25) — the 0.73px/s sub-pixel drift
rasterizes as a once-per-second down-right jitter; the owner wants no
movement.
2. `frontend/assets/styles.css` — **glow layers.** Replace the
whole-layer breathe with three independent spot layers:
- `body::after` — keep the phase-08 indigo spot exactly:
`background-image: radial-gradient(circle 56rem at 12% 8%,
rgb(109 120 242 / 0.14), transparent 62%);` and
`animation: bg-glow-a 26s ease-in-out infinite;`
- **new `html::before`** — the phase-08 cyan spot:
`background-image: radial-gradient(circle 60rem at 88% 92%,
rgb(34 211 238 / 0.10), transparent 62%);` and
`animation: bg-glow-b 34s ease-in-out -12s infinite;`
- **new `html::after`** — a third indigo spot:
`background-image: radial-gradient(circle 52rem at 14% 86%,
rgb(109 120 242 / 0.09), transparent 62%);` and
`animation: bg-glow-c 42s ease-in-out -23s infinite;`
- All three layers (and the grid layer) must declare:
`content: ""; position: fixed; inset: 0; z-index: -1;
pointer-events: none;` — no `transform`, no `background-position`,
no `filter` on any of them.
- Delete `@keyframes bg-glow-breathe { … }` and add the three
**opacity-only** keyframe blocks (nothing else may appear in any
`bg-*` keyframe):
```css
@keyframes bg-glow-a { 0%, 100% { opacity: 0.25; } 50% { opacity: 1; } }
@keyframes bg-glow-b { 0%, 100% { opacity: 0.20; } 50% { opacity: 1; } }
@keyframes bg-glow-c { 0%, 100% { opacity: 0.15; } 50% { opacity: 1; } }
```
3. `frontend/assets/styles.css` — **reduced motion:** in the existing
`@media (prefers-reduced-motion: reduce)` block that stills the
background (currently `body::before, body::after { animation: none; }`,
right after the spinner block), extend the selector list to all four
layers: `body::before, body::after, html::before, html::after
{ animation: none; }`. Do not touch the typing/spinner/thinking
reduced-motion blocks.
4. `frontend/assets/styles.css` — **section comment:** rewrite the
"Animated background" comment to document the new spec: no movement
(owner 2026-08-25); three independent soft spots, opacity-only fades
at 26/34/42s with negative delays → out of phase, so the total light
fluxuates smoothly and irregularly; `html::before`/`html::after` are
background layers (root stacking context: they paint above the
`var(--bg)` canvas and below the transparent, non-stacking `<body>`'s
content — the no-occlusion contract is unchanged).
5. **New `tests/unit/test_background_no_motion.py`** (repo source-pin
pattern — reuse the helpers from `tests/unit/test_background_animation.py`
(`_css`, `_css_no_comments`, `_rule_block`; note `_rule_block` matches
top-level `selector { … }`, which works for `html::before`/`html::after`
as written in step 2). Pin, at minimum:
- `body::before` carries **no `animation`** declaration;
`@keyframes bg-grid-drift` is absent from the file; the grid keeps
its static texture (`background-size: 44px 44px`, the two 60%-alpha
1px line gradients, the widened mask, both mask properties).
- `body::after` runs `bg-glow-a 26s ease-in-out infinite`;
`html::before` runs `bg-glow-b 34s ease-in-out -12s infinite`;
`html::after` runs `bg-glow-c 42s ease-in-out -23s infinite`;
each glow layer's `background-image` is exactly the single radial
gradient from step 2 (color, radius, position, 62% transparent stop).
- **No movement:** parse every `@keyframes bg-*` rule in the file —
the set of declared property names across all keyframe frames is
exactly `{opacity}` (no `transform`, `scale`, `background-position`,
…); none of the three glow layers declares `transform` or
`background-position`.
- The three glow durations are distinct and each ≥ 20s (slow).
- All four pseudo-layers: `position: fixed`, `inset: 0`,
`z-index: -1`, `pointer-events: none`, `content: ""`.
- Plumbing: `html` keeps `background: var(--bg)`; `body` keeps
`background: transparent` and declares none of `z-index`,
`transform`, `opacity`, `filter`.
- The reduced-motion block stills all four layers (all four selectors
present together with `animation: none`).
- No `filter` in any background layer block; no `blur` anywhere in
the file (comments stripped).
6. **Adapt `tests/unit/test_background_animation.py`** (the phase-22
source pins) to the new contract so the whole unit suite is green:
replace the grid-drift and breathe tests with their new-contract
equivalents (where a check is already covered by the new file, keep the
file self-contained rather than importing from it — the repo pattern is
local pins); keep the generic layer-plumbing tests
(`test_both_layers_are_fixed_zminus1_noninteractive` — extend it to
cover `html::before`/`html::after` — and
`test_html_owns_bg_and_body_stays_transparent`) and the no-blur/no-JS
anchor test; update the module docstring to describe the phase-25
design and point at `.agent/user_stories/background-no-motion.md`.
## Testing & Quality
- `uv run pytest tests/unit -v` green (new + adapted pins).
- `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ the
pre-change number (`app/` is untouched — the >90% gate holds).
- `uv run ruff check . && uv run pyright` clean.
- Coverage **>90%** on new/modified code: the functional change is CSS;
the new/modified Python is pytest source pins, exercised in full.
## Completion Criteria
- [ ] `styles.css`: no `bg-grid-drift`, no `bg-glow-breathe`, no
`animation` on `body::before`; exactly three opacity-only
`bg-glow-a/b/c` cycles on `body::after`, `html::before`,
`html::after` (26s/34s/42s, delays 0/−12s/−23s); reduced-motion
stills all four layers; comments document the owner's no-movement
spec.
- [ ] `uv run pytest tests/unit` green with
`tests/unit/test_background_no_motion.py` present and passing.
- [ ] `uv run pytest --cov=app` TOTAL ≥ pre-change; ruff + pyright clean.
- [ ] `.agent/screenshots/25_background_no_motion/before.png` and
`before_4s.png` captured **before** the CSS change.
@@ -1,44 +0,0 @@
# Phase 26 — Document Modal Viewer
**Source:** `TODO.md L4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
**Story:** `.agent/user_stories/document-modal.md`
**Context:** Phase 10 added the separate `/document.html` viewer page; phase 19 added the shared header bar that now lives on every page. The document content is served by the stateless `GET /api/documents/content` endpoint (PLAN §4).
## Objective
Stop opening cited documents in a new page/tab. Clicking a source chip or a Sources-table path link now opens the document in an **almost-fullscreen modal overlay** on the current page, fed by the same `/api/documents/content` endpoint. The existing `/document.html` page stays as the no-JS / direct-link fallback and its behaviour is unchanged.
## Dependencies
- `10_story_document_viewer` (complete) — the `/document.html` page, the `document.js` renderer, the `renderMarkdown` escape-first renderer in `markdown.js`, and the `#doc-content` / `.doc-md` / `.doc-raw` markup this phase reuses inside the modal.
- `19_shared_header` (complete) — the shared header bar the modal sits under; the modal must not disturb the header.
- `08_story_dark_tech_theme` (complete) — the Phase-08 tokens and the ≥4.5:1 contrast / `prefers-reduced-motion` contract the modal must honour.
## Tasks
1. `01_modal_css_and_html.md` — the modal CSS (overlay, backdrop, close button, scrollable content area) + inject the modal skeleton into `index.html`
2. `02_app_js_modal_intercept.md` — intercept document links in `app.js` + `sources.js`, fetch content via `/api/documents/content`, render inside the modal
3. `03_document_js_modal_mode.md` — adapt `document.js` to optionally render in modal mode (reuse the same API call) for the direct-link fallback path
4. `04_e2e_regression_suite.md` — update `test_document_viewer.py` to verify modal behaviour; the story gate, run in isolation
## Testing & Quality
- Unit/integration: none required for the modal itself (frontend-only); the `/api/documents/content` endpoint is unchanged (no `app/` change → no coverage delta).
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
- E2E: `tests/e2e/test_document_viewer.py` rewritten for the modal contract (task 4), green **in isolation** (prereq `podman compose up -d db`).
## Completion Criteria
- [ ] Clicking a source chip (chat) or a Sources-table path link opens the document in an almost-fullscreen modal on the **same page** (no new tab, no navigation).
- [ ] The modal renders the same content the `/document.html` page renders: md/markdown via the shared renderer (`.doc-md`), other formats in `<pre class="doc-raw">`, source/format/path/indexed/chunks meta.
- [ ] The modal has a visible close control, closes on Escape, closes on backdrop click, and keeps the dark theme + a11y frame (skip-link, focus trap, `:focus-visible`, aria-label).
- [ ] The existing `/document.html` page still works unchanged (direct link, back button, XSS-safe rendering, not-found state).
- [ ] No CDN tags on any touched page; every asset reference is same-origin or `data:`.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (gate >90%).
- [ ] `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5): modal content uses the standard centered column width for md; backdrop behind content; no 360px overflow.
- [ ] `.agent/user_stories/document-modal.md` exists.
- [ ] One `--no-gpg-sign` commit staging only this phase's files; `.agent/phases/todo/26_document_modal_viewer/` moved to `.agent/phases/complete/`.
## Locked decisions
- **No backend change** — the modal reuses `GET /api/documents/content` unchanged (A10 untouched: the API stays stateless).
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, zero new packages, no new assets, system font stack; the modal is pure CSS + JS.
- **No anchor revised** — this is a UI-behaviour change (PLAN §7.5 gains `#doc-modal`, `#doc-modal-backdrop`, `#doc-modal-close`, `#doc-modal-content`); the `/document.html` page and its story are unchanged.
- **A16 honoured** — one story E2E suite (rewritten) + adapted regressions.
- **A17 honoured** — one atomic `--no-gpg-sign` commit.
@@ -1,62 +0,0 @@
# Task 01 — Modal CSS + HTML skeleton
**Phase:** `26_document_modal_viewer` · **Source:** `TODO.md:4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
**Story:** `.agent/user_stories/document-modal.md`
## Objective
Add the modal markup to `index.html` and the CSS that styles an almost-fullscreen overlay (backdrop + panel + close button + scrollable content) using the Phase-08 tokens.
## Work
1. `frontend/index.html` — insert the modal skeleton just before the closing `</body>` (after the existing script tags, or before them — order doesn't matter for a static skeleton). The skeleton:
```html
<div class="doc-modal" id="doc-modal" hidden>
<div class="doc-modal-backdrop" id="doc-modal-backdrop" aria-hidden="true"></div>
<div class="doc-modal-panel" role="dialog" aria-modal="true" aria-labelledby="doc-modal-title" aria-describedby="doc-modal-desc">
<header class="doc-modal-header">
<h2 class="doc-modal-title" id="doc-modal-title">Loading…</h2>
<div class="doc-modal-actions">
<a class="doc-modal-open" id="doc-modal-open" target="_blank" rel="noopener" hidden aria-label="Open in full page">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"><path d="M18 13v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/><path d="M15 3h6v6"/><path d="M10 14 21 3"/></svg>
<span>Full page</span>
</a>
<button type="button" class="doc-modal-close" id="doc-modal-close" aria-label="Close document">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"><path d="M18 6 6 18M6 6l12 12"/></svg>
</button>
</div>
</header>
<div class="doc-modal-meta" id="doc-modal-meta" aria-live="polite"></div>
<p class="visually-hidden" id="doc-modal-desc" role="status">Document content is loading.</p>
<main class="doc-modal-content" id="doc-modal-content" tabindex="-1">
<p class="doc-modal-loading" role="status">Loading document…</p>
</main>
</div>
</div>
```
The `hidden` attribute keeps the modal off until JS opens it. The `#doc-modal-open` "Full page" link points at the same `/document.html?source=…&path=…&modal=…` URL the modal will build so a user can still open the dedicated page if JS is off.
2. `frontend/assets/styles.css` — add a `--doc-modal-*` block (Phase-08 tokens). Styling contract:
- `.doc-modal` — `position: fixed; inset: 0; z-index: 1000;` (above the shared header and every page layer, below the phase-25 background which is `z-index: -1`); the panel is flex, column; the backdrop + panel fill the viewport.
- `.doc-modal-backdrop` — `position: fixed; inset: 0; background: rgba(10,14,23,0.82);` backdrop blur is **not** used (phase-08 no-blur perf anchor); `opacity` transition 120ms.
- `.doc-modal-panel` — `display: flex; flex-direction: column; width: min(1100px, 96vw); height: 92vh; margin: auto; background: var(--surface, #121a2e); border: 1px solid var(--line, #232b52); border-radius: 12px; box-shadow: 0 24px 80px rgba(0,0,0,.55);` — "almost-fullscreen" = 96vw × 92vh, centered.
- `.doc-modal-header` — sticky top, same height/spacing as the doc header (64px / 58px pins from phase 12); title uses `--ink`; close button ≥44px target, focus-visible ring.
- `.doc-modal-content` — `flex: 1; overflow: auto;` (vertical scroll inside the panel, not the viewport); padding; the md content reuses `.doc-md` (≤46rem centered column) — the modal just provides the scroll container. For wide raw formats the `.doc-raw` pre already has `overflow-x: auto`.
- `.doc-modal-meta` — reuses the `.doc-meta` styling already defined for the viewer page (source/format/path/indexed/chunks badges); keep it compact (single row, wrap).
- `.doc-modal-close` — icon-only button, `aria-label` kept, `:focus-visible` 3px ring.
- Transitions respect `prefers-reduced-motion: reduce` (no opacity/transform animation, or `animation: none` under the reduced-motion media query — same pattern as the phase-25 background layers).
- `.doc-modal[hidden]` — `display: none` (the `hidden` IDL attribute default already hides it; add the rule to be explicit and testable).
- Ensure the modal panel does not add horizontal width at 360px (no `box-sizing` surprises; the panel is `96vw` ≤ viewport).
3. Verify the new CSS classes do not collide with any existing selector in `styles.css` (grep for `.doc-modal`, `.doc-modal-`).
## ASSUMPTIONS
- The modal panel is `96vw × 92vh` ("almost-fullscreen"). If the owner wants a different fraction, that's a follow-up.
- The "Full page" link is admin-agnostic (it just opens `/document.html`); it is shown for everyone since the viewer is public.
- The modal uses the existing `.doc-meta` badge classes already defined for the viewer page (no duplicate styling).
## Testing & Quality
- No unit/integration test for static CSS/HTML.
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
## Completion Criteria
- [ ] `index.html` contains the `.doc-modal` skeleton with the documented ids (`#doc-modal`, `#doc-modal-backdrop`, `#doc-modal-panel`, `#doc-modal-close`, `#doc-modal-title`, `#doc-modal-meta`, `#doc-modal-content`, `#doc-modal-open`).
- [ ] The modal CSS block is present, uses Phase-08 tokens, has no `filter: blur`/`backdrop-filter`, and the panel is `96vw × 92vh` centered.
- [ ] No selector collision (grep clean).
- [ ] `prefers-reduced-motion` stills any modal transition.
@@ -1,40 +0,0 @@
# Task 02 — Intercept document links → modal
**Phase:** `26_document_modal_viewer` · **Source:** `TODO.md:4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
**Story:** `.agent/user_stories/document-modal.md`
## Objective
Intercept document links on the chat page (`app.js` source chips) and the Sources page (`sources.js` table links): instead of navigating to `/document.html` in a new tab, fetch the document via `GET /api/documents/content` and render it inside the modal from task 01.
## Work
1. `frontend/assets/document.js` — extract the rendering logic into a reusable, importable function so both the standalone page (task 03) and the modal share the exact same renderer. Specifically:
- Export `renderDocument(doc, { containerEl, metaEl, titleEl })` that populates a title element, a meta element (the `.doc-meta` badges), and a content element (`.doc-md` for markdown via `renderMarkdown`, `<pre class="doc-raw">` otherwise). Keep the escape-first XSS-safety contract (`innerHTML` only through `renderMarkdown`; `textContent` for raw + badges).
- The standalone `document.html` page keeps its own `load()` that calls `renderDocument` with its page elements (task 03 wires this).
2. `frontend/assets/app.js` — add a `openDocumentModal(source, path)` helper:
- Build the modal URL: `/api/documents/content?source=…&path=…` (same encoding the chips already use).
- Show the modal: set `#doc-modal.hidden = false`, set the loading state, move focus into `#doc-modal-content` (a11y — the panel is `tabindex="-1"`).
- `fetch(contentUrl)` → on `!r.ok` render a short "document not found" line in the content area; on success call `renderDocument` with `#doc-modal-title`, `#doc-modal-meta`, `#doc-modal-content`.
- The "Full page" link (`#doc-modal-open`) is set to the `/document.html?source=…&path=…` URL on open.
- Keep the existing `documentUrl()` builder for the "Full page" link (unchanged output).
- Add modal close behaviour: `#doc-modal-close` click → `closeDocumentModal()`; backdrop click → close; `Escape` key → close; closing restores focus to the link that opened the modal (best-effort — store the triggering element).
3. `frontend/assets/app.js` — wire the source chips: replace `chip.target = "_blank"` navigation with `chip.addEventListener("click", e => { e.preventDefault(); e.stopPropagation(); openDocumentModal(s.source, s.path, chip); })`. Keep the `title`/aria-label truncation logic the chips already have. The chip keeps its `href` too (no-JS fallback would navigate to `/document.html`).
4. `frontend/assets/sources.js` — wire the table links the same way: the `.doc-link` click is intercepted, `preventDefault`, and `openDocumentModal(d.source, d.path, link)` is called. Since `openDocumentModal` lives in `app.js` (the chat page module) and `sources.js` is a separate module, **export** `openDocumentModal` from `app.js` and import it in `sources.js` — but `app.js` is loaded as a module on the chat page only. To avoid a second module instance, move the shared modal logic into a small new module `frontend/assets/document-modal.js` (task 02 step 1 refined below) and have both `app.js` and `sources.js` import it.
- **Refined split:** create `frontend/assets/document-modal.js` exporting `openDocumentModal(source, path, triggerEl)` and `closeDocumentModal()`. This module owns the modal DOM wiring (close on ESC / backdrop / button, focus management) and the `fetch` + `renderDocument` call. `app.js` and `sources.js` just call `openDocumentModal(...)` from their click handlers. This is the cleanest single-implementation approach (mirrors how `header.js` is the single owner of the shared header).
- `document.js` (standalone page) also imports `renderDocument` from itself (or a shared `document-render.js`) — keep the standalone page self-contained; it doesn't need the modal module.
## ASSUMPTIONS
- The modal module (`document-modal.js`) is a classic or module script loaded on both `index.html` and `sources.html`. It's a module (imports `renderDocument` from `document.js`), so both pages must load it via `<script type="module">`. `document.js` will export `renderDocument`.
- Close-on-`Escape` and close-on-backdrop are modal UX standards; the owner's item says "modal, not a new page", which implies standard modal affordances.
- The "Full page" link remains for users who want the dedicated viewer; it is optional and doesn't interfere with the modal.
## Testing & Quality
- No unit/integration test (frontend-only).
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
## Completion Criteria
- [ ] Clicking a source chip or a Sources-table path link opens the modal and renders the document (md via `.doc-md`, other formats via `.doc-raw`).
- [ ] The modal closes on button click, on backdrop click, and on `Escape`; focus returns to the triggering control.
- [ ] No new tab opens from either link type.
- [ ] The "Full page" link still navigates to `/document.html` (unchanged).
- [ ] XSS-safe rendering preserved (markdown escaped, raw set via `textContent`).
- [ ] Both pages load the modal module without a duplicate-module error.
@@ -1,29 +0,0 @@
# Task 03 — Standalone viewer page reuses the shared renderer
**Phase:** `26_document_modal_viewer` · **Source:** `TODO.md:4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
**Story:** `.agent/user_stories/document-modal.md`
## Objective
Keep `/document.html` working exactly as before (it is the no-JS / direct-link fallback) but refactor its `document.js` so the markdown/raw rendering lives in a shared function the modal module can reuse. No behavioural change to the standalone page.
## Work
1. `frontend/assets/document.js` — split the current inline renderer into an exported `renderDocument(doc, { titleEl, metaEl, contentEl })` function (the escape-first contract: markdown → `renderMarkdown` into a `.doc-md` div; other formats → `<pre class="doc-raw">` via `textContent`; badges via `textContent`). The page's existing `load()` IIFE now calls `renderDocument` with the page's `#doc-title`, `#doc-meta`, `#doc-content` elements. Everything else in `document.js` (query-param parsing, `back` target, not-found card, shared header wiring, New Chat button, `mainEl.focus()`) is **unchanged**.
2. `frontend/assets/document-modal.js` (new) — imports `renderDocument` from `./document.js`. Owns `openDocumentModal(source, path, triggerEl)` and `closeDocumentModal()` (see task 02). On open it fetches `/api/documents/content` and calls `renderDocument(doc, { titleEl: #doc-modal-title, metaEl: #doc-modal-meta, contentEl: #doc-modal-content })`. It also sets `#doc-modal-open.href` to the `/document.html?source=…&path=…` URL.
3. `frontend/index.html` — load `document-modal.js` as a module (add `<script type="module" src="/assets/document-modal.js"></script>` alongside the existing `app.js` module script). `index.html` already loads `markdown.js` as a classic script (needed by `renderDocument`).
4. `frontend/sources.html` — load `document-modal.js` as a module (it needs `document.js`'s `renderDocument`, so both `document.js` and `document-modal.js` must be module scripts; `markdown.js` classic script stays). The Sources page currently loads `sources.js` as a module; add the modal module script next to it.
5. Verify the no-CDN integration test (`tests/integration/test_api.py::test_index_html_served_locally`) still passes — the new module scripts are same-origin `<script src>`, so they satisfy the "local asset" rule. If the test counts script tags, update the expected count.
## ASSUMPTIONS
- `renderDocument` depends on `renderMarkdown` (from `markdown.js`), which is a classic script — so `document.js` (module) importing nothing but using the global `renderMarkdown` is fine, and `document-modal.js` (module) importing `renderDocument` from `document.js` also relies on the global `renderMarkdown` being present. Both pages load `markdown.js` before the module scripts (hoisting guarantees module scripts run after classic scripts already on the page).
- The standalone page's `document.js` no longer needs to be a module for its own rendering — but it stays a module because it imports `header.js` (shared header). Keep it a module.
## Testing & Quality
- No unit/integration test for the refactor itself.
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
## Completion Criteria
- [ ] `/document.html?source=…&path=…` still renders title/meta/content exactly as before (md in `.doc-md`, raw in `<pre.doc-raw>`).
- [ ] The not-found state, `back` target, shared header, and New Chat button on `/document.html` are unchanged.
- [ ] `document-modal.js` is loaded on `index.html` and `sources.html`; `document.js` exports `renderDocument`.
- [ ] No-CDN test still passes (new scripts are same-origin).
- [ ] No console errors on any of the three pages.
@@ -1,33 +0,0 @@
# Task 04 — E2E story suite (modal) + regressions
**Phase:** `26_document_modal_viewer` · **Source:** `TODO.md:4 — "New documents should open in an almost-fullscreen modal, not in a new page"`
**Story:** `.agent/user_stories/document-modal.md`
## Objective
Rewrite the phase-10 E2E suite to assert the **modal** contract (open in a modal on the same page, no new tab; close on button/Escape/backdrop; dark theme; no CDN; a11y frame), and confirm the standalone `/document.html` page still works.
## Work
1. `tests/e2e/test_document_viewer.py` — rewrite for the modal contract (the seeding harness from the phase-10 file — `_import_fixtures` / `_reset_db` / `_run_in_thread` — stays identical; only the assertions change):
- `test_source_chip_opens_modal` — from the chat page, ask the QUESTION, wait for the `kubernetes.md` source chip, click it (no `target=_blank` click → `expect_popup`); assert the modal `.doc-modal` is visible, NOT hidden; `#doc-modal-title` = "Kubernetes Homelab Cluster"; `#doc-content`/`.doc-md` present; content text "Talos Linux on three nodes". Assert the page URL is unchanged (still `/`).
- `test_sources_row_opens_modal` — log in, find the `gitlab-compose.yaml` row link, click it; assert the modal is open with the yaml rendered in `<pre.doc-raw>` containing "gitlab/gitlab-ce:17.2.1-ce.0", mono font.
- `test_modal_closes_on_button_escape_and_backdrop` — open the modal, click `#doc-modal-close` → hidden; re-open, click backdrop → hidden; re-open, press Escape → hidden.
- `test_modal_focus_and_a11y` — on open, focus is inside `#doc-modal-content`; the panel has `role="dialog"` + `aria-modal="true"`; the close button has `aria-label`.
- `test_modal_xss_safe` — seed an XSS fixture doc, open via modal, assert the `<script>` shows as escaped text and no dialog fires (same as the phase-10 test but inside the modal).
- `test_standalone_page_still_works` — the phase-10 assertions for `/document.html` (title/content/format badge, not-found state, dark theme, no-CDN, a11y frame, `#doc-content .doc-md` ≤ 736px) are **kept** — the dedicated page must not regress.
- `test_modal_theme_and_no_cdn` — dark theme (document background `rgb(10,14,23)`), and the modal panel uses Phase-08 surface colour.
2. `tests/integration/test_api.py` — if the no-CDN test counts `<script>` tags on `index.html` / `sources.html`, bump the expected count to include `document-modal.js` (and confirm `document.html` count is unchanged).
3. Regressions to run green in isolation after the change: `test_document_back_navigation.py` (source chips now open a modal; verify the back-navigation story doesn't assert a new tab — if it does, adapt), `test_header_consistency.py` (new module scripts don't disturb the header), `test_smoke.py`.
4. `.agent/user_stories/document-modal.md` — write the story file mapping the modal behaviour to the E2E scenarios above.
## ASSUMPTIONS
- The phase-10 `expect_popup` calls are removed (no new tab); the modal opens in-page.
- The standalone page test is kept to guard the no-JS / direct-link fallback.
## Testing & Quality
- E2E: `tests/e2e/test_document_viewer.py` rewritten — the story gate, green **in isolation** (prereq `podman compose up -d db`).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` green in isolation.
- [ ] `test_document_back_navigation.py`, `test_header_consistency.py`, `test_smoke.py` green in isolation (adapted if they asserted a new tab).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,46 +0,0 @@
# Task 01 — Steering update endpoint
**Phase:** `27_global_tuning` · **Source:** `TODO.md:3 — "Add a way to add 'global tuning' without having a chat to reply to. Also previous tunes should be editable."`
**Story:** `.agent/user_stories/global-tuning.md`
## Objective
Add a `PUT /api/steering/{note_id}` endpoint so an existing steering note can be updated in place (the chat-page + header UI currently support create + delete only).
## Work
1. `app/schemas.py` — add `SteeringNoteUpdate(BaseModel)`:
```python
class SteeringNoteUpdate(BaseModel):
"""``PUT /api/steering/{id}`` body: a new tuning instruction."""
note: str = Field(min_length=1, max_length=2000)
@field_validator("note", mode="before")
@classmethod
def _trim_note(cls, v): return v.strip() if isinstance(v, str) else v
```
Mirror `SteeringNoteIn`'s trim-before-length-constraint behaviour so an empty/whitespace body is a 422.
2. `app/api/steering.py` — add the route:
```python
@router.put("/{note_id}", response_model=SteeringNoteOut)
def update_steering_note(note_id, payload: SteeringNoteUpdate, db):
row = db.get(SteeringNote, note_id)
if row is None:
raise HTTPException(status_code=404, detail="steering note not found")
row.note = payload.note.strip()
db.commit(); db.refresh(row)
return SteeringNoteOut(id=row.id, note=row.note, created_at=row.created_at)
```
The router already carries `dependencies=[Depends(require_admin)]` (phase 16), so anonymous callers get 403 automatically — no extra guard. Keep the existing `list`/`create`/`delete` routes unchanged.
3. Update the module docstring to name the new `PUT` route and that it reuses the router-level admin dependency (no new auth surface).
## ASSUMPTIONS
- The update reuses the same 1–2000-char, trimmed contract as create (no separate validation policy).
- `created_at` is preserved on update (editing a note doesn't redate it; the list order by newest-first is stable for edits).
- The endpoint is `PUT` (idempotent, full replacement of `note`), matching the "edit" semantics.
## Testing & Quality
- Unit (`tests/unit/test_steering.py`): `update_steering_note` via the router — 200 with updated `note`; 404 unknown id; 422 empty/whitespace/over-2000; and the router-level 403 for anonymous (via `require_admin`).
- Integration: `PUT /api/steering/{id}` → 200 returns the new note; the updated note is then read back by `load_steering_notes` (oldest-first order preserved) and appears in `build_steering_section`; anonymous → 403.
## Completion Criteria
- [ ] `PUT /api/steering/{id} {note}` → 200 with the updated note; `GET /api/steering` reflects the new text and order.
- [ ] Unknown id → 404; invalid body → 422; anonymous → 403.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,52 +0,0 @@
# Task 02 — Tuning page HTML + CSS
**Phase:** `27_global_tuning` · **Source:** `TODO.md:3 — "Add a way to add 'global tuning' without having a chat to reply to. Also previous tunes should be editable."`
**Story:** `.agent/user_stories/global-tuning.md`
## Objective
Create the `/tuning.html` page: a title, a "Add a global tuning note" form, and a list of existing notes each with an inline edit control and a delete control. Follow the Phase-08 tokens, WCAG 2.1 AA, and the standard app frame (landmarks, skip-link, sticky header reuse via `header.js`).
## Work
1. `frontend/tuning.html` (new) — standard app frame:
- `<header class="app-header">` reusing the same markup as `index.html`'s header (brand + nav + New Chat + Sign in/out) so `header.js` wires it identically. The page gains an extra admin-only link:
```html
<a href="/tuning.html" class="nav-link is-active" id="nav-tuning" aria-current="page">Tuning</a>
```
(placed after the Sources link; it is always present but only visible to admins because the whole nav is admin-scoped on non-chat pages — the Sources link is already admin-only, so this is consistent).
- `<main id="main" class="app-main" tabindex="-1">` with a centered `container` column:
- `<h1>Global Tuning</h1>` + a sub-heading explaining every note steers all future answers.
- A create form `#tune-form`: a visually-hidden `<label for="tune-note">` + `<textarea id="tune-note" maxlength="2000" rows="3" placeholder="e.g. be more concise — or: assume I'm on NixOS">` + a submit button `#tune-save` ("Add note").
- A status/announcer `<p class="visually-hidden" id="tune-announcer" role="status" aria-live="polite"></p>`.
- A list `<ul id="tune-list">` (role=list) for existing notes; each `<li class="tuning-note">` holds:
- a `.tuning-note-text` span (textContent, XSS-safe),
- an `#edit` button (`.tuning-edit`, icon + "Edit"),
- a `.tuning-delete` button (icon + "Delete").
- An empty-state `<p id="tune-empty">No tuning notes yet — add one above.</p>` toggled by `tuning.js`.
- `<footer class="app-footer">` as on other pages.
- Load `assets/markdown.js` (classic) and `assets/tuning.js` (module) + the shared header (module).
2. `frontend/assets/styles.css` — add a `--tuning-*` block mirroring the `.steering-*` / `.tune-*` tokens:
- `.tuning-note` — flex row, align-items center, gap; text flexes, actions shrink-0; border-bottom divider.
- `.tuning-note-text` — `--ink`; ellipsis overflow if very long.
- `.tuning-edit`, `.tuning-delete` — icon + label buttons, ≥44px targets, `:focus-visible` ring; edit uses brand colour, delete uses the error colour (`--err-ink`/`--err-line`) consistent with `.steering-delete`.
- `.tuning-note.is-editing .tuning-note-text` — hidden while editing (replaced by the inline form).
- Inline edit form (`.tuning-edit-form`) — a `<textarea class="tuning-edit-input" maxlength="2000">` pre-filled + Save/Cancel, styled like the existing `.tune-form` / `.tune-save` / `.tune-cancel`.
- `#tune-form` + `#tune-note` — styled like the existing composer/`#message-input`; `#tune-save` styled like `.tune-save`.
- `#tune-empty` — `--ink-soft`, centered, italic.
- Ensure the centered column matches the chat/sources width discipline (≥80–90% of container; not a skinny list).
- No `filter: blur`, no CDN, system font stack.
3. Verify no selector collision (grep `.tuning-note`, `#tune-form`, `.tuning-edit-*`).
## ASSUMPTIONS
- The page title is "Global Tuning"; the nav link label is "Tuning" (consistent with the chat header's "Tuning" panel).
- Editing is inline (swap the text for a textarea + Save/Cancel in the same list row) — the owner's "previous tunes should be editable" is satisfied without a separate editor page.
- The list is newest-first (same as the header panel) for consistency.
## Testing & Quality
- No unit/integration test for static CSS/HTML.
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
## Completion Criteria
- [ ] `/tuning.html` renders the title, the create form, the note list, and the empty state.
- [ ] All controls are labeled, ≥44px, `:focus-visible`, contrast ≥4.5:1; landmarks + skip-link present.
- [ ] No selector collision; no CDN tags; the page loads `tuning.js` + `markdown.js`.
- [ ] The admin-only "Tuning" nav link is present in the header.
@@ -1,38 +0,0 @@
# Task 03 — Tuning page CRUD logic
**Phase:** `27_global_tuning` · **Source:** `TODO.md:3 — "Add a way to add 'global tuning' without having a chat to reply to. Also previous tunes should be editable."`
**Story:** `.agent/user_stories/global-tuning.md`
## Objective
Create `frontend/assets/tuning.js` — the single owner of the tuning page's behaviour: load notes, create, edit (inline), cancel, and delete, all announced through a polite live region.
## Work
1. `frontend/assets/tuning.js` (new, module) — mirrors the structure of `app.js`'s steering section but for the standalone page:
- Cache the elements: `#tune-form`, `#tune-note`, `#tune-save`, `#tune-list`, `#tune-empty`, `#tune-announcer`.
- `announce(msg)` — sets `#tune-announcer.textContent` (role=status, aria-live=polite).
- `loadNotes()` — `GET /api/steering`; on `r.ok` render the list, else keep the last rendered list (progressive enhancement). Populates `#tune-list` (role=list) with `<li>` rows; toggles `#tune-empty` (`hidden = notes.length > 0`); each row:
- `.tuning-note-text` span (`textContent`, XSS-safe),
- `.tuning-edit` button (icon + "Edit"),
- `.tuning-delete` button (icon + "Delete", aria-label "Delete tuning note: <note>").
- Create handler (`#tune-form` submit): `POST /api/steering {note}`; 201 → clear the textarea, announce "Tuning note added. Future answers will follow it.", reload; non-2xx → inline error under the form (kept, form not cleared) with the API detail; network error → friendly message. Disable `#tune-save` during the request.
- Edit flow (`tuning-edit` click): swap the row's `.tuning-note-text` span for an inline `.tuning-edit-form` containing a `<textarea class="tuning-edit-input" maxlength="2000">` pre-filled with the note + Save/Cancel; focus the textarea. Track the note id on the form (data attribute).
- Save-edit handler: `PUT /api/steering/{id} {note}`; 200 → replace the form with a `.tuning-saved` status (role=status) + announce "Tuning note updated."; non-2xx → keep the form, show inline error; cancel → revert to the text span.
- Delete handler (`tuning-delete` click): disable the button; `DELETE /api/steering/{id}`; 204/404 → remove the `<li>` from the DOM immediately (optimistic), announce, and if 404 reload; non-2xx → re-enable the button + announce retry.
- Keep the 1–2000-char contract on the client (maxlength on the textareas); the server re-validates.
2. Wire the shared header + whoami gate: `tuning.js` imports `{ initSharedHeader, fetchIsAdmin }` from `./header.js` and at boot awaits `initSharedHeader()` then `loadNotes()` only if `fetchIsAdmin()` is true (anonymous users see the page frame but the list stays empty / the create form 403s gracefully — consistent with the Sources page gate pattern). Actually simpler: the header already hides the "Tuning" nav link for anonymous (it's a nav link like Sources); but a direct anonymous URL should still be safe — `loadNotes()` swallows non-2xx, and the create form 403s. So the page is anonymous-safe without a hard gate.
## ASSUMPTIONS
- The edit is inline in the same list row (no separate editor page) — the owner's "editable" is satisfied.
- Optimistic delete (remove the `<li>` before the server confirms) matches the chat page's `deleteSteeringNote` UX.
- The page is anonymous-safe: the list won't render for anonymous (the create/delete calls 403), and the nav link is hidden for anonymous.
## Testing & Quality
- No unit/integration test (frontend-only).
- Coverage: frontend-only; the >90% `app/` gate is unaffected.
## Completion Criteria
- [ ] Create a note on `/tuning.html` → it appears in the list; the announcer announces the change.
- [ ] Edit a note inline → the saved text is updated in the list; the `<tuning>` prompt reflects it (verified via the integration test on the endpoint).
- [ ] Delete a note → the row is removed and the list updates.
- [ ] XSS-safe: note text rendered via `textContent`, never `innerHTML`.
- [ ] Empty state shows when there are no notes.
@@ -1,34 +0,0 @@
# Task 04 — Header "Tuning" button + E2E suite
**Phase:** `27_global_tuning` · **Source:** `TODO.md:3 — "Add a way to add 'global tuning' without having a chat to reply to. Also previous tunes should be editable."`
**Story:** `.agent/user_stories/global-tuning.md`
## Objective
Expose the tuning page via an admin-only header link and add the E2E story suite.
## Work
1. `frontend/tuning.html` header — add the admin-only "Tuning" nav link (is-active) in the `<nav class="app-nav">`, alongside the existing Sources link (also admin-only). The header markup mirrors `index.html`/`sources.html` (brand + nav + New Chat + Sign in/out) so `header.js` wires it identically. The "Tuning" link ships visible on the tuning page (it's the current page); on other pages the admin-only nav links are revealed by `header.js` whoami, consistent with Sources.
2. No change to `app.js`/`sources.html` nav needed — the tuning page is reached from its own header link. (The chat-page "Tune" button and header panel from phase 15 remain unchanged.)
3. `tests/e2e/test_global_tuning.py` (new — the story gate). Reuse the seeding harness pattern from `test_steering.py` / `test_document_viewer.py` (`_import_fixtures` / `_reset_db` / `_run_in_thread`) so the endpoint-under-test is exercised against a seeded KB. Test → mapping (Playwright Mapping Rule):
1. `test_create_note_without_chat` — log in, go to `/tuning.html`, type a note, submit; assert it appears in `#tune-list` and the announcer announced it. No chat turn was made.
2. `test_edit_note_inline` — create a note, click "Edit", change the text, Save; assert the list shows the new text and the edit form is gone.
3. `test_delete_note` — create a note, delete it; the row is removed and `#tune-empty` shows again.
4. `test_edit_note_steers_answer` — create a note via `/tuning.html`, then go to the chat, ask the QUESTION, and assert the note's marker leaks into the answer (same echo trick `test_steering.py` uses: `(tuning: <first note line>)`), proving the edited note is read into the system prompt.
5. `test_tuning_page_a11y_and_no_cdn` — landmarks, skip-link, labeled controls, ≥44px targets, `:focus-visible`; every `script[src]`/`link[href]` is same-origin or `data:`; dark theme.
6. `test_anonymous_cannot_manage` — anonymous `/tuning.html`: the list is empty, and a scripted `PUT /api/steering/{id}` returns 403 (or the create form 403s).
4. Regressions to run green in isolation: `test_steering.py` (the chat-page Tune button + panel still work — create + delete), `test_header_consistency.py` (the tuning page's header is consistent; the new nav link doesn't break the height/consistency assertions on the other pages), `test_document_viewer.py` (unrelated — modal still works), `test_smoke.py`.
5. `.agent/user_stories/global-tuning.md` — write the story file.
## ASSUMPTIONS
- The tuning page's "Tuning" nav link is admin-only (consistent with the Sources link — the catalog and tuning are admin-only).
- The chat-page "Tune" button + header panel are **not** removed (nothing regresses); the tuning page is the primary global manager, the button is a quick-add affordance.
- The edit-steers-answer test reuses the mock-LLM echo behaviour already established in `test_steering.py`.
## Testing & Quality
- E2E: `tests/e2e/test_global_tuning.py` — the story gate, green **in isolation** (prereq `podman compose up -d db`).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_global_tuning.py -v --no-cov` green in isolation.
- [ ] `test_steering.py`, `test_header_consistency.py`, `test_document_viewer.py`, `test_smoke.py` green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,37 +0,0 @@
# Task 01 — Settings: `BOR_GIT_SOURCES` + `BOR_SOURCES_DIR`
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "We shouldn't be hard-coding Homelab and Deployments. Instead, a list of git links should be specified."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Add the two new settings that drive the git-based source flow: the comma-separated list of git URLs and the dedicated local clone location.
## Work
1. `app/config.py` — in the "Import scope" section (near `import_extensions`) add:
- `git_sources: str = ""` — comma-separated git repository URLs (no default; when empty, `import_docs` falls back to `--source` / the old `DEFAULT_SOURCES`). Document: "list of git repo URLs to clone/pull into `sources_dir` before indexing (phase 28); empty means no git sources".
- `sources_dir: str = "~/bor-sources"` — the dedicated local directory the repos are cloned/pulled into (phase 28). Document: "where `import_docs` clones/pulls `git_sources` repos (expanded ~ via `Path.expanduser`)".
- Add a property `git_source_list` returning the non-empty, stripped URLs (list[str]) — used by the import script.
2. `tests/unit/test_config.py` — add tests:
- `git_sources` default is `""`; `git_source_list` returns `[]` when empty.
- `git_sources` env override parses a comma-separated list (whitespace trimmed, empties dropped).
- `sources_dir` default is `~/bor-sources` (raw string, not expanded in the setting — expansion happens in the script).
3. `.env.example` — document both under an "Import scope / git sources" comment block:
```
# --- Import sources (git; phase 28) ---
# BOR_GIT_SOURCES=https://github.com/user/homelab.git,https://github.com/user/deployments.git
# BOR_SOURCES_DIR=~/bor-sources
```
## ASSUMPTIONS
- `git_sources` is empty by default (backwards-compatible: the `import_docs` script keeps its `--source` default until the owner sets `BOR_GIT_SOURCES`).
- URLs are stored raw (no parsing of scheme/auth in the setting) — parsing happens in `git_sync.py`.
- `sources_dir` is stored as a raw string; `Path.expanduser()` is applied in the script (so the setting stays env-agnostic and testable).
## Testing & Quality
- Unit: `tests/unit/test_config.py` additions.
- Coverage: the new setting is exercised by the unit test; the >90% `app/` gate is maintained.
## Completion Criteria
- [ ] `Settings().git_source_list` returns `[]` by default and the parsed list when `BOR_GIT_SOURCES` is set.
- [ ] `sources_dir` defaults to `~/bor-sources`.
- [ ] `.env.example` documents both vars.
@@ -1,67 +0,0 @@
# Task 02 — Git clone/pull utility
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Create `scripts/git_sync.py` with a single `clone_or_pull(url, dest)` function that clones a repo (shallow, first run) or fast-forwards it (subsequent runs), returning the destination path. This is the only place `git` is invoked.
## Work
1. `scripts/git_sync.py` (new) — stdlib `subprocess` only (A11: no new packages):
```python
"""Git source sync for import_docs (phase 28).
clone_or_pull(url, dest) clones ``url`` into ``dest`` (shallow, depth 1)
the first time, or fast-forwards an existing checkout with ``git pull
--ff-only`` on subsequent runs. Auth is whatever the URL/SSH config
supplies — no credentials are stored here.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
class GitSyncError(RuntimeError): ...
def clone_or_pull(url: str, dest: Path | str) -> Path:
dest = Path(dest)
if not dest.exists() or not (dest / ".git").exists():
dest.parent.mkdir(parents=True, exist_ok=True)
_run(["git", "clone", "--depth", "1", url, str(dest)], cwd=dest.parent)
else:
_run(["git", "pull", "--ff-only"], cwd=dest)
return dest
def _run(argv, cwd):
try:
proc = subprocess.run(argv, cwd=cwd, capture_output=True, text=True)
except FileNotFoundError:
raise GitSyncError("git was not found on PATH — install git and retry")
if proc.returncode != 0:
raise GitSyncError(f"git {' '.join(argv[1:])} failed (exit {proc.returncode}): {proc.stderr.strip()}")
return proc.stdout
```
- `--depth 1` clone (fast, and the KB is re-imported incrementally anyway).
- `--ff-only` pull (refuses to merge unrelated histories — a broken checkout fails loudly rather than producing a dirty index).
- `GitSyncError` carries the `git` stderr so the caller can name the failing repo + reason.
- Auth: nothing special — a `https://…` URL uses the OS credential helper / prompts; an `git@host:repo.git` URL uses the machine's SSH key. Document this in the module docstring.
2. `tests/unit/test_git_sync.py` (new):
- `clone_or_pull` **clones** when the dest has no `.git` — verify by monkeypatching `subprocess.run` to a fake that records the argv and returns `returncode=0`; assert `["git","clone","--depth","1",url,str(dest)]` was called and `dest` returned.
- `clone_or_pull` **pulls** when `.git` exists — monkeypatch; assert `["git","pull","--ff-only"]` called.
- `clone_or_pull` raises `GitSyncError` on non-zero exit, carrying the stderr text.
- `clone_or_pull` raises `GitSyncError("git was not found…")` when `subprocess.run` raises `FileNotFoundError`.
- `dest.parent` is created before clone (assert the fake saw a parent that `mkdir` would create — or test the `mkdir` call path directly).
## ASSUMPTIONS
- `--depth 1` shallow clone is sufficient (the KB is re-imported incrementally; no need for full history).
- `--ff-only` pull is the right policy (refuse merges — a dirty/broken checkout fails loudly).
- Auth is delegated to the machine (SSH key / credential helper); no secrets are stored in code or `.env`.
- `git` CLI is assumed present (standard on homelab machines; a clear error is raised otherwise).
## Testing & Quality
- Unit: `tests/unit/test_git_sync.py` (clone vs pull dispatch, error propagation, missing-git error).
- Coverage: the new `scripts/git_sync.py` is fully covered.
## Completion Criteria
- [ ] `clone_or_pull` clones a fresh repo and pulls an existing one (verified via monkeypatched `subprocess`).
- [ ] A failing `git` call raises `GitSyncError` with the stderr; a missing `git` raises `GitSyncError` naming git.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,49 +0,0 @@
# Task 03 — `import_docs` resolves git sources → local dirs
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Rewire `scripts/import_docs.py` so that, when `BOR_GIT_SOURCES` is set, it clones/pulls each repo into `BOR_SOURCES_DIR/<name>/` and indexes the resulting directories — while keeping `--source <path>` overriding for manual local directories.
## Work
1. `scripts/import_docs.py`:
- Import `clone_or_pull` from the sibling module: `from scripts.git_sync import clone_or_pull` (or, since `main()` runs as `python -m scripts.import_docs`, a top-level `from git_sync import clone_or_pull` works — `scripts/` is on `sys.path`).
- Keep `DEFAULT_SOURCES` as the fallback for the no-`--source` + no-`BOR_GIT_SOURCES` case (backwards-compatible).
- New resolution logic in `main()`:
```python
settings = get_settings()
sources = _resolve_sources(args.source, settings)
```
where `_resolve_sources`:
- If `args.source` is given → expanduser each and return (unchanged manual behaviour; `--source` wins).
- Else if `settings.git_source_list` is non-empty → for each URL, `clone_or_pull(url, Path(settings.sources_dir).expanduser() / repo_name(url))`; collect the dest dirs; return them. Any `GitSyncError` propagates (the script exits non-zero naming the failing repo — see below).
- Else → return the old `DEFAULT_SOURCES`.
- `repo_name(url)` — derive a directory name from the URL: strip a trailing `.git`, take the basename after the last `/` (or `:` for scp-style `git@host:repo.git`). Fall back to a slug of the URL if no basename.
- Before importing, log which dirs are being imported (so the operator sees the cloned paths). The existing "source dir not found" warning still applies if a clone left an empty dir.
2. Keep the existing `--prune` / `--limit` flags and the summary print unchanged.
3. Exit code: if any `GitSyncError` is raised, let it propagate to a top-level `except` that prints `import_docs: git sync failed: <reason>` to stderr and returns 1 **before** importing anything (so a bad repo doesn't silently import partial junk). Structure:
```python
try:
sources = _resolve_sources(...)
except GitSyncError as e:
print(f"import_docs: git sync failed: {e}", file=sys.stderr)
return 1
```
## ASSUMPTIONS
- `--source` always wins over `BOR_GIT_SOURCES` (explicit CLI flag beats env).
- When `BOR_GIT_SOURCES` is set, `--source` is ignored (only one source mode at a time) — document this.
- `BOR_SOURCES_DIR` defaults to `~/bor-sources`; each repo is a subdirectory named after the repo.
- A repo that fails to clone/pull aborts the whole run (no partial import) — the operator fixes the URL and re-runs; the already-cloned repos are left on disk and will be pulled on the next run.
- The `documents.source` column will be the repo directory name (e.g. `homelab`), matching the current `source=root.name` behaviour in the importer.
## Testing & Quality
- Integration: `tests/integration/test_import_docs_git.py` (new) — monkeypatch `clone_or_pull` to a fake that creates a temp dir with a fixture `.md` file and returns it; assert `import_docs.main(["--source"])`-equivalent resolution picks the cloned dir and that `import_sources` is called with it. Also assert a `GitSyncError` from the fake → exit code 1 and no import attempt.
- Coverage: the new `main()` resolution branch is covered.
## Completion Criteria
- [ ] `BOR_GIT_SOURCES` set → `import_docs` clones/pulls each repo into `BOR_SOURCES_DIR/<name>/` and imports them.
- [ ] `--source <path>` still imports that manual directory (unchanged).
- [ ] A failing git sync → non-zero exit, message naming the repo, no partial import.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,37 +0,0 @@
# Task 04 — Integration test + docs
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Add the integration test that exercises the full `import_docs` git-resolution path with a mocked `clone_or_pull`, and finalise the docs.
## Work
1. `tests/integration/test_import_docs_git.py` (new) — drive `scripts.import_docs.main(argv)` end to end with the git flow mocked:
- Monkeypatch `scripts.import_docs.clone_or_pull` (import the name into the module's namespace after importing it) so it **creates** a temp directory containing a fixture `.md` file and returns that path (simulating a clone/pull landing real content).
- Set `settings.sources_dir` to a `tmp_path`-based dir (via `monkeypatch.setattr` on the settings or by patching `get_settings`).
- Call `main([])` with `BOR_GIT_SOURCES` patched to a single URL.
- Assert: `clone_or_pull` was called once with the URL; the returned dir was passed to `import_sources`; the summary printed includes the fixture file (added > 0).
- Negative case: monkeypatch `clone_or_pull` to raise `GitSyncError`; assert `main([])` returns `1` and prints a message naming the repo, and that `import_sources` was **not** called.
- Manual override case: `main(["--source", str(tmp_path)])` imports the manual dir and does **not** call `clone_or_pull`.
- Use the same `FakeEmbedder`-style approach the importer tests use so no real LLM is needed (the importer's two-phase embed is duck-typed; pass a fake `llm` if `main` allows injection, or let `import_docs` build the real `LLMClient` but patch `import_sources` to capture its `sources` arg and short-circuit). Simpler: monkeypatch `scripts.import_docs.import_sources` to a fake that records the `sources` list and returns a trivial `ImportSummary` — this isolates the git-resolution logic from the whole embed pipeline.
2. `README.md`:
- Section 5 (Import your knowledge base): document the two source modes — (a) git sources via `BOR_GIT_SOURCES` (clone/pull into `BOR_SOURCES_DIR`), (b) `--source <path>` for manual directories. Keep the `~/Homelab + ~/Deployments` note as the *previous* default, now replaced by `BOR_GIT_SOURCES`.
- Add a short "Git-based sources" subsection: set `BOR_GIT_SOURCES` (comma-separated URLs) + `BOR_SOURCES_DIR`; `import_docs` clones (first run) or pulls (subsequent runs) each repo and indexes them; `--source` overrides; a failed sync aborts the run.
- The "Clicking a chip opens that document in a new tab" bullet (line ~69) is now stale — update to "opens in an almost-fullscreen modal" (phase 26).
3. `.env.example` — already updated in task 01; double-check the comment block is present and correct.
4. `.agent/user_stories/git-sources.md` — write the story file.
## ASSUMPTIONS
- The integration test mocks `clone_or_pull` + `import_sources` so it runs without network, without `git`, and without a real embed endpoint — it isolates the resolution logic.
- The README keeps a migration note for owners currently relying on the hardcoded `~/Homelab`/`~/Deployments` default (set `BOR_GIT_SOURCES` to the same two repos).
## Testing & Quality
- Integration: `tests/integration/test_import_docs_git.py`.
- Coverage: the new `main()` branch covered.
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_import_docs_git.py -v --no-cov` green.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] README documents both source modes; the "opens in a new tab" bullet updated to "modal".
@@ -1,40 +0,0 @@
# Phase 29 — Global Tuning nav link
**Source:** `.agent/phases/complete/27_global_tuning` — the `/tuning.html` page and its create/edit/delete machinery shipped, but the "Tuning" header link that was supposed to *expose* the page (`27` task 04: *"Expose the tuning page via an admin-only header link"`) only exists on `tuning.html` itself. It is not present on the Chat or Sources pages, so there is no way to reach `/tuning.html` from anywhere except typing the URL.
**Story:** `.agent/user_stories/global-tuning.md`
**Context:** The shared header (phase 19) renders `[Chat, Sources — admin only]` in `<nav class="app-nav">` on the Chat and Sources pages. `frontend/assets/header.js` already contains logic to reveal an admin-only `#nav-tuning` link for admins (lines 69–70) — but that element is never in `index.html`/`sources.html` markup, so the reveal is a no-op. This phase adds the missing markup so the link the JS already handles actually exists.
## Objective
Make the Global Tuning page reachable by adding the admin-only **"Tuning"** nav link to the shared headers of the **Chat** (`index.html`) and **Sources** (`sources.html`) pages — hidden by default, revealed for admins by the existing `header.js` whoami gate (the exact same pattern as the Sources link). No backend, schema, or API change.
## Dependencies
- `19_shared_header` (complete) — the `<nav class="app-nav">` contract, the shared header markup/ids, and the 64px header-height rule on the Chat and Sources pages.
- `16_admin_auth` (complete) — the `/api/whoami` gate + `header.js` `fetchIsAdmin()` that reveals admin-only nav links.
- `27_global_tuning` (complete) — the `/tuning.html` page and `PUT/POST/DELETE /api/steering` endpoints that this link points to.
## Tasks
1. `01_tuning_links_markup.md` — add the admin-only `#nav-tuning` link to the Chat (`index.html`) and Sources (`sources.html`) headers, hidden by default.
2. `02_tuning_nav_e2e_and_regression.md` — add the story E2E suite (`test_tuning_nav_link.py`) + verify no regressions in the header/shared-header/no-CDN suites and commit.
## Testing & Quality
- Unit/integration: no new backend logic (frontend-only markup). The existing no-CDN integration test (`tests/integration/test_api.py::test_index_html_served_locally`) must still pass — the new link is same-origin `<a>`, no new tags.
- Coverage: frontend-only; the `app/` >90% gate is unaffected (unchanged).
- E2E (mandatory, A16): `tests/e2e/test_tuning_nav_link.py` — the story gate, run in isolation.
## Completion Criteria
- [ ] On the **Chat** page (`/`), a signed-in admin sees a **"Tuning"** link (next to Chat/Sources) that navigates to `/tuning.html`; an anonymous visitor does **not** see it (ships `hidden`, `header.js` reveals only for admin).
- [ ] Same on the **Sources** page (`/sources.html`).
- [ ] The link is absent from the anonymous DOM-reveal (verified via `test_anonymous` behavior); header markup stays valid and semantic.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged (no backend change).
- [ ] Existing header suites stay green in isolation: `test_header_consistency.py` (64px height + nav structure unchanged), `test_shared_header.py`, `test_smoke.py`, `test_global_tuning.py`.
- [ ] `uv run pytest tests/e2e/test_tuning_nav_link.py -v --no-cov` green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean (no Python change, but run the gate).
- [ ] UI Structure Check (AGENTS.md rule 5): the link is a labeled, focus-visible `<a>`; header landmarks/contrast unchanged; no CDN.
- [ ] One `--no-gpg-sign` commit staging only this phase's files; `.agent/phases/todo/29_tuning_nav_link/` moved to `.agent/phases/complete/`.
## Locked decisions
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages; the link is a same-origin `<a href="/tuning.html">`.
- **A10 untouched** — no new endpoint or auth surface; the link reuses the existing `#nav-sources` reveal path (`header.js` already gates `#nav-tuning` on `fetchIsAdmin()`).
- **No schema / migration** — purely a markup addition.
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
- **Scope boundary** — only the Chat and Sources pages (the two that share the standard `app-nav` with the Sources link). The viewer page (`document.html`) uses a different `.doc-header-actions` header variant and the login page is the auth gate; both stay out of scope for this fix.
@@ -1,41 +0,0 @@
# Task 01 — Tuning nav link on the Chat and Sources pages
**Phase:** `29_tuning_nav_link` · **Story:** `.agent/user_stories/global-tuning.md`
## Objective
Add the admin-only **"Tuning"** nav link to the shared headers of the **Chat** (`index.html`) and **Sources** (`sources.html`) pages, so `/tuning.html` is reachable from both.
## Work
Add one link to each page, inside the existing `<nav class="app-nav" aria-label="Primary">`, **immediately after** the Sources link, mirroring the Sources pattern exactly.
1. `frontend/index.html` (Chat page) — after the `#nav-sources` link, add:
```html
<!-- Phase 29: the Global Tuning link is admin-only (owner permission
2026-08-25) — hidden by default, header.js reveals it once
whoami says admin, exactly like the Sources link above. Points at
the standalone /tuning.html manager (phase 27). -->
<a href="/tuning.html" class="nav-link" id="nav-tuning" hidden>Tuning</a>
```
2. `frontend/sources.html` (Sources page) — same link, placed after its `#nav-sources` link:
```html
<!-- Phase 29: the Global Tuning link is admin-only (owner permission
2026-08-25) — hidden by default, header.js reveals it once
whoami says admin, exactly like the Sources link above. -->
<a href="/tuning.html" class="nav-link" id="nav-tuning" hidden>Tuning</a>
```
Rules for both edits:
- Keep `hidden` by default (anonymous-safe ship-hidden, identical to `#nav-sources`).
- Do **not** add `is-active` / `aria-current="page"` — those are "current page" visual states; this link is always hidden on the Chat/Sources pages, and `tuning.html` already sets `is-active` on its own link.
- Preserve surrounding markup and indentation so the header layout and the pinned 64px height (phases 12/19) are unchanged. The link is inline in the nav, so height is unaffected.
- **Do not modify `frontend/assets/header.js`.** It already handles this element (lines 69–70: `const navTuning = document.querySelector("#nav-tuning"); if (navTuning) navTuning.hidden = !admin;`). The link now exists for that existing reveal code to act on. If, against expectation, that reveal code were absent, add it next to the `navSources` reveal — but it is present, so leave `header.js` untouched.
## Testing & Quality
- No new unit/integration logic (frontend markup only).
- Coverage: frontend-only; `app/` coverage unaffected.
- This is a same-origin `<a>` — the existing no-CDN integration test (`tests/integration/test_api.py::test_index_html_served_locally`) still passes.
## Completion Criteria
- [ ] `index.html` and `sources.html` each contain an admin-only `#nav-tuning` `<a href="/tuning.html">` placed after `#nav-sources`, `hidden` by default, without `is-active`.
- [ ] Header structure/height unchanged (confirmed in task 02 via `test_header_consistency.py` / `test_shared_header.py`).
- [ ] Full test suite green; no behavior change in completed phases.
@@ -1,53 +0,0 @@
# Task 02 — Tuning nav-link E2E suite, regression checks, and commit
**Phase:** `29_tuning_nav_link` · **Story:** `.agent/user_stories/global-tuning.md`
## Objective
Prove the new admin-only "Tuning" link works (and is hidden for anonymous users) on the Chat and Sources pages, confirm no header/no-CDN regressions, and commit.
## Work
1. **New story E2E suite** — create `tests/e2e/test_tuning_nav_link.py`, following the harness already used in `tests/e2e/test_global_tuning.py`:
- Reuse `from e2e.auth_helpers import login` (real form login on `/login.html`; `login(page, app_url, next=...)` redirects to `next`) and the `db_ready` fixture. No KB seeding is needed for the link-visibility tests (the link is pure header markup gated by `/api/whoami`), so a light module fixture that just brings up the app + mock LLM is enough — mirror `test_global_tuning.py`'s `mock_llm`/`app_url`/`db_ready` params and its `_reset_db` if you want a clean DB.
- The visible `#nav-tuning` on an admin page is the sync point (exactly the Sources-link contract): `header.js` reveals it only after whoami says admin.
- Write these tests (Playwright `expect`, sync API):
1. `test_admin_sees_tuning_link_on_chat` — `login(page, app_url, next="/")`; on the Chat page assert `#nav-tuning` is visible and its `href` is `/tuning.html`; click it and assert the URL navigates to `/tuning.html` (and `#nav-tuning` becomes `is-active`, matching the page's own link).
2. `test_admin_sees_tuning_link_on_sources` — `login(page, app_url, next="/sources.html")`; assert `#nav-tuning` is visible on the Sources page and clicking it navigates to `/tuning.html`.
3. `test_anonymous_hides_tuning_link_on_chat_and_sources` — visit `/` and `/sources.html` **without** logging in; assert `#nav-tuning` is `hidden` on both pages, `#sign-in-link` visible, `#nav-sources` also hidden (consistent with the existing Sources gate).
4. `test_tuning_link_a11y_and_no_cdn` — on the admin Chat page, assert the link is a labeled, focus-visible `<a>` (tab through the header, focus lands on it), and assert every `script[src]`/`link[href]` on the page is same-origin or `data:` (no CDN) — reuse the same `evaluate` scan `test_global_tuning.py::test_tuning_page_a11y_and_no_cdn` uses.
2. **Regression verification** — after the new suite is green, run the header- and page-sensitive suites **in isolation** to prove no regression from the added markup/height:
```bash
uv run pytest tests/e2e/test_header_consistency.py -v --no-cov
uv run pytest tests/e2e/test_shared_header.py -v --no-cov
uv run pytest tests/e2e/test_smoke.py -v --no-cov
uv run pytest tests/e2e/test_global_tuning.py -v --no-cov
uv run pytest tests/integration/test_api.py::test_index_html_served_locally -v --no-cov
```
Fix anything the new link breaks (it should not — the link is inline in the existing nav and does not change header height).
3. **Full gate** — run the whole suite with coverage and confirm `app/` coverage TOTAL is **unchanged** (this phase adds no backend logic):
```bash
uv run pytest
uv run pytest --cov=app --cov-report=term-missing # TOTAL must not drop
```
4. **Lint / types** — `uv run ruff check . && uv run pyright` (no Python change expected, but run the gate).
5. **Commit** — one atomic `--no-gpg-sign` commit staging only this phase's files (the new E2E file + any committed edits), then move the phase directory to `.agent/phases/complete/`:
```bash
git add -f .agent/phases/todo/29_tuning_nav_link tests/e2e/test_tuning_nav_link.py
git commit --no-gpg-sign -m "feat(ui): expose Global Tuning from Chat + Sources headers (admin-only Tuning nav link)"
git mv .agent/phases/todo/29_tuning_nav_link .agent/phases/complete/29_tuning_nav_link
```
(`.agent/` is gitignored — use `git add -f` / `git mv -f` as needed.)
## Testing & Quality
- E2E (mandatory, A16): `tests/e2e/test_tuning_nav_link.py` green **in isolation**:
```bash
uv run pytest tests/e2e/test_tuning_nav_link.py -v --no-cov
```
- No new unit/integration logic (frontend markup only); `app/` coverage unchanged.
## Completion Criteria
- [ ] Admin sees a visible **"Tuning"** link on the Chat and Sources pages → clicking it opens `/tuning.html`.
- [ ] Anonymous visitors see **no** Tuning link on either page (ships `hidden`, revealed only for admin).
- [ ] Link is labeled, focus-visible; page stays CDN-free.
- [ ] `test_tuning_nav_link.py` green in isolation; `test_header_consistency.py`, `test_shared_header.py`, `test_smoke.py`, `test_global_tuning.py` green in isolation; `test_index_html_served_locally` green.
- [ ] `uv run pytest` green; `app/` coverage TOTAL unchanged; `ruff` + `pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase moved to `.agent/phases/complete/`.
@@ -1,26 +0,0 @@
# Task 01 — lite model setting + non-streaming chat()
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "The small model available on aipi.reeseapps.com is 'lite'." (enabler for "we need a small model to analyze non markdown documents")`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Make the aipi **`lite`** model callable from the app: a new model setting plus a non-streaming `LLMClient.chat()` one-shot completion method that the summarizer (task 03) and the phase-31 overview generator will use.
## Work
1. `app/config.py` — add `llm_summary_model: str = "lite"` (env `BOR_LLM_SUMMARY_MODEL`), documented like the other LLM settings.
2. `app/rag/llm.py` — add to `LLMClient`:
- `async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str` — `chat.completions.create(model=model or self.settings.llm_summary_model, messages=…, temperature=0.2, max_tokens=2048, stream=False)`; returns the first choice's `message.content` stripped.
- Raise `LLMError` (wrapped, with the base URL in the message — same style as `chat_stream`) on any transport/HTTP/malformed failure, and on an empty/missing content field (a silent empty summary must never be stored).
3. `.env.example` — document `BOR_LLM_SUMMARY_MODEL` (default `lite`).
4. `README.md` — models section: add `lite` (document summaries — this phase; KB overview in phase 31) next to `turbo`/`embed`.
- ASSUMPTION: the model name is `lite` per the TODO item; single (non-streaming) completion with `temperature=0.2`, `max_tokens=2048` — summaries/outlines are short, so a fixed budget is enough (no new setting).
## Testing & Quality
- Unit: `tests/unit/test_llm_client.py` — extend the existing mock-transport pattern: `chat()` returns content (trimmed); HTTP ≥400 → `LLMError`; empty content → `LLMError`; explicit `model=` overrides the default (`llm_summary_model`).
- Coverage: **>90%** on this task's new/modified code (`app/` TOTAL ≥ pre-change).
## Completion Criteria
- [ ] `Settings().llm_summary_model == "lite"` by default; `BOR_LLM_SUMMARY_MODEL` env override works (config test).
- [ ] `chat()` unit tests green; `uv run ruff check . && uv run pyright` clean.
- [ ] No change to `embed`/`chat_stream` behavior (existing suite green).
@@ -1,23 +0,0 @@
# Task 02 — Migration 0004: summary columns
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "provide a textual summary of those documents with a pointer back to the source" (storage)`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Add the schema for storing a document's summary and for marking the extra summary chunk: `documents.summary TEXT NULL` and `chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE`.
## Work
1. `alembic/versions/0004_summary_columns.py` — new revision (down_revision = the 0003 steering-notes revision, whatever `alembic/versions/` currently heads to):
- upgrade: `op.add_column("documents", sa.Column("summary", sa.Text(), nullable=True))`; `op.add_column("chunks", sa.Column("is_summary", sa.Boolean(), nullable=False, server_default=sa.text("false")))`.
- downgrade: drop both columns.
2. `app/models.py` — `Document.summary: Mapped[str | None] = mapped_column(Text, default=None)` (comment: lite-model summary, phase 30); `Chunk.is_summary: Mapped[bool] = mapped_column(Boolean, default=False)` (comment: summary chunk, position −1, phase 30).
3. `tests/integration/test_migration_0004.py` — mirror `tests/integration/test_migration_0002.py`'s style: upgrade to head → both columns exist, `is_summary` default `false`; downgrade to 0003 → both gone; upgrade again → back (round-trip).
## Testing & Quality
- Integration: the migration test above (real Postgres, as `test_migration_0002.py` does).
- Coverage: models are exercised by the existing model tests; `app/` TOTAL ≥ pre-change.
## Completion Criteria
- [ ] `uv run alembic upgrade head` applies cleanly on the dev DB (and `alembic downgrade -1` + `upgrade head` round-trips).
- [ ] `uv run pytest` green (including all pre-existing migration/importer tests — `is_summary` default keeps old rows valid).
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,30 +0,0 @@
# Task 03 — app/rag/summarizer.py (prompt + lite call + pointer)
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "we need a small model to analyze non markdown documents and provide a textual summary of those documents with a pointer back to the source"`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Create the summarizer module: build the `lite` prompt for one document, call the model (task 01), validate the output, and return the summary text with a **code-deterministic** pointer line back to the source.
## Work
1. `app/config.py` — add `summary_max_chars: int = 12_000` (env `BOR_SUMMARY_MAX_CHARS`): the cap on document content sent to the lite model in one call.
2. `app/rag/summarizer.py` (new) —
- `SUMMARY_MODE = "SUMMARY_MODE"` — marker constant the E2E mock keys on in the system prompt (same convention as `DEFLECT_MODE`).
- `build_summary_prompt(source: str, path: str, content: str, max_chars: int | None = None) -> tuple[str, str]` → `(system, user)`:
- system: `SUMMARY_MODE` + instruction — "Write a 3–6 sentence plain-text summary of this document in natural language. Cover what it configures/defines and its most important values. Do not use markdown. Do not invent anything that is not in the document."
- user: the document content, capped at *max_chars* (default `get_settings().summary_max_chars`); on overflow cut at the cap and append the shared `TRUNCATION_MARKER` (imported from `app.rag.retriever`).
- `async def generate_summary(llm, *, source: str, path: str, content: str) -> str` — calls `llm.chat([{"role":"system",…},{"role":"user",…}], model=llm.settings.llm_summary_model)`; validates non-empty after trim (else raise `LLMError` — the client already does this, but re-assert defensively); appends the deterministic pointer line: `f"\nSource: {source}/{path}"` (the pointer is **never** model-generated).
3. `tests/unit/test_summarizer.py` (new) — fake LLM object (duck-typed `chat` + `settings`):
- prompt: system contains `SUMMARY_MODE`; user == full content when under cap; user truncated + `TRUNCATION_MARKER` when over cap (custom and default cap).
- generation: returned text = model text + pointer line `Source: <source>/<path>`; whitespace model text → `LLMError`; `LLMError` from the client propagates.
- ASSUMPTION: the pointer is the literal line `Source: <source>/<path>` appended by code (the TODO's "pointer back to the source"); the model is told what to summarize but not to write the pointer.
## Testing & Quality
- Unit: the tests in Work step 3.
- Coverage: **>90%** on `app/rag/summarizer.py`.
## Completion Criteria
- [ ] `generate_summary` returns a non-empty summary ending in the deterministic pointer line; all unit tests green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] No app endpoint change yet (pipeline integration is task 05).
@@ -1,39 +0,0 @@
# Task 04 — Importer: generate, store, index summaries (best-effort)
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "a small model to analyze non markdown documents and provide a textual summary… The similarity search will have a higher chance of hitting those summaries than the original document"`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Hook summarization into the import pipeline: every **non-markdown** file gets a lite summary stored on `documents.summary` and indexed as one extra embedded chunk (`is_summary`, position −1) — best-effort, so a lite failure never loses the document.
## Work
1. `app/rag/importer.py`:
- `Embedder` protocol — add `async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str: ...` (task 01's `LLMClient.chat` already satisfies it; the protocol is what tests duck-type).
- `ImportSummary` — new counters `summaries: int = 0` and `summary_errors: int = 0`; include both in the `log()` summary line (`… summaries=%d summary_errors=%d`).
- `_index_file` — **after** the existing doc+chunks commit (so the document is safe):
- If `full_path.suffix.lower()` is **not** in `(".md", ".markdown")`: try
`summary = await generate_summary(llm, source=source, path=rel, content=content)`;
delete any existing `is_summary` chunk of this document (re-import replacement);
add `Chunk(document_id=doc.id, position=-1, content=summary, is_summary=True)`;
`vector = (await llm.embed([summary]))[0]`; set `chunk.embedding = vector`, `doc.summary = summary`; `session.commit()`; `summary.summaries += 1`; log `import: summary source=%s path=%s chars=%d`.
- On `LLMError | EmbeddingError`: `session.rollback()`, `summary.summary_errors += 1`, log `import: summary failed source=%s path=%s — %s`, and **continue** (the document row + content chunks stay committed; `doc.summary` remains NULL).
- Markdown files: no summary, `doc.summary` stays NULL, no `is_summary` chunk.
2. `scripts/import_docs.py` — the final `print` gains `summaries=%d summary_errors=%d` from the `ImportSummary`.
3. `tests/unit/test_importer.py` — extend the existing fake embeder with a `chat` method (deterministic: returns `"Summary of " + first token of content`; raises `LLMError` when the content contains the sentinel word `SUMMARY-BLOWUP`):
- non-md file (e.g. `.yaml`) → after import: `doc.summary` set, exactly one `is_summary` chunk at position −1 with a non-NULL embedding; `summary.summaries == 1`.
- `.md` file → `doc.summary is None`, no `is_summary` chunk, `summaries == 0`.
- fail-soft: content with `SUMMARY-BLOWUP` → document fully indexed (chunks present, embedding set), `doc.summary is None`, `summary_errors == 1`, no exception.
- replacement: re-import the same file with changed content → still exactly **one** `is_summary` chunk (old one deleted), new text.
- `log()` line includes the new counters (existing log-format test updated accordingly).
- ASSUMPTION: the summary chunk sits at `position = -1` (content chunks stay 0-based in order) so chunk ordering and the viewer are undisturbed; only one summary chunk per document at a time.
- ASSUMPTION: "non-markdown" = suffix not in `(.md, .markdown)` — every other A9 format (txt, yaml, yml, json, py) gets a summary.
## Testing & Quality
- Unit: the tests in Work step 3 (reuse the existing test file's session/fake-embedder fixtures).
- Coverage: **>90%** on the modified `app/rag/importer.py`; `app/` TOTAL ≥ pre-change.
## Completion Criteria
- [ ] All new unit tests green; existing importer tests green (protocol change is additive).
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] A manual `uv run python -m scripts.import_docs` run against the dev KB logs `summaries=N` for the non-md docs (observable in the importer log line, PLAN §9).
@@ -1,31 +0,0 @@
# Task 05 — Pipeline: summary hits resolve to the full source document
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — "the chat LLM can retrieve the source pointed to by the summary document (so as part of the pipeline: if retrieval == summary, fetch documents referenced by summary)"`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Make the retrieval + chat pipeline summary-aware: carry `is_summary` through both candidate lists into the fused result, and record in the per-turn log how many of the selected documents were hit via their summary chunk. The context assembly itself is **unchanged** — a summary chunk's parent *is* the source document, and `select_documents` already feeds the full document (A7 revised / phase 24); this task makes that resolution explicit, asserted, and observable.
## Work
1. `app/rag/retriever.py`:
- `RetrievedChunk` — add field `is_summary: bool = False` (documented: True for the lite-model summary chunk, position −1).
- `_vector_candidates` — select `Chunk.is_summary` and pass it into the constructed `RetrievedChunk`s.
- `_LEXICAL_SQL` — add `c.is_summary AS is_summary`; `_lexical_candidates` passes `row.is_summary`.
- `fuse` needs no change (dataclass passthrough) — but assert in tests that the flag survives fusion.
2. `app/api/chat.py`:
- `TurnPlan` — add `summary_hits: int = 0` (count of selected-document hit chunks with `is_summary`).
- `plan_turn` — after the selected-docs decision is known for each branch, compute `summary_hits = sum(1 for c in chunks if c.is_summary and c.document.id in selected_ids)` and store it on the `TurnPlan` (both HIGH and LOW branches).
- Per-turn log line — add `summary_hits=%d` after `fts_hits=%d` (PLAN §9 line extension; record it in the phase's locked decisions). Update any existing test that asserts the log line format verbatim.
- **No change** to `build_high_prompt`/`build_deflect_prompt` inputs or to `select_documents` — the full source document of a summary hit already lands in `<documents>`; task 06's E2E proves it end-to-end.
3. `tests/unit/test_retriever.py` — `is_summary` survives: vector candidates (flag set), lexical candidates (flag set), `fuse` (both a double-hit and a summary-only lexical hit keep the flag; default stays `False` for legacy chunks).
4. `tests/unit/test_chat_gate.py` — `plan_turn`: a summary chunk on a selected top document → `summary_hits == 1`; a summary chunk on a document **outside** the top-N selection → not counted; no summaries → `0` (existing cases unchanged).
## Testing & Quality
- Unit: the tests in Work steps 3–4; existing chat-gate and retriever tests stay green (new field is defaulted).
- Coverage: **>90%** on modified `app/rag/retriever.py` + `app/api/chat.py`; `app/` TOTAL ≥ pre-change.
## Completion Criteria
- [ ] A summary chunk retrieved via vector **or** lexical carries `is_summary=True` through `retrieve()` (unit).
- [ ] The per-turn log line (PLAN §9) now reads `… fts_hits=… summary_hits=… …` and existing log-format tests are updated + green.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] No prompt change: HIGH/LOW prompts byte-identical for summary-less KBs (covered by existing prompt tests).
@@ -1,22 +0,0 @@
# Task 01 — Migration 0005: kb_overview table
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "should be stored somewhere so it can be updated whenever we import new documents"`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
## Objective
Create the storage for the knowledge-base outline: a single-row `kb_overview` table and its SQLAlchemy model.
## Work
1. `alembic/versions/0005_kb_overview.py` — new revision (down_revision = phase 30's 0004):
- upgrade: create table `kb_overview` (`id INTEGER` PK `server_default sa.text("1")`, `content TEXT NOT NULL server_default sa.text("''")`, `updated_at TIMESTAMPTZ NOT NULL server_default=sa.func.now()`).
- downgrade: drop the table.
2. `app/models.py` — `class KbOverview(Base)`: `id: Mapped[int] = mapped_column(Integer, primary_key=True, server_default="1")`, `content: Mapped[str] = mapped_column(Text, server_default="")`, `updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())`. Docstring: single row, lite-generated KB outline, phase 31.
3. `tests/integration/test_migration_0005.py` — same style as `test_migration_0002.py` / `test_migration_0004.py`: upgrade → table exists with the three columns and defaults; downgrade → gone; upgrade again → back.
## Testing & Quality
- Integration: the migration test above (real Postgres).
- Coverage: model exercised by existing model-test patterns; `app/` TOTAL ≥ pre-change.
## Completion Criteria
- [ ] `uv run alembic upgrade head` clean on the dev DB; round-trip with `alembic downgrade -1` + `upgrade head`.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,29 +0,0 @@
# Task 02 — app/rag/overview.py (generator + loader)
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "basic categories of everything that's been read… generated by the lite model… updated whenever we import new documents"`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
## Objective
Create the overview generator module: build the `lite` prompt from the document catalogue, generate the outline, store it in the single row, and expose a cheap loader for the chat path.
## Work
1. `app/config.py` — add:
- `kb_overview_max_chars: int = 4_000` (`BOR_KB_OVERVIEW_MAX_CHARS`) — prompt-section budget (task 03).
- `overview_input_max_chars: int = 40_000` (`BOR_OVERVIEW_INPUT_MAX_CHARS`) — cap on the document list sent to the model.
2. `app/rag/overview.py` (new):
- `KB_OVERVIEW_MODE = "KB_OVERVIEW_MODE"` — marker the E2E mock keys on (same convention as `SUMMARY_MODE` / `DEFLECT_MODE`).
- `build_overview_prompt(rows: Sequence[tuple[str, str, str, str | None]], max_chars: int | None = None) -> tuple[str, str]` → `(system, user)`. Each row is `(source, path, title, summary)`; system = `KB_OVERVIEW_MODE` + instruction ("From the document list below, write a compact plain-text outline of the basic categories and topics this knowledge base covers. Group by source where useful, use `-` bullet lines, at most ~1500 characters, no markdown headings, and no topics not present in the list."); user = one line per doc `source — path — title — {first line of summary or ''}` joined by newlines, capped at *max_chars* (default `overview_input_max_chars`, overflow → shared `TRUNCATION_MARKER`).
- `load_kb_overview(db: Session) -> str` — the single row's `content` (trimmed) or `""` when the row is missing/empty.
- `async def regenerate_overview(llm, session: Session | None = None) -> bool` — load all documents (`source, path, title, summary` ordered by source, path); **zero documents → leave the existing row untouched, return False**; build the prompt; `text = await llm.chat([system, user], model=llm.settings.llm_summary_model)`; upsert the single row (`id=1`, `content=text`, `updated_at=now(UTC)`); commit; log `overview: regenerated docs=%d chars=%d`; return True. On `LLMError`: log `overview: regeneration failed — %s` and return False (previous row stays — see phase locked decisions).
3. `tests/unit/test_overview.py` (new) — fake LLM (duck-typed `chat` + `settings`), in-memory/SQLite session where the existing test infra allows (else a real-DB integration test in the same style as `test_steering.py`):
- prompt: system contains `KB_OVERVIEW_MODE`; user lines carry source/path/title/first summary line; cap truncates + marker.
- `load_kb_overview`: no row → `""`; row present → content.
- `regenerate_overview`: happy path upserts (content + fresh `updated_at`, returns True); zero docs → no DB write, returns False; `LLMError` → previous row unchanged, returns False.
## Testing & Quality
- Unit/integration: the tests in Work step 3.
- Coverage: **>90%** on `app/rag/overview.py`.
## Completion Criteria
- [ ] `regenerate_overview` is idempotent (single row, always id=1) and fail-soft; all tests green.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,30 +0,0 @@
# Task 03 — `<knowledge_base>` section in both prompts + chat wiring
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "The system prompt should inject basic categories of everything that's been read so the agent knows roughly what its knowledge base contains before the rag retrieval returns documents"`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
## Objective
Inject the stored overview into **every** chat turn's system prompt (HIGH and LOW modes) as a budgeted `<knowledge_base>` section — absent row → byte-identical prompts — and record `kb_chars` in the per-turn log line.
## Work
1. `app/rag/prompts.py`:
- `build_kb_section(overview: str, max_chars: int | None = None) -> str` — empty/whitespace → `""`; otherwise `<knowledge_base>\n` + intro line ("The basic categories of everything in this knowledge base (generated at import time):") + the overview content, budgeted at *max_chars* (default `get_settings().kb_overview_max_chars`) with the shared `TRUNCATION_MARKER` for overflow (exact pattern of `build_steering_section`, including its pathological-budget handling).
- `build_high_prompt(documents, notes=None, kb_overview: str | None = None)` and `build_deflect_prompt(titles, notes=None, kb_overview: str | None = None)` — insert the section **between `<relevance>` and the `<tuning>` section** (i.e. order: `<relevance>` → `<knowledge_base>` → `<tuning>` → mode body); with an empty overview the output is byte-identical to today's text in both modes.
2. `app/api/chat.py`:
- Load per turn: `kb_overview = load_kb_overview(db)` next to the steering-notes load (one PK lookup — no LLM call).
- `plan_turn(chunks, settings, notes=None, kb_overview: str | None = None)` — pass it to both prompt builders; `TurnPlan` gains `kb_chars: int = 0` (length of the stored overview text when a non-empty row exists, else 0).
- Per-turn log line (PLAN §9): add `kb_chars=%d` after `tuning=%d`. Update any existing test asserting the log line format verbatim.
3. `tests/unit/test_prompts.py` —
- HIGH: no overview → byte-identical to the pre-phase builder output (build the expected string with `notes=None, kb_overview=None`); with overview → section present, ordered before `<tuning>` when both exist.
- LOW: same pair of assertions (deflection prompt).
- budget: overview longer than `kb_overview_max_chars` → capped + `TRUNCATION_MARKER`.
4. `tests/unit/test_chat_gate.py` — `plan_turn` with an overview: both branches' `system_prompt` contains the section; `TurnPlan.kb_chars` == len(overview); empty overview → `kb_chars == 0` and prompt unchanged.
## Testing & Quality
- Unit: Work steps 3–4; existing steering/prompt/gate tests stay green (new param is defaulted).
- Coverage: **>90%** on modified `app/rag/prompts.py` + `app/api/chat.py`; `app/` TOTAL ≥ pre-change.
## Completion Criteria
- [ ] HIGH and LOW prompts are byte-identical to pre-phase text when no overview row exists (unit-asserted against the exact strings).
- [ ] With a row, both prompts carry the budgeted `<knowledge_base>` section in the locked order; the per-turn log line shows `kb_chars=…`.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,33 +0,0 @@
# Task 04 — import_docs regenerates the overview after a KB-changing import
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — "should be updated whenever we import new documents"`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
## Objective
Wire the "update whenever we import new documents" trigger into the import script: after an import that changed the KB, regenerate the stored overview (best-effort). Phase 32's admin sync button reuses the exact same `regenerate_overview` call.
## Work
1. `scripts/import_docs.py`:
- Restructure `main()`'s single `asyncio.run(import_sources(…))` into one `async def _run()` that (a) runs `import_sources(sources, llm, prune=args.prune, limit=args.limit)` and (b) — when the summary has `added + updated > 0` **or** no overview row exists yet — awaits `regenerate_overview(llm)` (import it from `app.rag.overview`). One event loop, same `LLMClient` instance.
- `--limit` debug runs skip the regeneration (an incomplete walk must not rewrite the outline — mirrors the existing `--prune`-with-`--limit` guard).
- The final `print` gains `overview=updated|skipped|failed` (failed = `regenerate_overview` returned False via LLMError; the import's own exit code is **unchanged** — a failed outline must not fail the import).
- `Limit` guard: when `limit` is set, `added + updated > 0` does *not* trigger regeneration (log `overview: skipped (--limit)`).
2. `tests/integration/test_import_docs_overview.py` (new) — with the git sync mocked out (reuse `test_import_docs_git.py`'s mocking style) and a fake LLM whose `chat` records calls:
- import with changed files → `kb_overview` row written; `chat` called once; print shows `overview=updated`.
- unchanged re-import (same hashes) → `chat` **not** called; print shows `overview=skipped`.
- `chat` raising `LLMError` → exit code still `0` (no import errors), print shows `overview=failed`, previous row untouched.
- `--limit` run with changes → `overview=skipped`.
- first-ever import (no row) with zero *changed* docs is not possible (new docs are "added") — but an empty-source run with no row → no row created, `overview=skipped`.
3. `.env.example` — `BOR_KB_OVERVIEW_MAX_CHARS` (default 4000), `BOR_OVERVIEW_INPUT_MAX_CHARS` (default 40000).
4. `README.md` — import workflow section: the import now refreshes the KB overview after a KB-changing run (fail-soft, `overview=` token in the summary line).
- ASSUMPTION: "whenever we import new documents" = whenever an import **added or updated** at least one document (or the row doesn't exist yet); unchanged re-imports and `--limit` debug runs do not burn a lite call.
## Testing & Quality
- Integration: Work step 2 (real Postgres, mocked git + fake LLM — no live aipi).
- Coverage: `app/` TOTAL ≥ pre-change (the script change is covered by the integration tests; the script itself is outside the `app/` gate).
## Completion Criteria
- [ ] All new integration tests green; `tests/integration/test_import_docs_git.py` stays green.
- [ ] A manual run (`uv run python -m scripts.import_docs`) against the dev KB logs `overview: regenerated docs=… chars=…` after a KB-changing import and `overview=skipped` otherwise.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,37 +0,0 @@
# Task 01 — Sync API: in-process runner + status
**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — "triggers a doc import sync by cloning the relevant repos and then running import doc script"`
**Story:** `.agent/user_stories/admin-sync-button.md`
## Objective
The backend of the sync button: an admin-only `POST /api/sync` that starts the clone → import → overview pipeline as one in-process background task, and `GET /api/sync/status` for the UI's polling loop.
## Work
1. `app/api/sync.py` (new):
- `@dataclass SyncStatus` — `state: Literal["idle", "running", "success", "failed"] = "idle"`, `started_at: datetime | None`, `finished_at: datetime | None`, `detail: dict[str, Any] = field(default_factory=dict)`, `error: str | None`; module-level `_status` + `_task: asyncio.Task | None`.
- `GET /api/sync/status` (`Depends(require_admin)`) → JSON `{state, started_at, finished_at, detail, error}` (datetimes ISO-8601 or null).
- `POST /api/sync` (`Depends(require_admin)`) — if `_task` is not done → `409 {"detail": "a sync is already running"}`; else `_task = asyncio.create_task(_run_sync())` → `202 {"detail": "sync started"}`.
- `async def _run_sync()`:
1. `_status.state = "running"`, `started_at = now(UTC)`.
2. Resolve repos from `settings.git_source_list` — empty → fail with `"no git sources configured (BOR_GIT_SOURCES)"`.
3. For each URL: `clone_or_pull(url, Path(settings.sources_dir).expanduser() / repo_name(url))` (imported from `scripts.git_sync` / `scripts.import_docs` — no git re-implementation; `GitSyncError` carries git's stderr).
4. `summary = await import_sources(sources, LLMClient(), prune=True)` (prune per phase locked decision).
5. If `summary.added + summary.updated > 0`: `await regenerate_overview(llm)`.
6. `_status.state = "success"`, `finished_at`, `detail = {files, added, updated, unchanged, pruned, errors, chunks, summaries, summary_errors, overview: bool}`; log `sync: done detail=…`.
7. Any `GitSyncError | EmbeddingError | Exception` → `_status.state = "failed"`, `finished_at`, `error = str(e)` (sanitized: no secrets; git's stderr is fine), `logger.exception("sync: failed")`.
2. `app/main.py` — `from app.api.sync import router as sync_router` + `app.include_router(sync_router, prefix="/api")` (next to the other routers).
3. `tests/integration/test_sync_api.py` (new) — sign in via the existing auth test helper (`tests/integration/test_auth_api.py` pattern):
- anonymous: `GET /api/sync/status` → 403; `POST /api/sync` → 403.
- admin: idle state initially; `BOR_GIT_SOURCES` set to one `file://` URL with `clone_or_pull`, `import_sources`, `regenerate_overview` **monkeypatched** in `app.api.sync` (the mock import returns a canned `ImportSummary`; the mock overview returns True) → `POST` → 202; poll status → `success` with the canned detail (all ImportSummary fields + `overview: true`).
- 409: mock runner sleeps briefly (asyncio.sleep) → second `POST` while running → 409.
- failure: mock `clone_or_pull` raises `GitSyncError("git clone failed …")` → status `failed`, `error` names the failure; import is **not** called.
- empty `BOR_GIT_SOURCES` → `POST` 202 → status `failed` with the "no git sources configured" message.
- prune: assert the monkeypatched `import_sources` received `prune=True`.
## Testing & Quality
- Integration: Work step 3 (real Postgres not required for the runner logic beyond none — keep DB-free; if the session needs Postgres for nothing, use the app fixture without DB).
- Coverage: **>90%** on `app/api/sync.py` (all states/branches hit).
## Completion Criteria
- [ ] All integration tests green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] SSE/API routes untouched — `test_chat_api.py` green (no middleware or router precedence change).
@@ -1,40 +0,0 @@
# Task 02 — The admin-only Sync button on Sources (§7.4 feedback)
**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — "a button that only the admin can see that triggers a doc import sync"`
**Story:** `.agent/user_stories/admin-sync-button.md`
## Objective
The UI: a **"Sync sources"** button in the Sources page header — hidden by default, revealed only for the signed-in admin (the existing `header.js` whoami gate) — with the full "never stale" feedback lifecycle: idle → "Syncing…" (disabled, spinner, 2 s status polling) → last-result label or error banner.
## Work
1. `frontend/sources.html` — in the header actions area (next to `.new-chat-btn`, inside the same `.header-inner` container the phase-19 shared header uses on this page):
- `<button type="button" class="sync-btn" id="sync-btn" hidden aria-label="Sync sources">` — a small refresh-cycle `<svg aria-hidden="true">` icon (spin it via CSS in the running state) + `<span class="sync-label" id="sync-label">Sync sources</span>`.
- `<span class="sync-result" id="sync-result" role="status" aria-live="polite"></span>` right after the button (announces last-result / counts to screen readers).
- Update the `.page-sub` copy: it still says "Re-run the import to refresh" — extend it to mention the button (e.g. "…or hit **Sync sources** in the header to clone the repos and re-import.").
2. `frontend/assets/header.js` — add `#sync-btn` to the **admin reveal** that already handles `#nav-sources` / `#nav-tuning` after `fetchIsAdmin()` (one fetch, no extra whoami call); anonymous users never see it (stays `hidden`).
3. `frontend/assets/sources.js` — the sync state machine (new, isolated section):
- On load (admin only — `header.js` exposes the whoami result or a shared `isAdmin` flag; reuse whatever mechanism it already provides for the nav reveals): `GET /api/sync/status` →
- `running` → enter the running state and start polling (the user may have reloaded mid-sync).
- `success` / `failed` → render the last result (below) but keep the button ready for a fresh sync.
- Click → `POST /api/sync` → `202` → running state: button `disabled` + `aria-busy="true"`, icon spinning, label **"Syncing…"**, start polling `GET /api/sync/status` every **2000 ms**.
- Terminal state (stop polling):
- `success` → enabled, icon reset, label **"Synced HH:MM"** (local time of `finished_at`), `#sync-result` = `"{added} added · {updated} updated · {pruned} pruned"` (omit zero terms) — announced via `aria-live`.
- `failed` → enabled, label **"Sync sources"** (retry-ready), and show the page error banner (the existing `role="alert"` pattern used elsewhere in `sources.js`, or the chat error-banner markup style) with the `error` text; `#sync-result` cleared.
- `409` on POST (a run started elsewhere) → just enter running state + polling (adopt the in-flight run); `403` → treat as not-admin (hide the button — defense in depth).
- **No client-side hard timeout** (phase locked decision — the poll is the feedback loop; the server state is authoritative).
4. `frontend/assets/styles.css` — `.sync-btn` styled like `.new-chat-btn`/`.auth-link` (dark tech theme tokens; text contrast ≥4.5:1 — use the dark-ink-on-brand pairing per PLAN §7.2 if the button is filled, else soft-ink on surface), `.sync-btn[disabled]` state, `.sync-btn .sync-icon.is-spinning { animation: spin 1s linear infinite }` with the existing `prefers-reduced-motion` opt-out, `:focus-visible` 3px outline, ≥44 px touch target on mobile.
5. `tests/unit/test_sync_button.py` (new, frontend-assertion style of `test_shared_header.py` / `test_frontend_feedback.py`):
- `sources.html` contains `#sync-btn` with `hidden`, `aria-label="Sync sources"`, and `#sync-result` with `role="status"` + `aria-live="polite"`.
- `header.js` reveals `#sync-btn` in the admin branch (assert the element id appears in the reveal logic, same as `#nav-sources`).
- `sources.js` references `/api/sync` (POST + status GET), the 2000 ms poll, `409` adoption, `403` hide, and the terminal labels (`Syncing…` / `Synced` / failure banner).
- no-CDN integration test stays green (no new external references).
## Testing & Quality
- Unit: Work step 5 (frontend-assertion tests); no-CDN integration test green.
- Coverage: `app/` TOTAL unchanged (frontend-only task); the UI is gated end-to-end by task 03's E2E.
## Completion Criteria
- [ ] Anonymous: `#sync-btn` never leaves `hidden` in the DOM; admin: it is revealed without a page reload round-trip beyond the existing whoami fetch.
- [ ] Full lifecycle works against the dev server (manual check): click → "Syncing…" (disabled) → "Synced HH:MM" + counts, or error banner with retry; reload mid-sync re-enters the running state.
- [ ] UI Structure Check (AGENTS.md rule 5): labeled + focus-visible + contrast ≥4.5:1 + `aria-live` result; reduced-motion respected; no CDN.
- [ ] `uv run pytest` green (including the new frontend-assertion tests); `uv run ruff check . && uv run pyright` clean.
@@ -1,28 +0,0 @@
# Task 01 — app/core/caching.py: the asset version token
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — "We need better cache busting, the pages are too sticky"`
**Story:** `.agent/user_stories/cache-busting.md`
## Objective
One source of truth for the asset version token: git short SHA when the checkout is a repo (a commit = a deploy), a stable content-hash fallback otherwise — computed once per process.
## Work
1. `app/core/caching.py` (new) —
- `def asset_version(static_dir: str | None = None) -> str`:
- `static_dir` defaults to `get_settings().static_dir` (resolves to `frontend`); the git repo root is its parent.
- **Git path:** if `(static_dir parent / ".git")` exists → `subprocess.run(["git", "-C", str(root), "rev-parse", "--short", "HEAD"], capture_output=True, text=True, timeout=5)` → the short SHA (e.g. `3841bd5`).
- **Fallback / failure path** (no `.git`, git missing, non-zero exit, timeout, unreadable): `hashlib.sha256` over the sorted list of `f"{relpath}:{mtime_ns}:{size}"` for every regular file under `static_dir`, first **12 hex chars**. A missing/empty `static_dir` → `"dev"`.
- `@functools.lru_cache(maxsize=None)` on the resolved-argument wrapper (settings are process-stable; the token must not be recomputed per request). Document that a process restart or new commit changes the token.
2. `tests/unit/test_caching.py` (new):
- git path: a `tmp_path` repo (`git init -q` + a commit of a dummy file, with a `frontend/` subdir inside) → token == `git rev-parse --short HEAD` of that repo; second call returns the same value (cache).
- fallback: a plain `tmp_path/frontend` with two files → 12-hex token; unchanged tree → same token; touch/modify a file (mtime or size change) + `asset_version.cache_clear()` → different token.
- failure: `static_dir` with a `.git` present but `git` removed from PATH (monkeypatch `subprocess.run` to raise `FileNotFoundError`) → falls back to the content hash, no exception.
- empty/missing dir → `"dev"`.
## Testing & Quality
- Unit: Work step 2.
- Coverage: **>90%** on `app/core/caching.py` (all branches).
## Completion Criteria
- [ ] All unit tests green (git, fallback, failure, empty — with `cache_clear()` between parametrized cases).
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,38 +0,0 @@
# Task 02 — Caching middleware + wiring + integration tests
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — "the pages are too sticky" (HTML revalidation + versioned asset URLs + immutable assets)`
**Story:** `.agent/user_stories/cache-busting.md`
## Objective
Apply the caching behavior at the transport layer: HTML pages `no-cache` with `?v=<token>` on every local asset reference; `/assets/*` immutable 1-year; everything else (all `/api/*`, including SSE) untouched.
## Work
1. `app/core/caching.py` (extend task 01's module):
- `HTML_PAGES: tuple[str, ...] = ("/", "/index.html", "/sources.html", "/document.html", "/login.html", "/tuning.html")`.
- `_ASSET_REF_RE = re.compile(r'((?:href|src)="(?:/)?assets/[^"?#]+)(")')` — matches `<link rel="stylesheet" href="/assets/styles.css">`, `<script src="assets/markdown.js"></script>` (no leading slash!), and `<script type="module" src="/assets/app.js">`; the rewrite appends `?v=<token>` before the closing quote, only when the reference has no query/hash yet (idempotent — never a double `?v=`).
- `def rewrite_asset_refs(html: str, token: str) -> str` — the pure, unit-testable rewrite.
- `def configure_caching(app: FastAPI) -> None` — one `@app.middleware("http")` (or equivalent Starlette middleware) that, **after** the response is produced:
- `path.startswith("/assets/")` → `response.headers["Cache-Control"] = "public, max-age=31536000, immutable"` (header only — never touch the body).
- `request.url.path` in `HTML_PAGES` **and** response content-type is `text/html` → `Cache-Control: no-cache` + `response.body = rewrite_asset_refs(body, asset_version())` (body via `await response.body()` — works for the buffered StaticFiles/FileResponse HTML responses; on any error or non-`text/html`, fall through to the unmodified response with only `no-cache`).
- everything else → completely untouched (no header, no body work). The `/api/*` SSE stream in particular must not be read or rewritten.
2. `app/main.py` — `from app.core.caching import configure_caching`; call `configure_caching(app)` inside `create_app()` after the routers (order: middleware wraps the whole app — call it before `return app`).
3. `tests/unit/test_caching.py` (extend) — `rewrite_asset_refs`:
- versioned: `href="/assets/styles.css"` → `href="/assets/styles.css?v=abc123"`; `src="assets/markdown.js"` (no slash) → versioned; `src="/assets/app.js"` (module script) → versioned.
- idempotent: an already-`?v=`-tagged reference is not double-tagged; a `#fragment` or existing `?query` reference is left alone.
- non-asset references untouched (`href="/sources.html"`, `href="data:…"`, `href="/login.html?next=…"`).
4. `tests/integration/test_api.py` (extend):
- `GET /` → 200, `cache-control: no-cache`; body contains `href="/assets/styles.css?v=<token>"` with a non-empty token matching `asset_version()`; the unversioned string `href="/assets/styles.css">` is **gone** from the body.
- each of the other four pages (`/sources.html`, `/document.html`, `/login.html`, `/tuning.html`) → `no-cache` + at least one versioned asset reference.
- `GET /assets/styles.css` → 200, `cache-control` contains `immutable` and `max-age=31536000`.
- `GET /api/health` → response has **no** `cache-control` injected (baseline: FastAPI's default) — assert equality with the pre-middleware behavior; the SSE chat endpoint (`tests/integration/test_chat_api.py`) stays green unmodified.
- the existing no-CDN test (`test_html_pages_served_locally_no_cdn`) stays green — the rewritten references are still same-origin.
5. `tests/unit/test_main.py` — app creation still succeeds with the middleware wired (existing creation tests stay green; add an assertion that `create_app()`'s middleware stack includes the caching middleware by name).
## Testing & Quality
- Unit: Work steps 3 + 5; Integration: Work step 4.
- Coverage: **>90%** on `app/core/caching.py`; `app/` TOTAL ≥ pre-change.
## Completion Criteria
- [ ] All new unit + integration tests green; the full suite green (especially `test_chat_api.py` — SSE unaffected).
- [ ] Manual check: `curl -si localhost:8000/ | grep -i cache-control` → `no-cache`; the HTML body shows `?v=…` asset refs; `curl -si localhost:8000/assets/styles.css | grep -i cache-control` → immutable.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,24 +0,0 @@
# Task 01 — Live tool-calling probe
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "The agent should be able to list the available sources as a tool and the read the ones it thinks are relevant"`
**Story:** `.agent/user_stories/agent-document-tools.md`
## Objective
Verify against the live aipi endpoint whether the `turbo` chat model supports OpenAI-style `tools` + streaming `tool_calls` before building the loop — the phase-17 "verified live" convention — and record which path the phase takes.
## Work
1. `scripts/llm_probe.py` — add a `--tools` flag: when set, send (a) a non-streaming `chat/completions` request with one trivial function (e.g. `get_time`, no parameters) and a user message that makes calling it natural, and (b) the same with `stream=True`; for each, print whether `finish_reason` is `tool_calls`, the parsed `function.name`/`arguments`, and (streaming) whether the calls arrive as `delta.tool_calls` chunks with `index`/`id`/partial `function.arguments` (the OpenAI wire convention). Reuse the existing env reading (`BOR_LLM_BASE_URL` / API key / `BOR_LLM_CHAT_MODEL`) and the script's existing output style.
2. Run it against the live endpoint (`uv run python -m scripts.llm_probe --tools`) and classify the verdict:
- **supported** → the phase uses OpenAI `tools`/`tool_calls` (tasks 02–03 as written).
- **not supported** → the phase uses the prompt-based structured-call fallback (task 03 documents it): the model is instructed to emit a single JSON block (`{"tool": "list_documents"}` / `{"tool": "read_document", "source": …, "path": …}`) before answering; `run_agent` parses it out of the content stream; the SSE `tool` contract and the budgets are identical.
- **intermittent** → treat as not supported (fail-loud house style) and note it.
3. Record the verdict + date where it will be read later: the task-03 `app/rag/agent.py` module docstring (task 03 writes it) and the phase commit message (task 06) — e.g. `probe: turbo tool_calls=streaming-ok 2026-08-26`.
## Testing & Quality
- The probe is a CLI script (no `app/` coverage impact). If the parsing of the probe response is factored into a function, add a small unit test for it.
- `uv run pytest` green (no regressions); ruff + pyright clean.
## Completion Criteria
- [ ] `uv run python -m scripts.llm_probe --tools` runs and prints a clear supported / not-supported verdict for both the non-streaming and the streaming request.
- [ ] The verdict (with date) is available for task 03's docstring and task 06's commit message.
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,36 +0,0 @@
# Task 02 — LLM client: `tools` + tool-call streaming
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "…read the ones it thinks are relevant… These values should be configured by environment variables"` (the client plumbing the env-tuned loop runs on)
**Story:** `.agent/user_stories/agent-document-tools.md`
## Objective
Teach `LLMClient.chat_stream` to pass an OpenAI `tools` list and to accumulate streaming `tool_calls` deltas into typed pieces — with `tools=None` producing a byte-identical request to today.
## Work
1. `app/rag/llm.py`
- New frozen dataclass next to `StreamPiece`:
```python
@dataclass(frozen=True)
class ToolCallPiece:
"""One model-requested tool call accumulated from stream deltas (phase 37)."""
id: str # the model's tool_call id; synthesized "call_<index>" when absent
name: str # "list_documents" | "read_document" (whatever AGENT_TOOLS names)
arguments: dict[str, Any]
```
- `chat_stream(self, messages, tools: list[dict[str, Any]] | None = None)`:
- When *tools* is not None, pass `tools=tools` to `chat.completions.create`; when None, do **not** include the key (byte-identical request to today).
- In the chunk loop, accumulate `delta.tool_calls` (a list of partials keyed by `index`; `id` and `function.name` arrive on the first partial for an index, `function.arguments` arrives in fragments to concatenate).
- At stream end (or when a chunk carries `finish_reason == "tool_calls"`), for each accumulated call **in index order** yield `ToolCallPiece(id, name, json.loads(arguments) or {})`.
- Malformed `arguments` JSON → raise `LLMError` (fail-loud house style — a silently dropped tool call would corrupt the loop).
- Return annotation becomes `AsyncIterator[StreamPiece | ToolCallPiece]`; update the docstring (wire convention + a pointer to the task-01 probe verdict).
- **If the task-01 verdict is "not supported"** (prompt-based fallback): skip the `tools` parameter and the delta accumulation entirely; keep `StreamPiece` unchanged; the JSON-block parse helper lands in `app/rag/agent.py` (task 03) instead.
2. `app/api/chat.py` — update the phase-17 typing import line (`StreamPiece # noqa: F401`) to also import `ToolCallPiece` so pyright sees the union; the dispatch wiring itself is task 04.
## Testing & Quality
- Unit (`tests/unit/` — extend the existing `chat_stream` tests): synthetic chunk sequences — (a) a content-only stream is unchanged (no `ToolCallPiece`; the captured `create()` kwargs have no `tools` key); (b) a tool-call stream with partials across chunks (name on the first, arguments in 2–3 fragments) → one `ToolCallPiece` with the merged JSON; (c) two calls in one stream (indices 0 and 1) → both, in index order; (d) malformed arguments JSON → `LLMError`; (e) `tools=[…]` present in the request when passed.
- Coverage: **>90%** on the modified module.
## Completion Criteria
- [ ] `chat_stream(messages)` (no tools) — all existing unit tests green unchanged.
- [ ] The new tool-call accumulation tests green; full `uv run pytest` green.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,50 +0,0 @@
# Task 03 — The basic agent loop (`app/rag/agent.py`)
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "it gets one opportunity to list documents and then one opportunity to add exactly one extra document to its context before being required to answer. These values should be configured by environment variables."`
**Story:** `.agent/user_stories/agent-document-tools.md`
## Objective
A testable agent loop: the model gets the two tools while budgets last, the app executes them against Postgres, and once both budgets are spent the tools are dropped so the model is required to answer.
## Work
1. `app/config.py` — two settings (documented, `BOR_` prefix per house style, near the RAG-tuning block):
```python
#: Per-turn opportunities to call the `list_documents` agent tool (phase 37);
#: 0 disables the tool entirely (pre-phase behavior with both at 0).
agent_list_calls: int = 1
#: Per-turn opportunities to call `read_document` (phase 37); 0 disables.
agent_read_calls: int = 1
```
2. `app/rag/agent.py` — new module. The module docstring carries: the phase, the loop contract, and the task-01 probe verdict + date (which path is in use — OpenAI tool_calls vs the prompt-based fallback).
- `AGENT_TOOLS`: the two OpenAI function definitions — `list_documents` (no parameters; description: "List every document indexed in the knowledge base, one `source/path — title` line each") and `read_document` (`source` + `path` required; description: "Add the full content of exactly one more indexed document to your context").
- DB accessors (module-level functions so unit tests can monkeypatch them):
- `list_catalog(db) -> list[tuple[str, str, str]]` — `select(Document.source, Document.path, Document.title).order_by(Document.source, Document.path)` (same order as `GET /api/docs`).
- `find_document(db, source, path) -> Document | None`.
- `@dataclass AgentHolder: read_docs: list[Document] = field(default_factory=list); tool_calls: int = 0` — the API layer (task 04) reads it after the stream.
- `async def run_agent(llm, db, *, system_prompt, user_message, seed_docs, settings, holder) -> AsyncIterator[StreamPiece | ToolCallPiece]`:
1. `messages = [{"role": "system", "content": system_prompt}, {"role": "user", "content": user_message}]`; `list_left = settings.agent_list_calls`; `read_left = settings.agent_read_calls`; `tools = AGENT_TOOLS if (list_left or read_left) else None`; `rounds = 0`; cap `max_rounds = 2 + settings.agent_list_calls + settings.agent_read_calls` (belt-and-braces — every tool round must consume a budget, so the cap only catches pathological streams).
2. Loop: stream `llm.chat_stream(messages, tools=tools)`; yield every piece as it arrives. Collect any `ToolCallPiece` (handle the first; if a stream yields both content and a tool call — rare — the content stays (it was already emitted) and the tool still runs).
- **No tool call** → return (the answer was streamed).
- **`list_documents`**: `list_left > 0` → `list_left -= 1`, `holder.tool_calls += 1`, result = `f"{n} documents:\n" + "\n".join(f"{s}/{p} — {t}")` (uncapped in v1 — **ASSUMPTION: the catalog is not truncated; the UI never shows it, only the model does**). Else result = `"No listing budget left — answer with what you have."` (no budget consumed).
- **`read_document`**: arguments must carry `source` and `path`. If that document is already in `seed_docs` or `holder.read_docs` → result = `"Already in your context."` (no budget consumed, no append). Else if `read_left > 0` → `doc = find_document(db, source, path)`; found → `read_left -= 1`, `holder.tool_calls += 1`, `holder.read_docs.append(doc)`, result = `f"Document {source}/{path}:\n{doc.content}"` (**full text, never truncated — the A7-revised contract**); not found → result = `f"No document at {source}/{path} — check the list_documents output."` (no budget consumed). `read_left == 0` → result = `"No reading budget left — answer with what you have."`.
- **Unknown tool name** → result = `"Unknown tool."` (no budget consumed).
- Append to `messages`: the assistant tool-call message (`{"role": "assistant", "content": None, "tool_calls": [{"id": tc.id, "type": "function", "function": {"name": tc.name, "arguments": json.dumps(tc.arguments)}}]}`) + the tool result (`{"role": "tool", "tool_call_id": tc.id, "content": result}`); then `tools = None if (list_left == 0 and read_left == 0) else AGENT_TOOLS` (once both budgets are spent, the next request must be answered).
- `rounds += 1`; if `rounds >= max_rounds` → force one final `chat_stream(messages, tools=None)` (yield its pieces) and return.
- **Prompt-based fallback (only if the task-01 verdict is "not supported")**: instead of `ToolCallPiece`s, scan each streamed content turn for a leading JSON block matching `{"tool": …}` (strip leading whitespace, parse with `json`; the block must be the first non-whitespace content of the turn). On a match: strip the block from the emitted stream (re-emit only trailing content, if any), then run the identical budget/result machinery. The SSE `tool` events and the budgets are unchanged either way.
3. `app/rag/prompts.py` — add the `<tools>` instructions section to the **HIGH prompt only** (a new constant, appended after the mode body; the LOW/deflection prompt stays byte-identical). Wording (tune against the live model if needed, but keep it stable — the E2E mock keys off the `<tools>` marker's *presence*, not the wording):
> If the documents in your context reference other files, or you need content that is not included above, call `list_documents` to see what is indexed, then `read_document` to pull in exactly one more document. Answer as soon as you have what you need — do not read more than one extra document.
## Testing & Quality
- Unit (`tests/unit/test_agent.py`, new): a scripted fake LLM (canned stream sequences) + monkeypatched `list_catalog`/`find_document` + a `Settings` with budgets set:
- list → read → answer: event order (tool pieces before content), `holder.read_docs == [doc]`, `holder.tool_calls == 2`, the request after budgets are spent carries `tools=None`, the follow-up request contains the assistant tool-call + tool-result messages.
- Budgets: `read_left` exhausted → a second `read_document` gets "No reading budget left" and appends nothing; `agent_list_calls=0` + `agent_read_calls=0` → exactly one request with `tools=None` (byte-identical single-call path).
- Edge: reading a doc already in `seed_docs` → "Already in your context." (no budget consumed); unknown path → "No document at …"; unknown tool name → "Unknown tool."; the round cap forces a final no-tools answer.
- The HIGH prompt gains the `<tools>` section; the LOW prompt is byte-identical to pre-phase (assert against the existing prompt-test fixtures).
- Integration (`tests/integration/`): `list_catalog` ordering + `find_document` hit/None against real Postgres (follow the existing DB-test patterns).
- Coverage: **>90%** on the new module.
## Completion Criteria
- [ ] `app/rag/agent.py` exists with the loop above; the module docstring records the probe verdict + date (task 01).
- [ ] The budget matrix (0/1) unit tests green, including "budgets 0/0 ⇒ one request, no tools".
- [ ] HIGH prompt carries `<tools>`; LOW prompt byte-identical to pre-phase.
- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.
@@ -1,36 +0,0 @@
# Task 04 — `POST /api/chat`: `tool` SSE events + agent wiring
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "This will require some reconfiguring of the UI since it will now need to show 'calling tool' in addition to 'thinking' and it will need a basic agent loop."`
**Story:** `.agent/user_stories/agent-document-tools.md`
## Objective
Wire the loop into the chat endpoint: the new SSE `tool` event, the agent on grounded turns only, and the read document reflected in `done.sources`, `query_log`, and the per-turn log line.
## Work
1. `app/schemas.py` — add next to `ChatThinkingEvent`:
```python
class ChatToolEvent(BaseModel):
"""SSE frame for one agent tool call (phase 37, PLAN §4 extension)."""
type: Literal["tool"] = "tool"
name: str # "list_documents" | "read_document"
argument: str | None = None # "source/path" for read_document
```
2. `app/api/chat.py`
- In `stream()`, after `plan_turn`: when `not plan.deflected` → run `run_agent(llm, db, system_prompt=plan.system_prompt, user_message=request.message, seed_docs=plan.docs, settings=settings, holder=holder)`; when deflected → keep the current direct `chat_stream` (A8 byte-identical).
- Event mapping: `StreamPiece` thinking/delta exactly as today (including the `BOR_STREAM_THINKING` kill-switch and the `thinking_chars` count); `ToolCallPiece` →
`yield sse_event(ChatToolEvent(name=tc.name, argument=f"{tc.arguments.get('source')}/{tc.arguments.get('path')}" if tc.name == "read_document" else None).model_dump())`.
- `done` event: `sources` = `plan.docs + holder.read_docs` deduped by `(source, path)`, order preserved; the `ChatDoneEvent` shape otherwise unchanged.
- `query_log.sources`: the same combined list (replaces the current `source_paths` build).
- Per-turn log line: add `tool_calls={holder.tool_calls}` after `thinking_chars=` (PLAN §9 required-line extension — the phase 17/30/31 precedent).
- Extend the module docstring with the phase-37 section (the flow, the grounded-only scope, the budgets, the "both budgets 0 ⇒ pre-phase behavior" note).
3. `.agent/PLAN.md` — record two revision notes in the project's established format (phase 17/24 precedent, "owner permission 2026-08-26"): the §4 SSE contract gains `{"type":"tool","name":…,"argument":…}` (client rule: render as a "calling tool" line/state; `delta`/`done` unchanged) and the §9 per-turn log line gains `tool_calls=N`.
## Testing & Quality
- Integration (`tests/integration/test_chat*.py` — extend the existing SSE-contract suite, mock LLM that emits tool calls): (a) grounded turn with tool calls → event sequence `thinking?/tool/tool/delta…/done`, `done.sources` includes the read doc, the `query_log` row's sources match, the log line carries `tool_calls=2`; (b) deflected turn → no `tool` events, the sequence byte-identical to today; (c) budgets 0/0 → no `tool` events, the mock received no `tools` parameter, single-request path.
- Coverage: **>90%** on the modified modules.
## Completion Criteria
- [ ] `POST /api/chat` (mock LLM, grounded) streams `tool` frames ahead of the `delta` frames; the deflected path is unchanged (existing suites green).
- [ ] `done.sources`/`query_log` include the read document (deduped); the log line carries `tool_calls=N`.
- [ ] The PLAN.md §4 + §9 revision notes exist (dated, owner permission 2026-08-26).
- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.
@@ -1,27 +0,0 @@
# Task 05 — UI: the "calling tool" state
**Phase:** `37_agent_document_tools` · **Source:** `TODO.md:3 — "…it will now need to show 'calling tool' in addition to 'thinking'"`
**Story:** `.agent/user_stories/agent-document-tools.md`
## Objective
The chat shell shows a "calling tool" state (button + status label) and a visible tool line in the bubble, distinct from the Thinking scratchpad; tool lines persist with the chat record (phase 14) and re-render after reload.
## Work
1. `frontend/assets/app.js` — in the turn handler where the SSE frames are dispatched (the same switch that handles `thinking` and `delta`):
- New `tool` branch:
- **Status/label:** keep `uiState = thinking` (the button stays disabled — never stale, PLAN §7.4) and set the typing-indicator label to "Brain of Reese is listing documents" (`name: "list_documents"`) / "Brain of Reese is reading <argument>" (`name: "read_document"`); the elapsed-seconds hint (`thinkingClock` aria-label) keeps working through tool frames.
- **Bubble line:** append a tool line to the same wrap the thinking block uses (above the answer, beside/below the Thinking summary): `<span class="tool-call">🔎 Listing documents</span>` / `<span class="tool-call">📄 Reading <code>source/path</code></span>` — a real visible line with its own icon + color, distinct from the Thinking block (styles step). Multiple calls append multiple lines, in order.
- Tolerate tool frames interleaved with thinking frames (append-only, the same rule as thinking); a tool frame after the first `delta` (should not happen in v1 — the loop completes before the answer stream) still appends rather than crashes.
- **Persistence (phase 14 convention):** the saved record gains an optional `tools: [{name, argument}]` array next to `thinking`; re-render the tool lines when a record is loaded (same code path as the thinking re-render).
- Keep the new label strings as plain literals — phase 39 centralizes brand strings; do **not** introduce a brand helper here.
2. `frontend/assets/styles.css` — `.tool-call` style: inline row, icon + text, an accent color distinguishable from the thinking block's, `code` styling for the path, contrast ≥ 4.5:1 in both themes; no layout shift on append (the centered 46rem chat column is untouched — no new container).
3. UI Structure Check (AGENTS.md rule 5) before finalizing: semantic landmarks unchanged; the tool lines live inside the `#messages` `aria-live="polite"` region (announced to screen readers); focus-visible unaffected (the lines are not interactive); no new top-level landmarks.
## Testing & Quality
- No new Python logic — gated by the story E2E (task 06) plus the existing frontend-adjacent suites staying green.
- No CDN (AGENTS.md rule 6): no new `<script>`/`<link>` tags.
## Completion Criteria
- [ ] During a tool call the button label shows the "calling tool" text (not a stale "Thinking…") and the tool lines render above the answer.
- [ ] After a page reload, the persisted record re-renders the tool line(s).
- [ ] `uv run pytest` green (no regressions); `uv run ruff check . && uv run pyright` clean.
@@ -1,24 +0,0 @@
# Task 01 — `git_sources.kind` + `path` (migration 0007)
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
**Story:** `.agent/user_stories/local-directory-sources.md`
## Objective
Extend the phase-35 `git_sources` table with a source-kind discriminator (`git` | `local`) and an optional local path, via a reversible migration — the foundation for the API, pipeline, and page tasks.
## Work
1. `alembic/versions/0007_git_sources_kind.py` — read `alembic/versions/0006_git_sources.py` first and chain from its **actual** `revision` id (filenames are not revision ids — the phase-35 task-01 convention):
- `upgrade()`: `ALTER TABLE git_sources ADD COLUMN kind TEXT NOT NULL DEFAULT 'git'` + `ADD CONSTRAINT ck_git_sources_kind CHECK (kind IN ('git', 'local'))`; `ALTER TABLE git_sources ADD COLUMN path TEXT`; a unique index on `path` — a partial unique index (`WHERE path IS NOT NULL`) where the 0006 style allows it, otherwise a plain unique index (Postgres treats NULLs as distinct, and the API enforces local-only paths anyway — mirror whatever 0006 chose for `url`).
- `downgrade()`: drop the index, constraint, and columns in reverse.
2. `app/models.py` — extend the phase-35 `GitSource` model: `kind: Mapped[str]` (default `"git"`) + `path: Mapped[str | None]` (nullable, unique) and update the docstring (phase 38: kind discriminator; local rows carry `path`, git rows keep `url`).
3. Apply to the dev database: `podman compose up -d db` (if needed) then `uv run alembic upgrade head`; verify the columns + constraint + that existing rows read as `kind='git'`, `path=NULL`.
4. Migration test — follow the 0004/0005/0006 pattern in `tests/integration/`: up adds the columns/constraint/index (existing rows keep `kind='git'`), down drops them (round-trip on the test DB).
## Testing & Quality
- Integration: the up/down test above; full `uv run pytest` green.
- Coverage: model + migration only — the `app/` gate stays >90%.
## Completion Criteria
- [ ] `alembic/versions/0007_git_sources_kind.py` chains off 0006's real revision id; up/down reversible.
- [ ] `uv run alembic upgrade head` applies cleanly; existing rows read `kind='git'`, `path=NULL`.
- [ ] The 0007 up/down integration test passes; full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,27 +0,0 @@
# Task 02 — The admin API: the local kind
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
**Story:** `.agent/user_stories/local-directory-sources.md`
## Objective
Extend the phase-35 git-sources CRUD to accept and return `kind=local` rows with fail-loud path validation, keeping the git contract byte-identical.
## Work
1. The phase-35 API module (the file behind `/api/git-sources` — identify it in the repo; phase 35 created it):
- `POST /api/git-sources` body: `{kind?: "git"|"local" (default "git"), url?, path?}`:
- `kind=git` → exactly today's `url` validation (trimmed, 1–500 chars, the `https?://` / `ssh://` / `git@` shape, 409 on duplicate without echoing the URL).
- `kind=local` → `path` required: trimmed; `Path(p).expanduser()`; must be **absolute after expansion** and an **existing directory on the server** → else `422 {detail: "local source path is not a directory: <path>"}` (a missing path is a user error — fail loud at add-time so the owner sees it immediately; the path is not a secret, so echo it); 409 on a duplicate `path` (detail may name the path).
- Wrong field combinations (git without url, local without path, both kinds' fields) → 422.
- `GET /api/git-sources` rows gain `kind` + `path` (git rows: `path: null`; the env-fallback rows report `kind: "git"`; `from_env: true` semantics unchanged — env rows are git-only).
- `DELETE /api/git-sources/{id}` — unchanged (removal prunes on the next sync, as today).
2. The phase-35 request/response models (wherever they live — `app/schemas.py` or the module) — extend for the new fields; keep the OpenAPI docs accurate.
3. Integration tests (extend `tests/integration/test_git_sources_api.py`): anonymous → 403 on all routes (regression); `kind=local` + an existing temp dir → 201 with the stored row (`kind=local`, `path` stored expanded); a relative path → 422; a missing path → 422 naming the path; a duplicate path → 409; a git row still validates exactly as before (regression); GET mixes kinds in added order.
## Testing & Quality
- Integration: the matrix above; the existing git-kind suite green unchanged.
- Coverage: **>90%** on the modified module.
## Completion Criteria
- [ ] The local-kind POST matrix (201 / 422 / 409) green; the git contract unchanged.
- [ ] GET rows carry `kind` + `path`; the env fallback stays git-only.
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,30 +0,0 @@
# Task 03 — Sync + `import_docs`: git and local together
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
**Story:** `.agent/user_stories/local-directory-sources.md`
## Objective
The canonical mirror action (the Sync button) and the CLI import DB **git + local** rows in one run: git rows clone/pull as today, local rows are walked directly; a missing local directory aborts the run loudly before anything is imported.
## Work
1. The phase-35 resolution module (`app/rag/git_sources.py` or wherever `effective_git_sources()` lives — identify it in the repo) — extend:
- `effective_sources(db) -> list[GitSource]` — the DB rows of **both** kinds (DB-wins / env-fallback semantics unchanged: while the table is empty, the env git URLs surface as synthetic `kind='git'` rows with `from_env`). Keep a backward-compatible alias (`effective_git_sources`) if other modules import the old name.
- Log the list origin as today (`origin=db|env`) plus the kind counts (`git=N local=M`).
2. `app/api/sync.py::_run_sync`:
- For each resolved row: `kind=git` → `clone_or_pull(url, sources_root / repo_name(url))` (unchanged); `kind=local` → `Path(row.path).expanduser()`, verify `.is_dir()` **at sync time** (the directory may have moved/deleted since add-time) → else raise a sync error `local source missing: <path>` (no credentials involved, but run it through the existing `_sanitize_error` for consistency).
- `import_sources(combined, llm, prune=True)` over the **single combined list** (git checkouts + local dirs) — pruning covers the union (phase-32 semantics); the overview regeneration is unchanged (change-gated).
- The fail-loud empty-config check: no git rows, no local rows, and no env URLs → `"no sources configured (git or local)"` (replaces phase 32's git-only message).
3. `scripts/import_docs.py` — `_resolve_sources` gains the DB path (after `--source`, which still wins over everything): no `--source` and the DB table non-empty → the combined list (git cloned/pulled + local direct); table empty → env git URLs (as today) → the legacy `DEFAULT_SOURCES`. A missing local dir → abort with the path named, before importing anything (the same pre-import fail-loud as a failing git clone).
4. Unit/integration:
- Unit: `effective_sources` — mixed kinds, DB-wins, env fallback git-only, both-empty (extend the phase-35 tests).
- Integration (the `test_sync_api.py` pattern, with a **temp local dir** — a host temp dir containing one fixture `.md`, since the app server runs on the same host): local-only, git-only, and mixed syncs — the local file lands in the KB (`GET /api/docs` as admin); a missing local path → status `failed` with `local source missing: …` in the sanitized error; `import_docs` (no `--source`) with a DB local row imports it; `--source` still wins over the DB.
## Testing & Quality
- The suites above; `test_git_sources_api.py`, `test_sync_api.py`, `test_import_docs_git.py` (phase 35's list) stay green through the indirection.
- Coverage: **>90%** on the modified modules.
## Completion Criteria
- [ ] A mixed git + local sync imports both in one run; union pruning works (a file deleted from the local dir is pruned on the next sync).
- [ ] A missing local dir fails the run loudly (the status error names the path) and imports nothing.
- [ ] `import_docs`: DB git + local resolution; `--source` wins; the env fallback is git-only; the both-empty message is updated.
- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.
@@ -1,24 +0,0 @@
# Task 04 — The page: the local-directory form + badges
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
**Story:** `.agent/user_stories/local-directory-sources.md`
## Objective
The sources page (phase 35) can add a **local directory** next to git repos, the list shows which is which, and the hint reflects the combined Sync semantics.
## Work
1. The phase-35 page (the git-sources HTML template + its page script — identify the actual paths in the repo):
- A second add form, **"Local directory"**: a labeled text input (placeholder `~/Notes`), an Add button, an inline error slot — the same never-stale-button + inline-error pattern as the git form (PLAN §7.4); on success the form clears and the list re-fetches.
- List rows: a kind badge — `Git` / `Local` (a small styled span, distinguishable by color **and** text, not color alone — WCAG) — plus the mono value (git URL as today; the full local path) + added date + Remove (Remove is unchanged — it prunes on the next sync).
- The hint text: "Sync clones/pulls the git repos and imports the local directories together (files removed from a source are pruned)."
- Anonymous: the sign-in gate unchanged; the admin-only nav link unchanged (phase 35/29).
2. `frontend/assets/styles.css` — the badge styles + the second form (reuse the existing form styles; contrast ≥ 4.5:1 in both themes).
3. UI Structure Check (AGENTS.md rule 5) before finalizing: landmarks intact, both inputs labeled, focus-visible on the new Add button, the page stays inside the shared layout (no new top-level structure).
## Testing & Quality
- Gated by the story E2E (task 05); no CDN (AGENTS.md rule 6 — no new external tags).
## Completion Criteria
- [ ] The admin adds a local directory through the page; the row appears with the Local badge; an invalid path shows the 422 detail inline and the button recovers (never stale).
- [ ] The git form + all existing page behavior unchanged.
- [ ] `uv run pytest` green (no frontend unit layer — the E2E is the gate); `uv run ruff check . && uv run pyright` clean.
@@ -1,27 +0,0 @@
# Task 01 — `GET /api/config`
**Phase:** `39_configurable_brand` · **Source:** `TODO.md:12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."`
**Story:** `.agent/user_stories/configurable-brand.md`
## Objective
A public, stateless endpoint that hands the frontend its display name (+ version) — the single source the brand layer reads.
## Work
1. `app/api/config.py` — a new router mirroring the `app/api/health.py` pattern:
```python
@router.get("/config")
def app_config(settings: Settings = Depends(get_settings)) -> dict[str, str]:
"""Public app metadata for the frontend brand layer (phase 39)."""
return {"app_name": settings.app_name, "version": settings.app_version}
```
Public (no `require_admin` — the brand must render for anonymous users too, before any sign-in); stateless (A10); the response carries **exactly** these two keys (no other setting may leak in later — the test asserts the key set).
2. `app/main.py` — register it with the other routers (`app.include_router(config_router, prefix="/api")`), before the static mount (API routes take precedence — same order as the existing routers).
3. Tests (follow the health test's location and pattern): 200 for anonymous; default values (`"Brain of Reese"`, the current `app_version`); with an overridden `Settings` (`app_name="Brain of Testy"`) the response follows; the response key set == `{"app_name", "version"}`.
## Testing & Quality
- The tests above; full `uv run pytest` green.
- Coverage: **>90%** on the new module (keep the handler + docstring tight so it stays covered).
## Completion Criteria
- [ ] `GET /api/config` (anonymous) → `{"app_name": "Brain of Reese", "version": "0.1.0"}` (the current default version).
- [ ] The suite green; `uv run ruff check . && uv run pyright` clean.
@@ -1,34 +0,0 @@
# Task 02 — `brand.js` + de-hard-coding
**Phase:** `39_configurable_brand` · **Source:** `TODO.md:12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."`
**Story:** `.agent/user_stories/configurable-brand.md`
## Objective
Every visible brand string on every page resolves from one place (`window.BOR_BRAND`, fed by `/api/config`) — the default "Brain of Reese" renders immediately, and a fetch failure falls back to the default (the page never breaks).
## Work
1. `frontend/assets/brand.js` — a new small **classic** script (vanilla, no CDN — A11; not a module, so its top level runs at parse time):
- Top level: `window.BOR_BRAND = "Brain of Reese"` (synchronous default — module scripts execute after parsing, so the page scripts can read it from the first line).
- DOM application (deferred: `if (document.readyState === "loading") document.addEventListener("DOMContentLoaded", applyBrand); else applyBrand();`):
1. `fetch("/api/config", { cache: "no-store" })`; on success with a non-empty `app_name`: `window.BOR_BRAND = app_name`, then:
2. `document.title = document.title.replaceAll("Brain of Reese", name)`.
3. Every `.brand-text` node: if the name starts with `"Brain of "` → `innerHTML = 'Brain of <strong>' + escapeHTML(rest) + '</strong>'` (the current look for the default name); else → `textContent = name` (plain, no bold). **HTML-escape the name** (an operator-controlled string must not inject markup).
4. A `TreeWalker` over the document's text nodes: replace the literal "Brain of Reese" with the name (catches the index empty-state h1 "Hey! I'm Brain of Reese." and any prose).
5. An attribute pass over `aria-label`, `placeholder`, and meta `content` attributes containing the literal → replace (the `#messages` aria-label "Conversation with Brain of Reese", the input label, the meta descriptions).
- On fetch failure: keep the default, `console.warn` (the `loadHealth` house style — progressive enhancement, never break the page).
- Small local `escapeHTML` helper (the `markdown.js` pattern — do not import across modules unless the build makes it easy).
2. The templates (`frontend/index.html`, `sources.html`, `tuning.html`, `document.html`, `login.html`) — add `<script src="assets/brand.js"></script>` **before** the page's module script on every page (classic script → runs at parse time; the module scripts execute later). No other template changes — the walker + attribute pass is the single mechanism; do **not** add `data-brand` markers. The phase-33 `?v=` rewriting picks the new ref up automatically (`app/core/caching.py` matches any `src="…assets/…"`).
3. `frontend/assets/app.js` — replace the "Brain of Reese" literals with `window.BOR_BRAND` reads: the `UI_STATE` labels (~L107: "… is thinking" / "… is answering"), `TYPING_LABEL` (~L112), the elapsed-hint aria-label (~L540). Pattern: `const brand = () => window.BOR_BRAND || "Brain of Reese";` + template strings. (Mid-turn staleness: a label set before the fetch lands keeps the old name for that turn — accepted, see the phase's locked decisions.)
4. `frontend/assets/document.js` — the page titles (L147/152: `${doc.title} · Brain of Reese` / `"Document not found · …"`) → use the same `window.BOR_BRAND` read (document.js is a module — `window.BOR_BRAND` is set by then).
5. `Containerfile` — add the esbuild line for the new file next to the others (L17–23 pattern, classic script like `markdown.js`): `esbuild ./assets/brand.js --minify --outfile=/out/assets/brand.js`.
6. Verify the no-op property: with the default settings the rendered DOM text is byte-identical to pre-phase on all five pages (the replace is a no-op for the default name) — the existing suites' title/label assertions are the guard; if any assert a string this task moved onto `window.BOR_BRAND`, the default path must render the identical bytes.
## Testing & Quality
- No Python logic — the story E2E (task 03) is the gate; the existing suites (which assert the default "Brain of Reese" titles/labels against the shared conftest server) must stay green **unchanged**.
- No CDN (rule 6): no new external tags. UI Structure Check (rule 5): no landmark/contrast change — the brand text keeps its existing classes and styling (the `innerHTML` rewrite only re-emits the same structure with the new name).
## Completion Criteria
- [ ] `BOR_APP_NAME` unset → all five pages render exactly as today (existing suites green).
- [ ] `BOR_APP_NAME="Brain of Testy"` → title/header/greeting/labels/aria all carry the new name (asserted by the story E2E, task 03).
- [ ] The Containerfile build includes brand.js; the asset ref is versioned like its siblings (phase 33).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,48 +0,0 @@
# Phase 30 — Document Summaries (lite-model summaries for non-markdown documents)
**Source:** `TODO.md L3 — "One issue I'm having is bad context for the embedder which causes poor retrieval results… we need a small model to analyze non markdown documents and provide a textual summary of those documents with a pointer back to the source… if retrieval == summary, fetch documents referenced by summary… The small model available on aipi.reeseapps.com is 'lite'."`
**Story:** `.agent/user_stories/document-summaries.md`
**Context:** The importer (`app/rag/importer.py::_index_file` — chunk → embed → upsert per file, A9 scope), hybrid retrieval (`app/rag/retriever.py` — A7: cosine ∪ FTS, RRF, chunk→parent-document mapping, full-document context never truncated, phase 24), the locked persona prompts (`app/rag/prompts.py`), the aipi client (`app/rag/llm.py` — A5: `turbo` chat streaming + `embed` embeddings), and the E2E mock LLM (`tests/e2e/mock_llm.py` — deterministic, keys on system-prompt markers like `DEFLECT_MODE` / `<tuning>`).
## Objective
Give every **non-markdown** A9 document (txt, yaml, yml, json, py) a natural-language summary generated at import time by the aipi **`lite`** model. The summary is stored on the document (`documents.summary`) **and indexed as one extra embedded chunk** (`chunks.is_summary`), so hybrid search has a well-embedding natural-language target to hit instead of the badly-formatted raw text. A summary hit resolves to its parent (the source document) — the existing chunk→document mapping then feeds the **full source document** to the LLM, implementing the TODO's "if retrieval == summary, fetch the documents referenced by the summary" step. The per-turn log line records how many summary hits landed in the selected context.
## Dependencies
- `29_tuning_nav_link` (complete) — the latest finished phase (sequencing only).
- Substantively builds on: `02_story_import_documents` / `24_whole_document_context` (import pipeline + full-document context contract), `09_story_retrieval_quality` (A7 hybrid retrieval the summary chunk flows through unchanged), `01_infrastructure` (models/alembic, LLM client, E2E mock).
## Tasks
1. `01_lite_model_client.md` — `BOR_LLM_SUMMARY_MODEL` (default `lite`) + non-streaming `LLMClient.chat()` for the lite model.
2. `02_migration_summary_columns.md` — Alembic 0004: `documents.summary TEXT NULL` + `chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE`.
3. `03_summarizer_module.md` — `app/rag/summarizer.py`: `SUMMARY_MODE` prompt (capped input), lite call, output validation + deterministic `Source: <source>/<path>` pointer line.
4. `04_importer_summary_integration.md` — importer generates/stores/indexes summaries for non-md files (best-effort fail-soft) + summary counters.
5. `05_pipeline_summary_resolution.md` — `is_summary` through the retriever, `summary_hits` in `TurnPlan` + the per-turn log line; full source document on summary hit (existing mapping, asserted).
6. `06_mock_and_e2e.md` — deterministic `lite` in `mock_llm.py`, sentinel fixture, `tests/e2e/test_document_summaries.py`, story file, README, commit.
## Testing & Quality
- Unit: summarizer (prompt/cap/pointer/errors), importer (summary happy path, md exclusion, fail-soft, replacement on re-import), retriever (`is_summary` through both candidate lists + `fuse`), chat gate (`TurnPlan.summary_hits`), LLM client (`chat()`).
- Integration: migration 0004 up/down.
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`, TOTAL ≥ pre-change number).
- E2E (mandatory, A16): `tests/e2e/test_document_summaries.py` — one story, run **in isolation** (`uv run pytest tests/e2e/test_document_summaries.py -v --no-cov`); proves summary hit → full source document reaches the answer (sentinel in the raw doc, absent from the mock summary).
- All existing E2E suites stay green (new columns are defaulted; all existing chunks have `is_summary=false`).
## Completion Criteria
- [ ] After `uv run python -m scripts.import_docs`, every non-markdown fixture/doc has `documents.summary` set and exactly one `is_summary` chunk (position −1, embedded); markdown docs have neither.
- [ ] A question whose best match is a summary chunk yields an answer grounded in the **full source document** (E2E sentinel) and the per-turn log line shows `summary_hits>=1`.
- [ ] A lite-model failure during import does **not** drop the document — it is indexed without a summary, logged, and counted (`summary_errors`).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
- [ ] `uv run pytest tests/e2e/test_document_summaries.py -v --no-cov` green in isolation; existing suites (`test_chat_rag.py`, `test_retrieval_quality.py`, `test_import_documents.py`, `test_whole_document_context.py`) stay green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/document-summaries.md` exists.
- [ ] `.env.example` + README document `BOR_LLM_SUMMARY_MODEL` / `BOR_SUMMARY_MAX_CHARS` and the summary behavior.
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc`); `.agent/phases/todo/30_document_summaries/` moved to `.agent/phases/complete/`.
## Locked decisions
- **A5 extended, not revised** — the `lite` model is served by the same OpenAI-compatible endpoint (`https://aipi.reeseapps.com/v1`) via a new `BOR_LLM_SUMMARY_MODEL` setting (default `lite`); no new model management, no new package.
- **A7 untouched** — hybrid retrieval logic is unchanged; the summary is an ordinary chunk, so it flows through the existing cosine ∪ FTS ∪ RRF path and the chunk→document mapping. The "fetch the referenced document" step is the existing full-document context contract (phase 24) — never truncated.
- **A9 untouched** — "non-markdown" means every *already-imported* A9 document except `md`/`markdown`. The TODO's quadlet-file example is **out of scope**: `.quadlet` is not an A9 format and `BOR_IMPORT_EXTENSIONS` may only narrow the locked set (flagged at roadmap confirmation; importing quadlet files would require an owner-permission A9 revision).
- **A13** — migration 0004 adds two columns (`documents.summary`, `chunks.is_summary`); no table rework, both reversible.
- **Summary generation is best-effort** — a lite failure logs + counts (`summary_errors`) and the file is still indexed without a summary (same fail-soft spirit as the per-file `EmbeddingError` handling, but weaker: the doc is already committed).
- **Pointer is code-deterministic** — the `Source: <source>/<path>` line is appended by `summarizer.py`, never trusted to the model.
- **A16 honoured** — one dedicated story E2E suite; E2E stays deterministic via the mock LLM's `SUMMARY_MODE` marker.
- **A17 honoured** — one atomic `--no-gpg-sign` commit.
@@ -1,35 +0,0 @@
# Task 06 — Deterministic lite mock + story E2E + docs + commit
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — (whole item: bad embedder context for non-markdown docs → lite summaries → summary hits fetch the referenced source; the aipi 'lite' model)`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Close the loop: a deterministic `lite` behavior in the E2E mock, a sentinel fixture proving that **a summary hit still delivers the full source document to the LLM**, the story E2E suite, the story file, README docs, and the phase commit.
## Work
1. `tests/e2e/mock_llm.py` — in the chat-completions handler (non-stream and stream paths), **before** the `DEFLECT_MODE` check: if the system prompt contains `SUMMARY_MODE`, return the deterministic digest
`f"This document covers {' '.join(TOKEN_RE.findall(_user(body).lower())[:24])}."`
— the first 24 tokens of the document content (the summarizer puts the content in the *user* message). Byte-stable for a given fixture.
2. E2E fixture — a new non-markdown fixture doc, e.g. `quadlet/qwen-llamacpp.yaml` under the existing E2E fixture KB (follow the import-dependent fixtures' pattern in `tests/e2e/conftest.py` / the fixture dir used by `test_whole_document_context.py`):
- The document **opens** with a header comment line dense in the question tokens (e.g. `# qwen 3.8 llama.cpp optimal parameters deployment notes`) so the mock's 24-token summary digest contains the question's words, followed by ~4–5 k of other yaml content (so the raw chunks dilute their overlap and the summary chunk ranks first — the mock's embeddings are a pure function of tokens, so the ranking is fully deterministic for a fixed fixture; iterate the fixture text until the E2E assertions hold).
- A unique sentinel `RESE-SUMMARY-SENTINEL-7f3a` on the **last line** of the document (outside the 24-token digest, unreachable from the summary).
3. `tests/e2e/test_document_summaries.py` (new, the story gate) — reuse the E2E conftest app/DB fixtures:
- Import the fixture KB (re-import pattern used by import-dependent stories).
- Ask the question (e.g. "What are the optimal parameters for qwen 3.8 on llama.cpp? show the end of your notes" — the phase-24 tail-echo trigger makes the answer quote the **last 160 chars of the document context**).
- Assert the rendered brain answer contains `RESE-SUMMARY-SENTINEL-7f3a` → the full **source** document was in the LLM context (only possible via the summary→parent-document resolution, since the summary digest cannot contain the sentinel).
- Assert the source chip shows the fixture doc's path and `deflected` is false.
- Control: a markdown fixture doc in the same KB gets **no** summary chunk — assert via the Sources table (admin, `#docs-table`) or a direct DB check in the test: markdown doc's chunk count == raw chunks only; the yaml doc has exactly one `is_summary` row.
4. `.agent/user_stories/document-summaries.md` (new) — narrative + acceptance criteria + Playwright mapping rule (story → `tests/e2e/test_document_summaries.py`), matching the style of `.agent/user_stories/git-sources.md`.
5. `README.md` — new "Document summaries" section: what gets summarized (non-markdown A9 docs), the `Source:` pointer, `BOR_LLM_SUMMARY_MODEL` / `BOR_SUMMARY_MAX_CHARS`, fail-soft behavior, and how summary hits appear in the per-turn log.
6. Commit: `git add` the phase's app/script/test/README files; `git commit --no-gpg-sign -m "feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc"`; move `.agent/phases/todo/30_document_summaries/` → `.agent/phases/complete/` (`.agent/` is gitignored by design — force-add only if the commit must record the plan change, otherwise leave the move out of git).
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_document_summaries.py -v --no-cov` green **in isolation** (Chromium installed; `podman compose up -d db` up; mock LLM — no live aipi needed).
- Regression: run `test_chat_rag.py`, `test_retrieval_quality.py`, `test_import_documents.py`, `test_whole_document_context.py` in isolation — all stay green.
- Full gate: `uv run pytest` + `uv run pytest --cov=app --cov-report=term-missing` (TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `tests/e2e/test_document_summaries.py` green in isolation; the sentinel assertion proves summary hit → full source document.
- [ ] All regression suites listed above green in isolation.
- [ ] Full test gate + lint/type gate green (per this phase's 00_phase.md).
- [ ] Story file + README + `.env.example` complete; one `--no-gpg-sign` commit made.
@@ -1,45 +0,0 @@
# Phase 31 — KB Overview in the System Prompt (lite-generated knowledge-base outline)
**Source:** `TODO.md L4 — "The system prompt should inject basic categories of everything that's been read so the agent knows roughly what its knowledge base contains before the rag retrieval returns documents. This part of the system prompt should be generated by the lite model and should be stored somewhere so it can be updated whenever we import new documents."`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
**Context:** Phase 15 steering notes (the `<tuning>` prompt section, its char budget, and the **byte-identical-when-absent** convention — `app/rag/prompts.py::build_steering_section`), phase 30 (the `lite` client method `LLMClient.chat`, and per-document summaries that make a much better overview input than raw titles), `scripts/import_docs.py` (the place "whenever we import new documents" happens), and the E2E mock's answer-echo convention (the `(tuning: …)` suffix — `tests/e2e/mock_llm.py`).
## Objective
Store a lite-generated, plain-text outline of the knowledge base's basic **categories** in a single-row `kb_overview` table, inject it into **both** chat prompts (HIGH and LOW) as a `<knowledge_base>` section so the agent knows roughly what the KB contains before retrieval, and regenerate it automatically whenever an import changes the KB.
## Dependencies
- `30_document_summaries` (todo) — `LLMClient.chat` + `BOR_LLM_SUMMARY_MODEL` (task 01) and the stored per-document summaries (task 04) that feed the overview input.
- `15_steering_notes` (complete) — the prompt-section pattern this phase mirrors (budget, marker, byte-identical-when-absent, per-turn load in `app/api/chat.py`).
- `11_long_answers` / README import workflow (complete) — `scripts/import_docs.py`'s `main()` structure, which this phase extends with the post-import regeneration.
## Tasks
1. `01_migration_kb_overview.md` — Alembic 0005: single-row `kb_overview` table + `KbOverview` model.
2. `02_overview_generator.md` — `app/rag/overview.py`: `KB_OVERVIEW_MODE` prompt builder, `load_kb_overview`, `regenerate_overview` (best-effort upsert).
3. `03_prompt_injection.md` — `<knowledge_base>` section in HIGH + LOW prompts (budgeted, byte-identical when absent); `plan_turn`/chat wire it in; `kb_chars` in the per-turn log.
4. `04_import_trigger.md` — `import_docs` regenerates the overview after a KB-changing import (shared with phase 32's sync).
5. `05_mock_and_e2e.md` — deterministic `KB_OVERVIEW_MODE` mock + `(kb: …)` echo, `tests/e2e/test_kb_overview.py`, story file, commit.
## Testing & Quality
- Unit: overview generator (prompt build/cap, load, regenerate upsert/fail-soft/zero-docs), prompts (section present/budgeted/absent → byte-identical, ordering vs `<tuning>`), chat gate (`kb_chars`, prompt carries the section).
- Integration: migration 0005 up/down; `import_docs` regeneration trigger (changed vs unchanged imports, failure isolation).
- Coverage: **>90%** on `app/` (`app/` TOTAL ≥ pre-change).
- E2E (mandatory, A16): `tests/e2e/test_kb_overview.py` — one story, run **in isolation**; the injected section is observable in the mock answer via the `(kb: …)` echo (steering precedent).
## Completion Criteria
- [ ] After a KB-changing import, `kb_overview` holds a fresh outline (log line `overview: regenerated docs=… chars=…`); an unchanged re-import does **not** call the lite model.
- [ ] Every chat turn's system prompt (HIGH and LOW) contains the `<knowledge_base>` section when a row exists; with no row, both prompts are **byte-identical** to the pre-phase text (unit-asserted).
- [ ] The per-turn log line records `kb_chars=<n>`; section overflow beyond `BOR_KB_OVERVIEW_MAX_CHARS` is capped with the shared `[…truncated…]` marker.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
- [ ] `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` green in isolation; existing prompt/steering/chat suites stay green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/kb-overview-prompt.md` exists; `.env.example` + README document `BOR_KB_OVERVIEW_MAX_CHARS` / `BOR_OVERVIEW_INPUT_MAX_CHARS` and the regeneration behavior.
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, <knowledge_base> section in HIGH+LOW prompts`); `.agent/phases/todo/31_kb_overview_prompt/` moved to `.agent/phases/complete/`.
## Locked decisions
- **A13** — migration 0005 adds one single-row table `kb_overview` (`id INTEGER PK DEFAULT 1`, `content TEXT NOT NULL DEFAULT ''`, `updated_at TIMESTAMPTZ`); no other schema change.
- **A5 extended** — the overview is generated by the same `lite` model via the same `BOR_LLM_SUMMARY_MODEL` setting and `LLMClient.chat` (phase 30); no new model or package.
- **Prompt-section convention (phase 15 precedent)** — the section is budgeted by `BOR_KB_OVERVIEW_MAX_CHARS` (default **4000**) with the shared `TRUNCATION_MARKER` overflow; **zero/empty row → prompts byte-identical** to pre-phase text. Section order: `<relevance>` → `<knowledge_base>` → `<tuning>` → mode body.
- **Regeneration is best-effort and change-gated** — runs only when an import added/updated at least one document (or no row exists yet); a lite failure logs and leaves the previous overview intact (an old outline is better than none).
- **Overview input is capped** — `BOR_OVERVIEW_INPUT_MAX_CHARS` (default **40 000**) on the document list (source/path/title/first summary line) sent to the model.
- **No per-turn LLM call** — chat turns only *read* the stored row (one indexed PK lookup); generation happens at import/sync time (phase 32's button triggers the same `regenerate_overview`).
- **A16 / A17 honoured** — one dedicated story E2E suite; one atomic `--no-gpg-sign` commit.
@@ -1,31 +0,0 @@
# Task 05 — Deterministic KB_OVERVIEW_MODE mock + story E2E + commit
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — (whole item: system prompt injects basic categories of everything read, lite-generated, stored, updated on import)`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
## Objective
Make the overview observable end-to-end in a deterministic E2E: the mock generates a `KB_OVERVIEW_MODE` outline and echoes the injected section into its answer (the `(tuning: …)` precedent), plus the story suite, story file, and the phase commit.
## Work
1. `tests/e2e/mock_llm.py`:
- Generation: in the chat-completions handler, if the system prompt contains `KB_OVERVIEW_MODE` → return the deterministic outline `f"Knowledge base outline:\n- {first 8 tokens of _user(body), space-joined}"` (the user message carries the document list).
- Echo: in `compose_answer` (all answer paths), if the system prompt contains a `<knowledge_base>` section, append `(kb: <first bullet line of the section>)` — parse with a regex in the `first_tuning_note` style (skip the intro line, take the first `-` line, strip the dash). This mirrors the steering echo exactly.
2. `tests/e2e/test_kb_overview.py` (new, the story gate):
- Seed the `kb_overview` row directly in the DB (the E2E test has DB access via the conftest fixtures — content with a recognizable first bullet, e.g. `- Kubernetes cluster and node maintenance notes`), so the test exercises the **injection** path deterministically (the CLI trigger path is covered by task 04's integration tests).
- Ask a normal on-topic question (the fixture KB is already imported by the conftest pattern) → assert the rendered brain answer ends with `(kb: Kubernetes cluster and node maintenance notes)`.
- Deflection control: ask an off-topic question (deflection path) → the answer still carries the `(kb: …)` echo (the section is in the LOW prompt too).
- Absence control: delete the row → a fresh question's answer has **no** `(kb: …)` suffix (byte-identical prompt behavior is unit-asserted in task 03; this proves it end-to-end).
3. Integration test (same task): extend `tests/integration/test_chat_api.py` (or a new `test_kb_overview_api.py`) with the capturing-fake-LLM pattern — no row: the system prompt sent to the model equals the pre-phase construction (assert the exact string via the existing `build_high_prompt`/`build_deflect_prompt` with `kb_overview=None`); row present: it contains the section in both HIGH and LOW turns.
4. `.agent/user_stories/kb-overview-prompt.md` (new) — narrative + acceptance criteria + Playwright mapping rule, styled like the other story files.
5. Commit: `git commit --no-gpg-sign -m "feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, <knowledge_base> section in HIGH+LOW prompts"`; move `.agent/phases/todo/31_kb_overview_prompt/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` green **in isolation**.
- Regression: `test_steering.py`, `test_chat_rag.py`, `test_honest_deflection.py` stay green in isolation (prompt change is additive and defaulted).
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `tests/e2e/test_kb_overview.py` green in isolation (injection, deflection, and absence all asserted).
- [ ] Integration prompt-capture tests green; existing steering/chat suites green.
- [ ] Full test + lint/type gates green (per this phase's 00_phase.md).
- [ ] Story file + `.env.example` + README complete; one `--no-gpg-sign` commit made.
@@ -1,45 +0,0 @@
# Phase 32 — Admin Sync Button (one-click doc import sync)
**Source:** `TODO.md L5 — "Need a button that only the admin can see that triggers a doc import sync by cloning the relevant repos and then running import doc script"`
**Story:** `.agent/user_stories/admin-sync-button.md`
**Context:** Phase 28 (`scripts/git_sync.py::clone_or_pull` — shallow clone / `--ff-only` pull; `BOR_GIT_SOURCES` + `BOR_SOURCES_DIR`; `repo_name` in `scripts/import_docs.py`), phase 31 (`regenerate_overview` — the sync refreshes the KB outline), phase 16 (`require_admin` dependency + the `header.js` `fetchIsAdmin()` reveal gate for admin-only UI like `#nav-sources` / `#nav-tuning`), PLAN §7.4 "never stale" feedback contract (the UI can never sit on a stale button state).
## Objective
Give the admin a **"Sync sources"** button (Sources page, visible to the admin only) that triggers the full document sync in-process — clone/pull every `BOR_GIT_SOURCES` repo, re-import (with prune) so the KB mirrors the repos, and refresh the KB overview — with live, non-stale UI feedback driven by a polled sync-status endpoint.
## Dependencies
- `31_kb_overview_prompt` (todo) — `regenerate_overview(llm)` is the sync's final step; `LLMClient.chat` for it.
- `28_git_based_sources` (complete) — `clone_or_pull` / `GitSyncError` / `BOR_GIT_SOURCES` / `repo_name` (the sync reuses them, does not re-implement git).
- `16_admin_auth` (complete) — `require_admin` for the new endpoints; the `header.js` whoami gate for the button.
- `19_shared_header` / `29_tuning_nav_link` (complete) — the Sources page header actions area where the button lives.
## Tasks
1. `01_sync_api.md` — in-process sync runner: `POST /api/sync` (admin, 409 when running) + `GET /api/sync/status` (admin).
2. `02_ui_button.md` — the admin-only button on Sources with §7.4 feedback states (polling, last-result, error banner) + frontend unit assertions.
3. `03_e2e_and_docs.md` — `tests/e2e/test_sync_button.py` (real `file://` git fixture), README, story file, commit.
## Testing & Quality
- Integration: sync API — anonymous 403s, admin idle/running/success/failed transitions, 409 double-trigger, GitSyncError → `failed` with the repo named (git + import + overview mocked, as `test_import_docs_git.py` does).
- Unit (frontend-assertion style, cf. `tests/unit/test_shared_header.py`): button markup hidden-by-default + labeled; `header.js` reveal; `sources.js` polling/terminal-state logic.
- Coverage: **>90%** on `app/` (`app/api/sync.py` fully covered); `app/` TOTAL ≥ pre-change.
- E2E (mandatory, A16): `tests/e2e/test_sync_button.py` — one story, run **in isolation**; uses a **real** local `file://` git repo fixture (deterministic, no network) with the mock LLM for embeddings.
- UI Structure Check (AGENTS.md rule 5): labeled button, focus-visible, contrast ≥4.5:1, `aria-live` result region, no CDN.
## Completion Criteria
- [ ] Anonymous: the button is not revealed (stays `hidden`) and both endpoints return 403.
- [ ] Admin: clicking "Sync sources" starts the sync (202), the button goes disabled with "Syncing…" while polling `GET /api/sync/status` every 2 s, and on completion shows the last result (`Synced HH:MM` + `N added · M updated`); a failed sync re-enables the button with an error banner (`role="alert"`) naming the failure.
- [ ] A double trigger while running returns 409 and the UI never starts a second poll loop.
- [ ] After a successful sync against the `file://` fixture repo, the newly committed fixture doc appears in the Sources table and the `kb_overview` row is fresh (phase-31 trigger).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
- [ ] `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` green in isolation; `test_admin_auth.py`, `test_shared_header.py`, `test_import_documents.py` stay green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/admin-sync-button.md` exists; README documents the button (behavior, states, prerequisites).
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status`); `.agent/phases/todo/32_admin_sync_button/` moved to `.agent/phases/complete/`.
## Locked decisions
- **A10 extended (recorded, not a revision)** — two new **admin-only** endpoints (`POST /api/sync`, `GET /api/sync/status`) behind the existing `require_admin`; the public API surface stays stateless, the signed cookie remains the only session state (same pattern as `/api/steering`).
- **A12 untouched** — the sync runs **in-process** (one `asyncio` background task + a module-level status object in `app/api/sync.py`). The app is a single instance on the homelab; no Valkey/queue. Status is in memory — a restart mid-sync loses the running state (accepted: the next click re-syncs idempotently).
- **Sync semantics** — the button targets `BOR_GIT_SOURCES` only (manual `--source` dirs have no repo to clone; an unset/empty `BOR_GIT_SOURCES` → the sync fails loudly with "no git sources configured"); the import runs with **`prune=True`** so files deleted upstream leave the index (the button is the canonical "mirror the repos" action — the CLI default of no-prune is unchanged); phase-31's `regenerate_overview` runs after the import when docs changed.
- **Concurrency** — one sync at a time: `POST /api/sync` while running → `409 {"detail": "a sync is already running"}`; the UI reflects the in-flight run (re-attaches on page load while a sync is running).
- **§7.4 adaptation (recorded)** — the 120 s client guard applies to LLM turns; a sync can legitimately run for minutes (clone + embed), so the button has **no client-side hard timeout** — the 2 s status poll is the feedback loop and the server state is authoritative. The button is disabled until the run reaches a terminal state, so it can never be stale *or* stuck: a failed run re-enables it, a running run always shows "Syncing…".
- **A16 / A17 honoured** — one dedicated story E2E suite (real `file://` git fixture — git is a documented environment prerequisite, as in phase 28); one atomic `--no-gpg-sign` commit.
@@ -1,34 +0,0 @@
# Task 03 — Story E2E (real file:// git fixture) + README + commit
**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — (whole item: admin-only button → clone the relevant repos → run the import script)"`
**Story:** `.agent/user_stories/admin-sync-button.md`
## Objective
The story gate: a deterministic E2E that runs the **real** sync path end-to-end (real `git clone` of a local `file://` fixture repo, real import against the mock LLM, real overview regeneration) and verifies both the admin-only visibility and the full button lifecycle, plus README docs, story file, and the phase commit.
## Work
1. `tests/e2e/test_sync_button.py` (new) — module fixtures:
- Build a temp git repo in a `tmp_path` via `subprocess` (`git init -q`, write `notes/sync-fixture.md` containing a unique sentinel `RESE-SYNC-SENTINEL-9b2c`, `git add -A && git -c user.email=e@x -c user.name=t commit -qm one`); set the app's env for this module: `BOR_GIT_SOURCES=file://<repo>`, `BOR_SOURCES_DIR=<tmp_path>/checkouts` (follow the E2E conftest pattern for per-module app env; git is a documented environment prerequisite — phase 28).
- Truncate `query_log` (and the KB tables the fixture needs) per the existing E2E isolation pattern so the run starts clean.
2. Tests (in isolation):
- **`test_anonymous_sees_no_button`** — load `/sources.html` logged out: `#sync-btn` is `hidden` (or absent from the revealed DOM); `POST /api/sync` via `page.request` → 403.
- **`test_admin_sync_lifecycle`** — sign in (reuse `tests/e2e/auth_helpers.py`):
- `#sync-btn` visible with label "Sync sources".
- Click → button disabled, label "Syncing…".
- Wait (poll with Playwright, generous timeout ~60 s — real git + embed against the mock LLM): label becomes `Synced …`, `#sync-result` shows `1 added` (the fixture doc).
- The Sources table (`#docs-tbody`) now contains the fixture path `notes/sync-fixture.md`; the `kb_overview` row is non-empty (phase-31 regeneration ran — DB check in the test).
- Re-click → a second run completes with `0 added · 1 unchanged` (idempotent pull + hash-skip).
- **`test_double_trigger_409`** (integration-level, may live in `tests/integration/test_sync_api.py` if E2E timing is too flaky — the integration test already covers this; include here only if deterministic): start a sync, immediately `POST /api/sync` again → 409.
3. `README.md` — new "Sync from the UI" subsection under the import workflow: what the button does (clone/pull → import `--prune`-equivalent → KB overview refresh), the states (Syncing…/Synced/error), the 409 behavior, and prerequisites (`BOR_GIT_SOURCES` set; git available).
4. `.agent/user_stories/admin-sync-button.md` (new) — narrative + acceptance criteria + Playwright mapping rule.
5. Commit: `git commit --no-gpg-sign -m "feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status"`; move `.agent/phases/todo/32_admin_sync_button/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` green **in isolation** (Chromium + `podman compose up -d db` + git on PATH; mock LLM — no live aipi).
- Regression in isolation: `test_admin_auth.py`, `test_shared_header.py`, `test_import_documents.py`, `test_global_tuning.py`.
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] All E2E tests green in isolation, including the idempotent second run and the `kb_overview` freshness check.
- [ ] Regression suites green; full test + lint/type gates green (per this phase's 00_phase.md).
- [ ] README + story file complete; one `--no-gpg-sign` commit made.
@@ -1,40 +0,0 @@
# Phase 33 — Cache Busting (un-stick the pages)
**Source:** `TODO.md L6 — "We need better cache busting, the pages are too sticky"`
**Story:** `.agent/user_stories/cache-busting.md`
**Context:** `app/main.py` serves the whole `frontend/` directory through one `StaticFiles(html=True)` catch-all mount; the five HTML pages reference assets **without any version** (`href="/assets/styles.css"`, `src="assets/markdown.js"`, `src="/assets/app.js"`, …), so browsers happily keep stale CSS/JS/HTML after a deploy — the "too sticky" report. The no-CDN integration test (`tests/integration/test_api.py::test_html_pages_served_locally_no_cdn`) asserts no `https://` references — appending `?v=` keeps every reference same-origin, so it stays green. SSE/API live under `/api/*` and must be untouched (SSE already ships `Cache-Control: no-cache` itself).
## Objective
A deploy must be visible without a hard refresh: HTML pages are **always revalidated** (`Cache-Control: no-cache`) and reference their assets with a version token (`?v=<token>`); assets are served **immutable for 1 year** (the token in the URL identifies the content, so long caching is safe). Zero new services, zero build-step changes, no CDN.
## Dependencies
- `32_admin_sync_button` (todo) — sequencing only; no shared code (this phase is transport-layer and independent of the RAG work).
## Tasks
1. `01_asset_version_token.md` — `app/core/caching.py::asset_version()`: git short SHA (homelab checkouts have a `.git`), stable mtime+size content-hash fallback, computed once per process.
2. `02_caching_middleware.md` — the response middleware (HTML `no-cache` + `?v=` rewrite; `/assets/*` immutable) wired into `create_app` + integration tests.
3. `03_e2e_and_docs.md` — `tests/e2e/test_cache_busting.py` (Playwright header assertions), README, story file, commit.
## Testing & Quality
- Unit: version token (git path, fallback path, failure path), the asset-reference rewrite (both `href`/`src` and leading-slash-less `assets/…` refs, no double-`?v=`).
- Integration: page headers + rewritten references; asset headers; no-CDN test green; SSE endpoint responses untouched (existing chat SSE tests green).
- Coverage: **>90%** on `app/` (the new `app/core/caching.py` fully covered); TOTAL ≥ pre-change.
- E2E (mandatory, A16): `tests/e2e/test_cache_busting.py` — one story, run **in isolation**; real Chromium asserting the headers and the versioned request URLs the browser actually makes.
## Completion Criteria
- [ ] Every HTML page (`/`, `/sources.html`, `/document.html`, `/login.html`, `/tuning.html`) is served with `Cache-Control: no-cache` and its asset references carry `?v=<token>` (token non-empty, stable across requests, changes when the frontend content changes).
- [ ] `/assets/*` responses carry `Cache-Control: public, max-age=31536000, immutable`.
- [ ] `/api/*` (incl. the SSE chat stream) responses are byte-for-byte header-wise unaffected beyond what they already send; no-CDN integration test green.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
- [ ] `uv run pytest tests/e2e/test_cache_busting.py -v --no-cov` green in isolation; `test_smoke.py` + one RAG E2E stay green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/cache-busting.md` exists; README documents the caching behavior + how the token changes on deploy.
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `perf(ui): cache busting — HTML no-cache + versioned asset URLs (?v=) with immutable 1y asset caching`); `.agent/phases/todo/33_cache_busting/` moved to `.agent/phases/complete/`.
## Locked decisions
- **Version token** — `asset_version()`: if the project checkout has a `.git` (the homelab reality), the token is `git rev-parse --short HEAD` (a commit = a deploy, so the token flips on every deploy); otherwise a stable hash of the frontend tree (sorted `relpath + mtime_ns + size`, first 12 hex chars) so dev checkouts still bust. Computed **once per process** (`lru_cache`) — zero per-request git/file cost.
- **Rewrite scope** — only the five known HTML pages are rewritten (a small regex over `href="…assets/…"` / `src="…assets/…"` appending `?v=` when absent). No templating layer, no build step, no changes to the static files themselves (the `Containerfile` esbuild stage is untouched).
- **Asset caching** — `/assets/*` are cached `immutable` for 1 year **because** the URL carries the token; the unversioned path keeps working (StaticFiles ignores the query string), so old tabs and tests referencing `/assets/x.js` directly still resolve.
- **Middleware boundary** — the middleware touches exactly two shapes: the five page paths (body rewrite + `no-cache`) and `/assets/*` (header only). Everything else — all `/api/*` including SSE — passes through byte-identical (SSE keeps its own `no-cache`). Implemented as a Starlette middleware that only rewrites `text/html` responses under the page paths; if `Response.body()` turns out to misbehave on the `FileResponse` streaming path, the fallback is five explicit FastAPI routes that read + rewrite the files (identical observable behavior — the executor picks whichever passes the tests).
- **A11 untouched** — no CDN, no new packages, no new services (A12 untouched).
- **A16 / A17 honoured** — one dedicated story E2E suite; one atomic `--no-gpg-sign` commit.
@@ -1,32 +0,0 @@
# Task 03 — Story E2E (Playwright header assertions) + README + commit
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — (whole item: better cache busting — the pages are too sticky)"`
**Story:** `.agent/user_stories/cache-busting.md`
## Objective
The story gate: a real-browser E2E that asserts what the browser actually receives (HTML `no-cache`, versioned asset request URLs, immutable asset headers), plus README docs, story file, and the phase commit.
## Work
1. `tests/e2e/test_cache_busting.py` (new) — collect responses with `page.on("response")`:
- `test_html_pages_are_no_cache_and_versioned` — navigate to `/`:
- the document response's `cache-control` header is `no-cache`;
- the `styles.css` request URL contains `?v=` and the response's `cache-control` contains `immutable` + `max-age=31536000`;
- the `app.js` request URL contains the **same** token value as the CSS one (single token per process);
- the served HTML (`page.content()`) contains no unversioned `/assets/styles.css"` reference.
- `test_other_pages_share_the_token` — navigate to `/sources.html` then `/login.html`: each document response is `no-cache`; both pages' CSS requests carry the same token.
- `test_api_responses_unaffected` — `page.request.get("/api/health")` → no `cache-control: no-cache`/immutable injection (the endpoint's baseline headers only); a chat SSE POST still streams (reuse the minimal chat-request helper from an existing E2E — the stream must complete with `done`).
2. `README.md` — new short "Caching / deploys" section: HTML is always revalidated; assets are cached 1 year immutable and carry `?v=<token>`; the token is the git short SHA (falls back to a content hash in non-git checkouts) and flips on every commit/deploy — no hard refresh needed anymore; API/SSE caching is unchanged.
3. `.agent/user_stories/cache-busting.md` (new) — narrative + acceptance criteria + Playwright mapping rule.
4. Commit: `git commit --no-gpg-sign -m "perf(ui): cache busting — HTML no-cache + versioned asset URLs (?v=) with immutable 1y asset caching"`; move `.agent/phases/todo/33_cache_busting/` → `.agent/phases/complete/`.
5. **Deploy note (post-commit, owner action)**: after this phase lands, the *first* deploy also requires browsers to see the new HTML once (revalidation) — one normal navigation; thereafter every commit is picked up automatically.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_cache_busting.py -v --no-cov` green **in isolation** (Chromium + `podman compose up -d db` for the app boot; the mock LLM keeps the SSE check deterministic).
- Regression in isolation: `test_smoke.py`, `test_chat_rag.py`.
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] All E2E tests green in isolation (headers + token consistency + SSE unaffected).
- [ ] Regression suites green; full test + lint/type gates green (per this phase's 00_phase.md).
- [ ] README + story file complete; one `--no-gpg-sign` commit made.
- [ ] `podman compose up -d` (full app) smoke: a fresh browser profile loads the site and every asset request is versioned (manual confirmation recorded in the commit message or phase notes).
@@ -1,47 +0,0 @@
# Phase 35 — Admin Page to Add / Remove Git Sources
**Source:** `TODO.md` L4 — "I need a page only the admin can access where I can add and remove git sources for docs"
**Story:** `.agent/user_stories/git-sources-admin.md`
**Context:** Phase 28 introduced git-based sources (`BOR_GIT_SOURCES` env var + `scripts/git_sync.clone_or_pull`) and phase 32 the one-click Sync button (`POST /api/sync`) — but the *list itself* can only be changed by editing `.env` and restarting. This phase makes the list admin-managed: a Postgres-backed table, an admin-only CRUD API, and a dedicated admin page, with the sync pipeline and `import_docs` resolving the effective list from the DB (env var demoted to an empty-table fallback).
## Objective
Deliver a page **only the admin can access** (`/git-sources.html`, soft-gated like Sources) to **add and remove git sources**, stored in a new `git_sources` table; the Sync button (phase 32) and `import_docs` (phase 28) use the stored list, `BOR_GIT_SOURCES` remains a fallback while the table is empty, and phase 32's fail-loud "no git sources configured" is preserved when both are empty.
## Dependencies
- `28_git_based_sources` (complete) — `scripts/git_sync.clone_or_pull`, `repo_name`, the `BOR_GIT_SOURCES` settings + `git_source_list`, the import resolution order (`--source` wins).
- `32_admin_sync_button` (complete) — the `POST /api/sync` / `GET /api/sync/status` pipeline this phase re-points at the DB list; the Sync button the page's hint refers to.
- `16_admin_auth` (complete) — `require_admin` (the router-level pattern from `app/api/sync.py`), the soft-gate page pattern (`sources.html`), the `fetchIsAdmin()` frontend gate.
- `34_consistent_navbar` (todo) — the identical five-page header this phase's admin-only "Git sources" nav link plugs into (phase 29 pattern).
- `29_tuning_nav_link` (complete) — the admin-only ship-hidden nav-link pattern to copy.
## Tasks
1. `01_model_and_migration.md` — `GitSource` model + migration `0006_git_sources.py` (reversible).
2. `02_git_sources_api.md` — admin-only `GET/POST /api/git-sources` + `DELETE /api/git-sources/{id}` with validation, the env-fallback listing, and the integration suite.
3. `03_sync_and_importer_use_db.md` — `effective_git_sources()` shared by `app/api/sync.py` and `scripts/import_docs.py` (DB wins, env fallback, fail-loud unchanged) + test updates.
4. `04_admin_page.md` — `/git-sources.html` + `git-sources.js` (soft-gated, list / add / remove, env note, sync hint) + styles.
5. `05_nav_link.md` — the admin-only "Git sources" nav link on all five pages + the `header.js` reveal.
6. `06_e2e_and_docs.md` — the story E2E suite `test_git_sources_admin.py`, `test_nav_consistency.py` nav-inventory update, README/`.env.example` notes, regressions, commit.
## Testing & Quality
- Unit/integration: `tests/unit/` for `effective_git_sources` (DB-wins / env-fallback / both-empty); `tests/integration/test_git_sources_api.py` for the CRUD contract (403/201/409/422/404, env fallback); the migration up/down test following the 0004/0005 pattern; the existing `test_sync_api.py` + `test_import_docs_git.py` suites stay green with the resolution indirection.
- Coverage: **>90%** on `app/` for the new module + API.
- E2E (mandatory, A16): `tests/e2e/test_git_sources_admin.py` — the story gate, run in isolation.
## Completion Criteria
- [ ] Migration 0006 applied (`uv run alembic upgrade head`); `git_sources` table exists with `url` unique.
- [ ] `GET /api/git-sources` (admin) lists DB rows; while the table is empty it returns the env list with `from_env: true`; anonymous gets 403 on all three routes.
- [ ] `POST` creates (201, trimmed, shape-validated, 409 duplicate without echoing the URL); `DELETE` removes (204/404).
- [ ] `POST /api/sync` and `import_docs` resolve the list via `effective_git_sources` (origin logged `db|env`); both-empty still fails loudly; `--source` override unchanged.
- [ ] `/git-sources.html`: anonymous sees the sign-in gate; the admin sees list + add + remove with a never-stale button and inline errors; the admin-only "Git sources" nav link is visible on all five pages for the admin and hidden for anonymous.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov` green in isolation; regressions (task 06 list) green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
## Locked decisions
- **A3 / A13 honoured** — the list lives in Postgres via Alembic (no JSON file, no new store).
- **A10 extended per the phase-16 pattern** — a new admin-only router behind `require_admin`; the public API surface stays stateless; no new auth mechanism.
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages.
- **Env var demoted, not removed** — `BOR_GIT_SOURCES` keeps working exactly as today while the table is empty (the fallback); once the table has rows it is ignored (the UI is the source of truth). Phase 32's fail-loud empty-config behavior is preserved.
- **Scope boundary** — adding/removing a repo does NOT immediately clone, import, or prune: the existing Sync button performs that (removal prunes on the next sync, `prune=True`). The page's hint says so.
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
@@ -1,46 +0,0 @@
# Task 01 — GitSource model + migration 0006
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
Add the `git_sources` table (one row per admin-managed repo URL) via the model + a reversible Alembic migration, following the exact conventions of migrations 0003–0005.
## Work
1. `app/models.py` — add the model (next to `SteeringNote`, docstring citing this phase + the A13 convention):
```python
class GitSource(Base):
"""One admin-managed git source (phase 35).
The UI-maintained list of repo URLs the Sync button (phase 32) and
import_docs (phase 28) clone/pull. DB rows win over the
BOR_GIT_SOURCES env var, which is a fallback while this table is
empty (see app.rag.git_sources.effective_git_sources).
"""
__tablename__ = "git_sources"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
)
url: Mapped[str] = mapped_column(Text, unique=True, nullable=False)
added_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now()
)
```
(Import `Text` — already imported in the file; verify.)
2. `alembic/versions/0006_git_sources.py` — new migration:
- Read `alembic/versions/0005_kb_overview.py` first and chain from its actual `revision` id (the filenames are not the revision ids).
- `upgrade()`: `CREATE TABLE git_sources (id UUID PRIMARY KEY, url TEXT NOT NULL, added_at TIMESTAMP WITH TIME ZONE DEFAULT now() NOT NULL)` + `CREATE UNIQUE INDEX uq_git_sources_url ON git_sources (url)` (use `sa.Uuid` / the same column types the other migrations use — mirror their style, including `op.create_table` kwargs and the `UniqueConstraint`-vs-index choice 0003/0004 made).
- `downgrade()`: drop the index + table.
3. Apply it to the dev database: `podman compose up -d db` (if needed) then `uv run alembic upgrade head`.
4. Migration test — follow the existing pattern (see how 0004/0005 are integration-tested — `tests/integration/` migration suite): assert 0006 up creates the table + unique constraint and down drops it (round-trip on the test DB).
## Testing & Quality
- Unit/integration: the migration up/down test above; `uv run pytest` green overall.
- Coverage: model-only for now — the `app/` gate stays >90% (models are thin).
## Completion Criteria
- [ ] `alembic/versions/0006_git_sources.py` exists, chains off 0005's real revision id, and is reversible.
- [ ] `uv run alembic upgrade head` applies cleanly on the dev DB; `git_sources` visible (`\d git_sources` equivalent).
- [ ] The 0006 up/down integration test passes; full `uv run pytest` green.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,39 +0,0 @@
# Task 02 — Admin-only git sources CRUD API
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
The admin CRUD contract for the stored list: `GET /api/git-sources` (DB rows, or the env fallback while the table is empty), `POST /api/git-sources` (validated create), `DELETE /api/git-sources/{id}` — all behind `require_admin`, exactly like `app/api/sync.py`.
## Work
1. `app/schemas.py` — add (mirroring the steering schemas' style):
- `GitSourceIn` — `url: str = Field(min_length=1, max_length=500)` + a `mode="before"` trim validator (whitespace-only → 422, same trick as `SteeringNoteIn`).
- `GitSourceOut` — `id: uuid.UUID | None`, `url: str`, `added_at: datetime | None` (both nullable: env-fallback rows carry neither).
- `GitSourceList` — `sources: list[GitSourceOut]`, `from_env: bool` (`True` only when the table is empty and the list comes from `BOR_GIT_SOURCES`).
2. `app/api/git_sources.py` (NEW) — `router = APIRouter(prefix="/git-sources", tags=["git-sources"], dependencies=[Depends(require_admin)])` (copy the sync.py pattern + docstring style):
- `GET ""` → `GitSourceList`: DB rows ordered by `(added_at, id)`; if the table is empty → the `get_settings().git_source_list` env URLs as rows with `id=None, added_at=None` and `from_env=True`; `from_env=False` whenever DB rows exist (the env var is then ignored — the phase's locked decision).
- `POST ""` (201) → create:
- Shape validation (module-level `URL_RE = re.compile(r"^(https?://|ssh://|git@)")` with a docstring): the trimmed URL must match — covers the phase-28 real URLs (HTTPS + `git@` SSH); scp-style `host:repo` is deliberately rejected (422).
- ASSUMPTION: the accepted shapes are exactly `http://`, `https://`, `ssh://`, `git@…`; the 422 detail is generic ("not a valid git URL (expected https://, ssh:// or git@…)") and never echoes the input.
- Duplicate (same trimmed URL already stored) → 409 with a generic detail ("a git source with this URL already exists") — **never echo the URL** (URLs may embed `user:pass@` credentials; phase 32's masking discipline).
- Success → insert, commit, return the created `GitSourceOut`.
- `DELETE "/{source_id}"` → 204; unknown id → 404 `git source not found`.
3. `app/main.py` — register the router alongside the existing `include_router` calls (check how `sync` is included and mirror it).
4. `tests/integration/test_git_sources_api.py` (NEW) — follow `tests/integration/test_sync_api.py`'s auth pattern (`_login` via `POST /api/login` with the fixture admin password, admin client as context manager):
- Anonymous → 403 `{"detail": "admin only"}` on GET, POST, and DELETE.
- Admin + empty table + env set (monkeypatch the settings `git_sources`) → GET returns the env rows, `from_env=True`, null ids.
- Admin + empty table + env empty → GET returns `sources=[]`, `from_env=True`.
- POST: valid `https://…` → 201 + the row appears in GET with `from_env` now `False`; a duplicate → 409 and the detail contains no URL; an invalid shape (`not a url`, `host:repo`) → 422; whitespace-only / >500 chars → 422; a `git@github.com:…` URL → 201 (accepted).
- DB rows win over env: seed a row AND set the env → GET returns only the DB rows, `from_env=False`.
- DELETE: known id → 204 + gone from GET (back to env fallback if the table is now empty); unknown id → 404.
## Testing & Quality
- Integration: the suite above — the full CRUD contract, auth split, fallback semantics.
- Coverage: **>90%** on the new module (`app/api/git_sources.py` + schemas).
## Completion Criteria
- [ ] All three routes exist under `/api/git-sources`, admin-only (403 anonymous), registered in `app/main.py`.
- [ ] `tests/integration/test_git_sources_api.py` green; `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
- [ ] No credential-echo path: 409/422 details never contain the submitted URL (a test asserts this).
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,38 +0,0 @@
# Task 03 — Sync + import_docs resolve the effective list (DB wins, env fallback)
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
One shared resolver — DB rows win, `BOR_GIT_SOURCES` is a fallback only while the table is empty, fail-loud unchanged when both are empty — used by **both** the in-app sync pipeline (`app/api/sync.py::_run_sync`) and the CLI (`scripts/import_docs.py`), so the admin page's list is what actually gets cloned and indexed.
## Work
1. `app/rag/git_sources.py` (NEW) — the single resolver (importable by both the app and the CLI — `scripts` already imports `app.rag.*`):
```python
def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]:
"""(urls, origin) — DB rows in (added_at, id) order win; while the
table is empty the BOR_GIT_SOURCES env list is the fallback; both
empty → ([], "env")."""
```
- DB rows: `select(GitSource).order_by(GitSource.added_at, GitSource.id)`.
- Fallback: `get_settings().git_source_list` (the phase-28 CSV parse — reuse, don't re-implement).
2. `app/api/sync.py` — `_run_sync` replaces `settings.git_source_list` with the resolver:
- Open a `SessionLocal()` (close in `finally`) around the resolution — the background task has no request session.
- Log the origin: the existing `sync: started repos=N` line gains `origin=db|env`.
- Both-empty: keep the fail-loud `GitSyncError("no git sources configured …")` (extend the message to mention both origins — e.g. `(git_sources table empty and BOR_GIT_SOURCES unset)`; if `tests/integration/test_sync_api.py::test_no_git_sources_configured_fails_loudly` asserts the old text, update that expectation — it is this phase's file to update).
3. `scripts/import_docs.py` — the git-URL resolution branch (today `settings.git_source_list`) resolves via the same function: open a short `SessionLocal()` at resolution time (the import needs the DB anyway — no DB-down fallback to design). The `--source` override still wins (manual mode), the log line records the origin (`git sources: N repo(s) origin=db|env` before the clone loop).
4. **Test updates:**
- `tests/unit/` — NEW unit tests for `effective_git_sources`: DB rows win (seeded table + env set → DB list, origin `db`); env fallback (empty table + env set); both empty → `([], "env")`. (Use a test DB session or a stubbed session following the existing unit-test conventions.)
- `tests/integration/test_sync_api.py` — where it stubs `settings.git_source_list` to drive sync scenarios, keep those scenarios working through the new indirection: either seed the `git_sources` table or monkeypatch `effective_git_sources` (whichever the file's existing fixture style favors); add one scenario asserting a DB row is used over the env when both are set (clone/import mocked — the file already mocks them).
- `tests/integration/test_import_docs_git.py` — same treatment for the CLI path (`--source` override scenario untouched); add the DB-over-env scenario at the CLI level.
## Testing & Quality
- Unit: `effective_git_sources` — all three branches.
- Integration: sync + import_docs suites green with the resolver in the path, incl. the new DB-over-env scenarios and the updated fail-loud expectation.
- Coverage: **>90%** on the new module + the modified call sites.
## Completion Criteria
- [ ] `effective_git_sources` exists in `app/rag/git_sources.py` and is the ONLY place (besides the API's GET fallback, which may call it too) that combines DB + env.
- [ ] `_run_sync` and `import_docs` both resolve through it; origin visible in their logs.
- [ ] Both-empty still raises the fail-loud error (sync) / the CLI's existing no-sources behavior (import_docs) — assertions kept/updated.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
@@ -1,41 +0,0 @@
# Task 04 — The /git-sources.html admin page
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
The page itself: a soft-gated (anonymous → sign-in gate, exactly like `sources.html`) full-width manager with an add form, a sources list with per-row Remove, an env-fallback note, and a hint pointing at the Sync button. Never-stale buttons, inline errors, WCAG 2.1 AA basics, dark tech theme.
## Work
1. `frontend/git-sources.html` (NEW) — modeled on `frontend/sources.html` (same frame, same gate pattern, the standard full header from phase 34):
- Head: same meta/favicon/stylesheet pattern; `<title>Git sources · Brain of Reese</title>`; skip-link; the identical header block (nav [Chat, Sources, Tuning] + Tuning toggle + `#sync-btn` + New chat + auth pair — the "Git sources" nav link itself arrives in task 05, so this file lands without it for now).
- `<main id="main">` — `#steering-panel` first (phase 34 contract), then:
- **Gate** `#git-sources-gate` — the `#sources-gate` soft-gate markup pattern (sign-in card + link `/login.html?next=/git-sources.html`), visible for anonymous, hidden for admin.
- **Content** `#git-sources-content` (hidden until admin):
- `.page-head` — `<h1>Git sources</h1>` + sub: "The repositories the Sync button clones and indexes. Add or remove them here — no `.env`, no restart."
- **Env note** `#git-sources-env-note` (hidden by default; shown when the API returns `from_env: true`): "These sources currently come from `BOR_GIT_SOURCES` in `.env` — adding or removing one here switches management to the database."
- **Add form** `#git-source-form` — visible label (or visually-hidden label per the tuning-page pattern — use a visible `<label for="git-source-url">Add a git source</label>`), input `#git-source-url` (type text, `maxlength="500"`, `autocomplete="off"`, placeholder `https://github.com/you/homelab.git`, `required`), submit button `#git-source-add` ("Add source"), error line `#git-source-error` (`role="alert"`, hidden) — §7.4 never-stale: the button disables + label changes while the POST is in flight, re-enables on success/failure (the form is kept on failure, same as the tuning forms).
- **List** `#git-sources-list` (a full-width table or list per §7.1 — **no skinny single-column list**: use the Sources-page table pattern — columns: URL (mono `<code>`), Added, actions) + empty state `#git-sources-empty` ("No git sources stored yet." — and, with `from_env`, the env note already explains where the active list comes from).
- **Hint box** (`role="note"`): "Use the **Sync sources** button in the header (or on the Sources page) to clone the repos and refresh the index — removing a repository prunes its documents from the index on the next sync."
2. `frontend/assets/git-sources.js` (NEW) — the page module (loaded `type="module"`, imports `./header.js` like its siblings):
- Boot: `const admin = await initSharedHeader()` (one cached whoami) — anonymous → show the gate, stop; admin → hide the gate, `loadSources()`.
- `loadSources()` — `GET /api/git-sources` → render the table rows (`textContent` only — URLs may contain credentials; never innerHTML the URL), the added date (localized, `—` for null), the per-row Remove button (`.git-source-remove`, `aria-label="Remove git source: <url>"`), the env note's `hidden` on `from_env`, the empty state. Non-2xx → the content area shows a `role="alert"` error state with a retry (never a stuck page).
- Add submit — client-side non-empty check; disable `#git-source-add` (label "Adding…"); `POST /api/git-sources` with `{url}`; success → clear the input, re-enable (label "Add source"), `loadSources()`, focus the new row (a11y); failure → `#git-source-error` with the server detail (422 shape-aware like the tuning forms), re-enable, input kept.
- Remove click — `window.confirm("Remove this git source from the list? Its documents stay indexed until the next sync prunes them.")` — cancel → nothing; ok → disable the row button, `DELETE /api/git-sources/{id}`, `loadSources()`; failure → row error state + re-enable.
- Focus management + keyboard: all controls focus-visible (theme CSS covers it), the list rows' buttons are real `<button>`s.
3. `frontend/assets/styles.css` — the `.git-source-*` rules + gate reuse:
- The table: full-width in the 72rem container (the Sources-page table styles are a good starting point — reuse classes where they fit), mono URL cells with horizontal scroll on overflow (long URLs with credentials), rows ≥44px touch targets, `:focus-visible` 3px outline.
- Env note: an info chip in the theme palette (brand-soft `#232b52` surface, brand-ink `#a5b4fc` text ≈6.9:1); hint box: the page-sub styling family; error/alert states reuse the existing `#fca5a5`/`#2d1318` error treatment.
- Gate: reuse the `#sources-gate` styles (the page is the same shape as Sources — one gate visual language).
4. `frontend/assets/header.js` — no change (the page's controls are the standard shared ones; `initSharedHeader` already handles everything that ships in the header).
## Testing & Quality
- Frontend-only — no Python change this task; the no-CDN integration test must still pass (same-origin markup; the E2E in task 06 exercises the page).
- Coverage: `app/` gate unaffected.
- Manual smoke (dev server, signed in): add a real-looking URL → row appears; remove → confirm → row gone; invalid URL → inline 422 error, button re-enabled; signed out → gate only.
## Completion Criteria
- [ ] `/git-sources.html` served at the route; anonymous sees only the gate (list/form absent or inert); admin sees list + form + env note + hint.
- [ ] Add / remove round-trip works against the task-02 API; every in-flight state disables its control and re-enables on resolution (never stale); errors are inline `role="alert"`.
- [ ] UI Structure Check (AGENTS.md rule 5): landmarks, labeled controls, contrast ≥4.5:1, focus-visible; full-width table (no skinny list); no CDN (rule 6).
- [ ] `uv run pytest` green (no-CDN test); `uv run ruff check . && uv run pyright` clean.
@@ -1,32 +0,0 @@
# Task 05 — Admin-only "Git sources" nav link on all five pages
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
Make the new page reachable from everywhere: the admin-only **"Git sources"** nav link in all five identical headers (phase 34's contract), revealed for the admin by `header.js` — the exact phase-29 pattern.
## Work
1. The five page headers — `frontend/index.html`, `frontend/sources.html`, `frontend/document.html`, `frontend/tuning.html`, `frontend/git-sources.html` — inside `<nav class="app-nav" aria-label="Primary">`, **immediately after** the `#nav-sources` link, add (mirroring the `#nav-tuning` markup, comment citing this phase + owner permission 2026-08-26):
```html
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Git sources</a>
```
- Rules (phase-29 contract): `hidden` by default on every page (anonymous-safe ship-hidden); NO `is-active` / `aria-current` on the four pages that aren't the Git sources page.
- **Exception:** on `frontend/git-sources.html` the link carries `class="nav-link is-active"` + `aria-current="page"` (the current page, like Tuning on `tuning.html`).
- Nav order on every page becomes: Chat, Sources, **Git sources**, Tuning.
2. `frontend/assets/header.js` — next to the `navTuning` reveal block, add the same ship-hidden/reveal-for-admin contract:
```js
const navGitSources = document.querySelector("#nav-git-sources");
if (navGitSources) navGitSources.hidden = !admin;
```
Update the file-header comment (the admin-only link list now includes Git sources).
## Testing & Quality
- Frontend-only — no Python change; the no-CDN integration test is unaffected (same-origin `<a>`).
- Coverage: `app/` gate unaffected.
- Manual smoke: admin sees "Git sources" on all five pages → each navigates to `/git-sources.html` (with `is-active` there); anonymous never sees it (ships hidden, no flash).
## Completion Criteria
- [ ] All five headers contain `#nav-git-sources` after `#nav-sources`, `hidden` by default; `git-sources.html`'s carries `is-active` + `aria-current="page"`.
- [ ] `header.js` reveals it for the admin on the cached whoami (no extra request) and hides it for anonymous on every page.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,37 +0,0 @@
# Task 06 — Story E2E + nav-inventory update + docs + commit
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
Prove the story end-to-end with its dedicated Playwright suite, keep phase 34's nav-consistency contract in sync with the new link, document the env-var demotion, and close the phase with the full gate + one commit.
## Work
1. `tests/e2e/test_git_sources_admin.py` (NEW — the story gate, run in isolation). Fixtures: the standard E2E app + DB (`tests/e2e/conftest.py`); admin session via `tests/e2e/auth_helpers.py`; **no git, no network** — this suite is UI + API only (the clone/import path is mocked at the integration level, task 03).
- **Anonymous:** `/git-sources.html` shows the sign-in gate (list + add form absent/inert); `#nav-git-sources` hidden on the five pages; the API 403s (assert via the page context's `fetch` or `context.request` — follow `test_admin_auth.py`'s pattern for anonymous API assertions).
- **Admin — nav:** `#nav-git-sources` visible on all five pages; clicking it from `/` lands on `/git-sources.html` with the link `is-active`.
- **Admin — list:** seed two rows via the API (or `SessionLocal`) before load → both rows render (mono URL text, added date); the env note is hidden (DB rows exist).
- **Admin — add:** submit `https://example.com/reese/new-repo.git` → the row appears, the input clears, the button re-enables (never stale); submit a duplicate → inline `role="alert"` error, no new row, button re-enabled; submit `not a valid url` → inline 422 error, button re-enabled.
- **Admin — remove:** click a row's Remove → accept the confirm dialog (Playwright `page.on("dialog")`) → the row disappears; cancel a second removal → the row stays.
- **Admin — env fallback:** truncate `git_sources`, set the E2E app's `BOR_GIT_SOURCES` (follow how `test_sync_button.py` controls the env on the app fixture), reload → the env rows render + `#git-sources-env-note` visible.
- The sync-origin behavior (DB over env in the pipeline) is integration-level (task 03) — do not trigger a real sync in this suite.
2. `tests/e2e/test_nav_consistency.py` (phase 34 — UPDATE): the admin nav inventory now includes **"Git sources"** (four links, order Chat, Sources, Git sources, Tuning); the anonymous hidden set gains `#nav-git-sources`; the per-page inventory comparison stays order-sensitive.
3. **Docs:**
- `.env.example` — the `BOR_GIT_SOURCES` comment: now the **empty-table fallback**; the primary management UI is the admin Git sources page (phase 35).
- `README.md` — the import/update workflow section: the git-sources list is managed on the admin page (stored in Postgres); `BOR_GIT_SOURCES` only applies while that list is empty; `--source` still overrides for manual runs.
4. **Regression pass — each in isolation** (`uv run pytest tests/e2e/<file>.py -v --no-cov`): `test_git_sources_admin.py` (new), `test_nav_consistency.py` (updated), `test_sync_button.py`, `test_shared_header.py`, `test_header_consistency.py`, `test_tuning_nav_link.py`, `test_smoke.py`.
5. Full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (>90%), `uv run ruff check . && uv run pyright`.
6. **UI Structure Check** (AGENTS.md rule 5) on the new page (full-width table, labels, contrast, focus-visible, aria-live on the list updates) + no CDN (rule 6).
7. **Commit** (A17): stage this phase's files (`app/**`, `alembic/**`, `frontend/**`, `tests/**`, `README.md`, `.env.example`), message `feat(sources): admin page to add and remove git sources (TODO.md L4)`, always `--no-gpg-sign`. Move `.agent/phases/todo/35_git_sources_admin/` to `.agent/phases/complete/`.
## Testing & Quality
- E2E: `tests/e2e/test_git_sources_admin.py` green **in isolation** (A16: one story, one file).
- Unit/integration: from tasks 01–03 — all green under `uv run pytest`.
- Coverage: **>90%** on `app/` (new API module + resolver fully covered).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov` green in isolation.
- [ ] `test_nav_consistency.py` updated for the fourth nav link and green; every suite in the task 06 regression list green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] README + `.env.example` document the fallback semantics.
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
@@ -1,40 +0,0 @@
# Phase 36 — Document Summary Shown Together With the Original
**Source:** `TODO.md` L5 — "When I click on a document with a summary I should be able to see the summary and the original document together."
**Story:** `.agent/user_stories/summary-in-viewer.md`
**Context:** Phase 30 stores a lite-model summary on `documents.summary` for every non-markdown document (plus an indexed `is_summary` chunk) — but the viewer never shows it: `GET /api/documents/content` omits the field and the shared renderer `renderDocument` (`frontend/assets/document.js`, used by BOTH the full-page viewer `document.html` and the chat/sources modal, phase 26) only renders the raw content. Markdown documents carry no summary (phase 30) and must render exactly as before.
## Objective
When a document **has** a summary, show it and the original content together — a labeled Summary panel above the content, on both viewer surfaces at once (shared renderer); documents without a summary are unchanged.
## Dependencies
- `30_document_summaries` (complete) — the `documents.summary` column (migration 0004), the summarizer, the `summary_kb` E2E fixture + the deterministic mock `SUMMARY_MODE` digest.
- `10_story_document_viewer` + `26_document_modal_viewer` (complete) — the viewer page, the modal, and the shared `renderDocument(doc, {titleEl, metaEl, contentEl})` core both surfaces render through.
- `16_admin_auth` (complete) — the soft rule this phase must not touch: the content endpoint stays public + stateless (catalog gated, viewer public).
## Tasks
1. `01_content_api_summary_field.md` — `DocContent.summary` + the endpoint returns it; integration tests.
2. `02_viewer_summary_panel.md` — the shared renderer draws the `.doc-summary` panel (both surfaces) + theme-matched styles.
3. `03_e2e_and_regression.md` — the story E2E suite `test_summary_in_viewer.py`; regressions; full gate; commit.
## Testing & Quality
- Unit/integration: the content endpoint returns the summary for a summarized non-markdown doc and `null` for a markdown doc; anonymous access unchanged.
- Coverage: **>90%** on `app/` (the touched endpoint stays covered).
- E2E (mandatory, A16): `tests/e2e/test_summary_in_viewer.py` — the story gate, run in isolation.
## Completion Criteria
- [ ] `GET /api/documents/content` returns `summary` (string or null); no auth/shape change beyond the added nullable field; the endpoint is still public.
- [ ] A summarized document shows the labeled Summary panel **above** the original content in the full-page viewer AND the modal; the original content (including content the summary digest doesn't contain) is fully visible.
- [ ] A markdown document (no summary) renders exactly as before on both surfaces — no empty panel.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run pytest tests/e2e/test_summary_in_viewer.py -v --no-cov` green in isolation; regressions (task 03 list) green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5): the panel is a labeled section, contrast ≥4.5:1, no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
## Locked decisions
- **A7 / A15 untouched** — retrieval, context assembly, and the SSE contract are unchanged; this is a display + API-field phase.
- **Phase 16 soft rule untouched** — `GET /api/documents/content` stays public + stateless (anyone who can open a document sees its summary; the catalog stays admin-gated).
- **Phase 30 untouched** — summaries are still generated at import, still markdown-excluded, still fail-soft (NULL possible); this phase only surfaces the existing field.
- **Shared-renderer principle (phase 26)** — the panel is drawn in `renderDocument`, so the page and the modal can never drift.
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages; summary text rendered with `textContent` (XSS contract unchanged).
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
@@ -1,32 +0,0 @@
# Task 01 — The content endpoint returns the summary
**Phase:** `36_summary_in_viewer` · **Source:** `TODO.md:5 — "When I click on a document with a summary I should be able to see the summary and the original document together."`
**Story:** `.agent/user_stories/summary-in-viewer.md`
## Objective
Surface the existing `documents.summary` field (phase 30) on the viewer's data contract: `DocContent` gains a nullable `summary` and `GET /api/documents/content` returns it — endpoint stays public, stateless, and otherwise byte-identical.
## Work
1. `app/schemas.py` — `DocContent` gains:
```python
#: Lite-model summary (phase 30) — non-markdown A9 docs only; None for
#: markdown documents, pre-phase-30 rows, and the fail-soft path where
#: summary generation failed but the document was still indexed.
summary: str | None = None
```
(Place it after `format` / before `content`, with the docstring mirroring `Document.summary`'s.)
2. `app/api/docs.py` — `get_document_content` returns `summary=doc.summary` in the `DocContent(...)` construction. Nothing else changes (no auth, no query change — `Document` is already selected in full).
3. Integration tests — extend the existing `/api/documents/content` test module (find it in `tests/integration/` — the phase-10 content-endpoint tests):
- A non-markdown document row seeded with `summary="…"` → response JSON carries `summary` verbatim.
- A markdown document row with `summary=None` → `"summary": null`.
- Anonymous (no admin cookie) still gets 200 (the phase-16 soft rule — public viewer) for both.
- The existing assertions (404 on unknown pair, content/format fields) stay green unmodified.
## Testing & Quality
- Integration: the cases above; `uv run pytest` green overall.
- Coverage: **>90%** on `app/` — the touched endpoint stays covered (the new field is exercised by the new assertions).
## Completion Criteria
- [ ] `GET /api/documents/content?source=…&path=…` returns the summary for a summarized doc and `null` for a markdown doc; anonymous access unchanged (200).
- [ ] No other field, status code, or auth behavior of the endpoint changed.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
@@ -1,44 +0,0 @@
# Task 02 — The shared renderer draws the Summary panel (both surfaces)
**Phase:** `36_summary_in_viewer` · **Source:** `TODO.md:5 — "When I click on a document with a summary I should be able to see the summary and the original document together."`
**Story:** `.agent/user_stories/summary-in-viewer.md`
## Objective
`renderDocument` — the single rendering core both the full-page viewer (`document.html`) and the chat/sources modal (`document-modal.js`) go through (phase 26) — draws a labeled Summary panel above the original content whenever `doc.summary` is non-empty; `null`/empty renders nothing, so markdown documents and fail-soft rows are byte-for-byte unchanged.
## Work
1. `frontend/assets/document.js` — in `renderDocument(doc, { titleEl, metaEl, contentEl })`, **after** the meta row is populated and **before** the content is built, append the summary section to `contentEl` (which is then filled with the usual `.doc-md` / `<pre class="doc-raw">` content after it):
```js
if (doc.summary && doc.summary.trim() !== "") {
const section = document.createElement("section");
section.className = "doc-summary";
section.setAttribute("aria-label", "Summary");
const title = document.createElement("h2");
title.className = "doc-summary-title";
title.textContent = "Summary";
const body = document.createElement("p");
body.className = "doc-summary-text";
body.textContent = doc.summary; // text node — XSS contract unchanged
section.append(title, body);
contentEl.appendChild(section);
}
```
- The meta badge row (`#doc-meta` / `#doc-modal-meta`) is untouched; the panel sits between meta and content on **both** surfaces because both call this one function.
- Update the file-header comment (the renderer now also owns the optional summary panel) and the `renderDocument` doc comment.
2. `frontend/assets/styles.css` — `.doc-summary` (dark tech theme, phase-08 palette):
- A clearly-distinct "summary, not content" look: surface `#121a2e` with a 3px brand left border (`#6d78f2`) or a brand-soft (`#232b52`) header strip — pick one and keep it simple; `border-radius` matching the existing content cards; padding ~1rem; `margin-bottom` separating it from the content.
- `.doc-summary-title` — small-caps/label treatment: `#a5b4fc` (brand-ink on the brand-soft chip, ≈6.9:1) or `#a5b4fc` on surface (verify ≥4.5:1 — if short, use the chip).
- `.doc-summary-text` — `var(--ink)` (`#e8ebf4`) on the surface (≈14.5:1); wraps inside the same width the content uses (the ≤46rem centered column for md docs, the raw-content width otherwise — the panel is a child of `contentEl`, so it inherits the column; verify for the `<pre class="doc-raw">` case where the content is wider).
- No animation (nothing for `prefers-reduced-motion` to still); the section is static content — no focusability needed (it carries `aria-label` + heading).
3. Do NOT touch `document-modal.js` (it calls `renderDocument` — the panel comes for free), `markdown.js`, or the page scripts.
## Testing & Quality
- Frontend-only — no Python change; the no-CDN integration test is unaffected.
- Coverage: `app/` gate unaffected.
- Manual smoke (dev server with an imported non-md doc, e.g. a yaml from the fixture KB via the importer + mock, or a hand-seeded `documents.summary` row): modal from the Sources table AND the full page both show panel + content; a markdown doc shows no panel.
## Completion Criteria
- [ ] A document with a summary shows the labeled panel above the original content in **both** the modal and the full-page viewer; the original content is fully visible (nothing hidden/collapsed).
- [ ] A document without a summary (`null` or empty/whitespace) renders exactly as before on both surfaces — no panel, no empty box.
- [ ] The summary text is written with `textContent` (XSS contract); the meta row is unchanged.
- [ ] `uv run pytest` green (no-CDN test); `uv run ruff check . && uv run pyright` clean.
@@ -1,31 +0,0 @@
# Task 03 — Story E2E + regression pass + commit
**Phase:** `36_summary_in_viewer` · **Source:** `TODO.md:5 — "When I click on a document with a summary I should be able to see the summary and the original document together."`
**Story:** `.agent/user_stories/summary-in-viewer.md`
## Objective
Prove the story end-to-end with its dedicated Playwright suite — summary panel + original content visible together on both surfaces, no panel for markdown docs — then the full gate + one commit.
## Work
1. `tests/e2e/test_summary_in_viewer.py` (NEW — the story gate, run in isolation). Reuse `tests/e2e/test_document_summaries.py`'s machinery as closely as possible (its fixture KB + import helpers are the reference):
- Fixtures: the standard E2E app + DB + the deterministic mock LLM (`tests/e2e/mock_llm.py` — its `SUMMARY_MODE` answer is a byte-stable 24-token digest); import `tests/fixtures/summary_kb/` so `quadlet/qwen-llamacpp.yaml` gets its stored summary + `notes/qwen-llamacpp-notes.md` stays summary-less (the existing tail sentinel `RESE-SUMMARY-SENTINEL-7f3a` sits on the yaml's LAST line — **outside** the 24-token digest, so it is a marker for "the original, not the summary").
- **Full page:** open `/document.html?source=summary_kb&path=quadlet%2Fqwen-llamacpp.yaml` → `.doc-summary` visible with the deterministic digest text AND the original content visible with the sentinel (`RESE-SUMMARY-SENTINEL-7f3a` present in the rendered content) — summary and original **together**.
- **Modal:** from the Sources table (admin session via `tests/e2e/auth_helpers.py`), click the yaml's row → the modal shows the same panel + content (sentinel present, digest present); then "Full page" still lands on the dedicated page with the panel (the two surfaces agree).
- **No-summary control:** the markdown doc (`notes/qwen-llamacpp-notes.md`) → no `.doc-summary` element on the full page and in the modal; the content renders as before.
- **API shape (cheap, via the page context's `fetch` or `context.request`):** `GET /api/documents/content` for the yaml carries `summary` (string), for the md doc `null`; anonymous fetch → 200 (soft rule unchanged).
- The E2E must not depend on a real LLM (the mock's digest is deterministic — the same pattern `test_document_summaries.py` relies on).
2. **Regression pass — each in isolation** (`uv run pytest tests/e2e/<file>.py -v --no-cov`): `test_summary_in_viewer.py` (new), `test_document_viewer.py`, `test_document_summaries.py`, `test_document_back_navigation.py`, `test_chat_rag.py` (the source-chip modal path), `test_smoke.py`.
3. Full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (>90%), `uv run ruff check . && uv run pyright`.
4. **UI Structure Check** (AGENTS.md rule 5): the panel is a labeled section (`aria-label` + heading), theme contrast ≥4.5:1, it does not break the centered 46rem chat-column-width content layout, no CDN (rule 6 — the no-CDN integration test covers it).
5. **Commit** (A17): stage this phase's files (`app/schemas.py`, `app/api/docs.py`, `frontend/assets/document.js`, `frontend/assets/styles.css`, `tests/**`), message `feat(viewer): show document summary together with the original (TODO.md L5)`, always `--no-gpg-sign`. Move `.agent/phases/todo/36_summary_in_viewer/` to `.agent/phases/complete/`.
## Testing & Quality
- E2E: `tests/e2e/test_summary_in_viewer.py` green **in isolation** (A16: one story, one file).
- Unit/integration: from task 01 — all green under `uv run pytest`.
- Coverage: **>90%** on `app/` (the endpoint change is covered).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_summary_in_viewer.py -v --no-cov` green in isolation.
- [ ] Every suite in the task 03 regression list green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
@@ -1,39 +0,0 @@
# Phase 39 — Configurable app name (brand)
**Source:** `TODO.md` L12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."
**Story:** `.agent/user_stories/configurable-brand.md`
**Context:** `app/config.py` (`app_name`, already `BOR_APP_NAME`, currently used only for the OpenAPI title at `app/main.py:48`), the five templates (`frontend/*.html` — the `.brand-text` spans, the `<title>`s, the meta descriptions, the index empty-state h1, the `aria-label`s), `frontend/assets/app.js` (status labels ~L107/112, the elapsed-hint aria ~L540), `frontend/assets/document.js` (page titles L147/152), `app/api/health.py` (the public stateless endpoint pattern), `app/core/caching.py` (phase 33: `?v=` rewriting of `assets/…` refs — automatic for any new asset file), the `Containerfile` esbuild stage (explicit per-asset lines, L17–23).
## Objective
Make **one env var** (`BOR_APP_NAME`, default "Brain of Reese") drive the app's display name everywhere — titles, the header brand, the status labels, the aria text, the greeting — via a public `/api/config` endpoint + a small `brand.js`, with zero behavior change when the variable is unset.
## Dependencies
- `34_consistent_navbar` (todo — runs before this phase) — the standard five-page header (the `.brand-text` nodes this phase re-skins).
- `01_infrastructure` (complete) — the public stateless endpoint pattern (`app/api/health.py`).
## Tasks
1. `01_config_endpoint.md` — public `GET /api/config` → `{app_name, version}` + tests.
2. `02_frontend_branding.md` — `brand.js` + the template/JS de-hard-coding + the Containerfile build line.
3. `03_e2e_docs_commit.md` — the story E2E (its own app instance with the overridden name), `.env.example` + README, commit, move the phase dir.
## Testing & Quality
- Unit/integration: the endpoint (200, the values, anonymous access, exactly two keys — no settings may leak later).
- E2E (mandatory, A16): `tests/e2e/test_configurable_brand.py` — a **second** app instance booted with `BOR_APP_NAME` overridden; run in isolation.
- Coverage: **>90%** on `app/`.
- Regression: the default-name behavior is byte-identical — the existing suites (which assert "Brain of Reese" titles/labels against the shared conftest server) stay green **unchanged**.
## Completion Criteria
- [ ] `GET /api/config` (anonymous) → `{"app_name": "Brain of Reese", "version": "0.1.0"}` by default; the response key set is exactly `{app_name, version}`.
- [ ] With `BOR_APP_NAME="Brain of Testy"`: the index title "Brain of Testy"; the header `.brand-text` renders `Brain of <strong>Testy</strong>`; the empty-state h1 "Hey! I'm Brain of Testy."; the `#messages` aria-label "Conversation with Brain of Testy"; the chat status label "Brain of Testy is thinking"; the other pages' titles carry the name; the document viewer title "… · Brain of Testy".
- [ ] With the variable unset: the existing E2E + unit suites green unchanged (no rename leak).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run pytest tests/e2e/test_configurable_brand.py -v --no-cov` green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
## Locked decisions
- **The existing `app_name` setting is the source of truth (owner permission 2026-08-26)** — `BOR_APP_NAME` (default "Brain of Reese"); no new variable, no setting rename.
- **A11 honoured** — runtime fetch + JS application (no build-time template injection, no Jinja, no CDN); the static templates stay static.
- **A10 honoured** — `/api/config` is public and stateless; the response carries no secrets (exactly `app_name` + `version`).
- **Runtime fetch, brief flash accepted (owner permission 2026-08-26)** — the default name renders immediately and is replaced when `/api/config` answers (LAN latency; no re-paint machinery for in-flight turns — a mid-turn label keeps the previous name for that turn).
- **A16/A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
@@ -1,27 +0,0 @@
# Task 03 — Story E2E + docs + commit
**Phase:** `39_configurable_brand` · **Source:** `TODO.md:12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."`
**Story:** `.agent/user_stories/configurable-brand.md`
## Objective
The story's isolated Playwright suite against an app instance booted with the overridden name, the env docs, and the phase commit.
## Work
1. `tests/e2e/test_configurable_brand.py` (the story gate — one story, one file, run in isolation):
- A **second app instance** — the shared `app_server` conftest fixture keeps the default name (the other suites' title/label assertions depend on it). Copy the conftest `app_server` env block (same DB, the mock-LLM base URL, `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET`, `BOR_STATIC_DIR`, `BOR_RELEVANCE_THRESHOLD`) with two changes: `BOR_APP_NAME="Brain of Testy"` and a distinct port (`APP_PORT + 1` per the conftest convention). A session-scoped fixture **inside the test file**, started after `mock_llm` is available.
- Assertions (custom instance): index `document.title` == `"Brain of Testy"`; the `.brand-text` `innerHTML` == `Brain of <strong>Testy</strong>`; the empty-state h1 text == `"Hey! I'm Brain of Testy."`; the `#messages` `aria-label` == `"Conversation with Brain of Testy"`; the sources page title `"Sources · Brain of Testy"`; the login page title `"Sign in · Brain of Testy"`; one chat turn with a pre-token window (the `think out loud` marker) → the button label shows `"Brain of Testy is thinking"`.
- Default-name assertion (cheap regression in the same file): the shared conftest server's index title still == `"Brain of Reese"`.
- The chat-turn assertion works on the default (possibly empty) KB — a deflected answer is fine; the label assertion is pre-token, so no DB seeding is required.
2. `.env.example` — document `BOR_APP_NAME` in the App section (the display name on all pages; default "Brain of Reese").
3. README — the configuration section: `BOR_APP_NAME` (what it affects: titles, the header brand, the status labels, the aria text; the default; the bold-split rendering rule: names starting "Brain of " bold the remainder, any other name renders in normal weight).
4. Regression pass: `uv run pytest` + the coverage gate (>90%) + the isolated story E2E + the suites that assert brand strings (`test_smoke.py`, `test_shared_header.py`, `test_header_consistency.py`, `test_chat_persistence.py`) green.
5. Commit — one atomic `--no-gpg-sign` Conventional Commits commit for the whole phase (AGENTS.md rule 8), e.g. `feat(brand): configurable app name — BOR_APP_NAME drives /api/config + the frontend brand layer`; move the phase directory to `.agent/phases/complete/`.
## Testing & Quality
- The gates above are this task's quality bar (A16: one story, one isolated E2E file, coverage >90%).
## Completion Criteria
- [ ] The story E2E is green in isolation, deterministic across two consecutive runs (custom-name instance + the default-name assertion).
- [ ] The step-4 regression list green; coverage >90%.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
@@ -1,55 +0,0 @@
# Phase 21 — Thinking Window: No Scroll Back, Just Live
**Status:** Complete
**Date:** 2026-08-25
## Objective
Disable user scrolling in the Thinking window so it always shows the live tail
of the reasoning stream in a fixed 320px clip.
## What was implemented
### Task 01 — CSS change + unit pins (completed by prior agent)
- `frontend/assets/styles.css`: `details.thinking .thinking-text` changed from
`overflow-y: auto` to `overflow-y: hidden` (with owner-choice comment).
- `tests/unit/test_thinking_no_scroll.py`: source-level pins for CSS value,
comment, and JS bottom-pin integrity.
### Task 02 — E2E story suite, story file, validation, commit (completed this run)
- `tests/e2e/test_thinking_no_scroll.py` (5 tests):
1. `test_thinking_window_not_user_scrollable` — wheel/drag/keyboard on the
frozen live tail do not move the window.
2. `test_thinking_window_tracks_live_tail` — per-chunk pin keeps the window
glued to the tail; last chunk's text is inside the visible rectangle.
3. `test_thinking_window_css_contract` — computed `overflow-y` is `hidden`,
`max-height` is `320px`, clip is real.
4. `test_answer_bubble_still_scrollable` (regression, phase 11) — answer
bubble overflow is untouched, page scrolls normally.
5. `test_restored_collapsed_thinking_unaffected` (regression, phase 17) —
stored thinking restores collapsed with full text.
- `.agent/user_stories/thinking-no-scroll.md` — story file with bug report,
narrative, acceptance criteria, owner-confirmed decisions, and Playwright
mapping.
## Test / lint / coverage results
| Check | Command | Result |
|-------|---------|--------|
| E2E story suite | `uv run pytest tests/e2e/test_thinking_no_scroll.py -v --no-cov` | **5/5 passed** |
| Regression: thinking display | `uv run pytest tests/e2e/test_thinking_display.py -v --no-cov` | **5/5 passed** |
| Regression: long answers | `uv run pytest tests/e2e/test_long_answers.py -v --no-cov` | **2/2 passed** |
| Full suite | `uv run pytest --no-cov` | **319 passed** |
| Coverage | `uv run pytest --cov=app --cov-report=term-missing` | **99%** (gate: >90%) |
| Lint | `uv run ruff check .` | **All checks passed** |
| Types | `uv run pyright` | **0 errors** |
## Notable decisions
- No JS changes: the phase-17 bottom-pin (`textEl.scrollTop = textEl.scrollHeight`)
already works under `overflow-y: hidden`, so the CSS-only change is sufficient.
- The E2E tests use the mock's `think out loud then hesitate` trigger (phase 20)
to create a 4s frozen window where user scroll would be observable if regressed.
- The long thinking body (~2,700 chars, lengthened in phase 21) overflows the
320px window by ~2x, making the live-tail clip contract E2E-observable.
## Deviations
None. All changes follow LOCKED DECISIONS A11 (no CDN, pure CSS), A16 (one E2E
suite per story), and the owner-confirmed roadmap A2 decisions.
@@ -1,186 +0,0 @@
# Phase 22 / Task 01 — Background animation diagnosis + fix
**Owner report (2026-08-24):** "background animation not working, just
blinking". **Method:** live dev server (`uv run uvicorn app.main:app`,
db up), headless Chromium 1280×800 (Playwright), ≥15s observation per
session, computed-style sampling + pixel-diffed screenshots. All
artifacts in `.agent/screenshots/22_background_animation/`.
## Diagnosis (per the 00_phase.md checklist)
**1. Per-layer visibility.** Both pseudo-layers exist, computed
`position: fixed`, `z-index: -1`, `pointer-events: none`, both animating.
Per-layer screenshots (`before_glow_only.png`, `before_grid_only.png`):
the glow is clearly visible in its two corners; the grid is drawn **only
in a narrow band below the header** (top ~25% of the viewport) and is
barely distinguishable from the page bg.
**2. Grid layer — moving but imperceptible.** `background-position` on
`body::before` sampled over 16.8s: `0.86px → 13.2px` (≈0.79px/s — the
spec'd 44px/60s, seamless one-cell drift). The drift is *real*.
Isolating it (glow hidden, two frames 5s apart,
`before_gridonly_frame1/2.png`):
| metric | value |
|---|---|
| pixels changed (>12/765) | 1.5% |
| mean pixel change | **0.48/765** (≈0.2/255 per channel) |
| bottom-half quadrants | **≈0.07** (no grid there at all — masked out) |
Why: 1px lines at **35% of `--line`** blend to ≈rgb(20,26,41) on
rgb(10,14,23) — a ≈10/12/18-level delta — and the radial mask
(`120% 90% at 50% 0%, black 25%, transparent 78%`) fades the layer to
~zero past ~560px at center (probe: lines gone by y≈254 at the edges).
A 0.73px/s drift of that faint texture sits below the perception
threshold → the grid contributes no visible motion.
**3. Glow layer — the only visible motion = the "blink".** Full-background
frame diff (5s apart, `before_frame1/2.png`): 1.4% of pixels changed,
concentrated exactly on the two glow centers — top-left mean 3.07,
bottom-right 2.92 (glow positions 12%/8% and 88%/92%), top-right 1.23,
bottom-left 0.70. The 0.65↔1.0 opacity swing (35% amplitude, + scale)
is the entire perceived animation → reads as a blink/pulse, not
breathing.
**4. Occlusion check — ruled out.** Forced the grid lines to bright red:
3127 red-line pixels render (`before_occlusion_probe.png`, x 132–1276,
y 64–254 — the mask shape). Computed: `html` bg `rgb(10,14,23)`
(= `var(--bg)`), `body` bg `rgba(0,0,0,0)` (transparent), no later rule
paints over the layers. Assumption (c) **does not fire**.
### Root cause
**Assumptions (a)+(b) confirmed, (c) disproven.** The grid drift works
but is invisible (faint 35%-alpha lines × small masked region × 0.73px/s),
so the only perceived motion is the glow's large opacity swing — which
is exactly what "just blinking" is.
## Fix (styles.css — pure CSS, zero JS, no blur, palette untouched)
| property | before (phase 08) | after (phase 22) |
|---|---|---|
| grid line alpha | `rgb(38 48 74 / 0.35)` ×2 | **`rgb(38 48 74 / 0.6)`** ×2 |
| grid mask | `120% 90% at 50% 0%, black 25%, transparent 78%` | **`140% 110% at 50% 0%, black 40%, transparent 90%`** |
| grid drift | 60s, 0→44px | **unchanged** (60s, 0→44px — one-cell seamless) |
| glow opacity | 0.65 ↔ 1 | **0.85 ↔ 1** |
| glow period/scale | 14s ease-in-out alternate, 1↔1.05 | **unchanged** |
- Grid: +71% line contrast and a wider fully-visible mask radius make
the same 60s one-cell drift readable as smooth motion; the grid now
extends across most of the viewport, fading to the corners.
- Glow: amplitude cut 35%→15% — breathing, not pulsing.
- Durations deliberately untouched: the owner-confirmed intended effect
is the phase-08 design (60s drift + 14s breathe), and
`tests/e2e/test_dark_tech_theme.py::test_animated_background` pins the
live 60s/14s durations. Both layers keep `fixed; inset: 0; z-index:
-1; pointer-events: none`; `html` keeps `var(--bg)`, `body` stays
transparent; reduced-motion stilling block untouched.
## After (same viewport, same instrumentation)
| metric | before | after |
|---|---|---|
| grid-only 5s diff, mean \|d\| | 0.48 | **1.82** (3.8×) |
| grid-only, pixels changed | 1.5% | **6.1%** |
| grid-only bottom-half | ≈0.07 (absent) | **1.43–1.47** (present + moving) |
| full 5s diff, pixels changed | 1.4% | **4.2%** |
| full quadrant means (TL/TR/BL/BR) | 3.07 / 1.23 / 0.70 / 2.92 | 2.92 / 2.11 / 1.69 / 2.48 |
Motion is now distributed across the whole viewport (drift everywhere +
gentler corner breathing) instead of concentrated in the two glow
corners; after frames (`after_frame1/2.png`, 5s apart) confirm no static
frame. Visual check of `after_frame1.png`: grid clearly visible, still
subtle — no text/contrast impact, no jank (compositor-friendly
`background-position`/`opacity`/`transform` only), no 360px overflow
(layers remain `fixed; inset: 0`; `test_responsive_polish.py` 7/7 green).
## Screenshots
`before_frame1/2.png` (5s pair), `before_glow_only.png`,
`before_grid_only.png`, `before_occlusion_probe.png`,
`before_gridonly_frame1/2.png` (isolated grid pair), `after_frame1/2.png`
(5s pair), `after_gridonly_frame1/2.png`, `diagnosis_log.txt` (timeline
samples), `diagnosis_diff.txt` (all diff numbers).
## Tests (task 01)
- New: `tests/unit/test_background_animation.py` (10 source pins on the
final values + the no-occlusion + no-blur/zero-JS anchors) — green.
- `uv run pytest --cov=app` — 329 passed, app coverage 99%.
- `uv run ruff check . && uv run pyright` — clean.
- Regressions (isolated): `test_dark_tech_theme.py` 6/6,
`test_responsive_polish.py` 7/7.
---
# Task 02 — E2E story suite, story file, validation, commit
## Story gate: `tests/e2e/test_background_animation.py` (5 tests)
The layers are CSS pseudo-elements, so the suite asserts via computed
style + the Web Animations API. **Implementation note (deviation from
the task file, documented per the plan's "found, not guessed" rule):**
the task assumed Chromium reports pseudo-element CSS animations through
`element.getAnimations()` — verified false on Chromium 151
(`document.body.getAnimations()` returns `[]` for pseudo-layers); the
animations ARE enumerated by `document.getAnimations()`, with
`animationName` / `playState` / `currentTime` intact. The suite uses the
document-level list and matches on `animationName`. A second finding:
headless Chromium starts the document animation timeline ~1.4s after
navigation (currentTime stays 0 until then), so `test_animations_advance`
polls until both timelines are alive before sampling.
1. `test_grid_layer_animation_running` — computed `body::before`:
`animationName = bg-grid-drift`, timing `linear`, iterations
`infinite`; matching `CSSAnimation` with `playState === "running"`.
2. `test_glow_layer_animation_running` — same for `body::after`
(`bg-glow-breathe`); `playState === "running"`.
3. `test_animations_advance` — `currentTime` of both layers advances by
≥200ms over a 500ms wait — the timelines are truly running, not
paused (no static frame).
4. `test_background_layers_contracts` — both pseudo-elements: `position:
fixed`, `z-index: -1`, `pointer-events: none`, `inset: 0` (0px on all
four edges); `<html>` computed background `rgb(10, 14, 23)` (canvas
stays on html), `<body>` computed background `rgba(0, 0, 0, 0)`
(no occlusion).
5. `test_no_horizontal_overflow_with_layers` — 360px viewport:
`documentElement.scrollWidth <= clientWidth` (phase-07 pin,
replicated locally).
## Results (all commands, final state)
| command | result |
|---|---|
| `uv run pytest tests/e2e/test_background_animation.py -v --no-cov` | **5 passed** (run 4× — stable) |
| `uv run pytest tests/e2e/test_dark_tech_theme.py -v --no-cov` | **6 passed** (isolated) |
| `uv run pytest tests/e2e/test_responsive_polish.py -v --no-cov` | **7 passed** (isolated) |
| `uv run pytest` | **329 passed** |
| `uv run pytest --cov=app --cov-report=term-missing` | 329 passed, **app coverage 99%** (≥90% gate; unchanged from task 01) |
| `uv run ruff check . && uv run pyright` | **clean** (0 errors) |
Prerequisite honored: `podman compose up -d db` — `brain-of-reese_db_1`
Up (healthy) throughout.
## UI Structure Check (AGENTS.md rule 5)
- **Layers behind content:** E2E `test_background_layers_contracts`
pins `position: fixed`, `z-index: -1`, `pointer-events: none`,
`inset: 0` on both pseudo-layers in the live viewport (plus the
html-canvas/body-transparent no-occlusion pair).
- **No text/contrast impact:** the fix touches only line alpha, mask
radius, and glow opacity — no palette token, no text on the layers,
all WCAG AA pairs unchanged (phase-08 suite green).
- **No 360px overflow:** E2E pin `scrollWidth <= clientWidth` at
360×740 — live check `scrollWidth=360 clientWidth=360`; the
`fixed; inset: 0` layers add no width (phase-07 suite green).
## Final screenshots (fresh server, headless Chromium 151)
- `final_frame1.png` / `final_frame2.png` — 1280×800, 3s apart: grid
drift visible across the viewport (live sample at frame 2:
`background-position 2.55px` after ≈3.5s ≈ 0.73px/s — the spec'd
one-cell/60s drift), glow corners breathing gently.
- `final_360px.png` — 360×740, no horizontal overflow.
- Task-01 before/after evidence (root-cause + fix delta) in the same
directory (`before_*`, `after_*`, `diagnosis_log.txt`,
`diagnosis_diff.txt`).
## Story file
`.agent/user_stories/background-animation.md` — goal, bug report
verbatim (TODO.md L5), owner-confirmed A3 decisions, found root cause,
fix table, acceptance criteria, UI structure, Playwright mapping table.
@@ -1,120 +0,0 @@
# Phase 23 report — Containerfile: build the whole app image again
**Date:** 2026-08-24 · **Source:** `TODO.md` L6 ("Fix Containerfile build
not working") · **Story:** `.agent/user_stories/containerfile-build.md`
## Root cause (verified, not guessed — logs in this directory)
1. **Build failure:** phase 19's absolute `import … from
"/assets/header.js"` is resolved by esbuild as the *filesystem* path
`/assets/header.js` → `✘ [ERROR] Could not resolve "/assets/header.js"`,
reproduced with the exact pinned esbuild **0.25.5**
(`repro_esbuild_0.25.5.log`, `repro_podman_build_before_fix.log`).
2. **Under-shipped image:** stage 1 bundled only `app.js` +
`sources.js` and copied only `index.html` + `sources.html` —
`document.html` / `login.html` / `document.js` / `login.js` /
`markdown.js` were all missing from the image.
3. **Double-evaluation trap:** the four direct `<script
src="/assets/header.js">` tags would evaluate the header module
twice in the image (bundled copy + raw module) → duplicate sign-out
listener, double init.
4. **Second blocker (task 01):** a local `npm install` of esbuild left
the binary off the next `RUN`'s PATH (`esbuild: not found`) → the
stage installs it globally.
## Fix
| piece | after |
|---|---|
| page-script header import | `from "./header.js"` (relative — owner-confirmed over an esbuild alias) |
| direct header.js `<script>` tags | **removed from all four pages** (the hoisted import covers evaluation order) |
| esbuild install | `npm install --no-audit --no-fund -g esbuild@0.25.5` (pinned, on PATH) |
| stage-1 bundles | `app.js`, `sources.js`, `document.js`, `login.js` |
| stage-1 minifies | `styles.css` + `markdown.js` (no `--bundle` — classic script) |
| stage-1 copies | all four HTML pages into `/out` |
## Build gate — `podman build` (task 01, `podman_build_after_fix.log`)
```
[1/3] STEP 3/5: RUN npm install --no-audit --no-fund -g esbuild@0.25.5
[1/3] STEP 5/5: RUN mkdir -p /out/assets && esbuild ./assets/app.js --bundle … && … && cp ./index.html ./sources.html ./document.html ./login.html /out/
../out/assets/app.js 14.3kb ⚡ Done in 2ms
../out/assets/sources.js 2.3kb ⚡ Done in 2ms
../out/assets/document.js 2.4kb ⚡ Done in 2ms
../out/assets/login.js 1.5kb ⚡ Done in 2ms
[3/3] COMMIT brain-of-reese/app:phase23test
Successfully tagged localhost/brain-of-reese/app:phase23test
39f5563a1b101b274d81d86aa06f86a4d6d1bba453e8dba40d6f48f987537019
BUILD_EXIT=0
```
## Image smoke (task 01, `image_smoke.log`)
Throwaway Postgres 17 + pgvector (`bor-smoke-db`, isolated network),
production env, teardown done:
- app boots: migrations applied, uvicorn up, `/api/health` →
`{"status":"ok","db":"up",…}`
- pages: `GET /`, `/sources.html`, `/document.html`, `/login.html` → **200**
- assets: `app.js`, `sources.js`, `document.js`, `login.js`,
`markdown.js`, `styles.css` → **200**
- single-evaluation: raw `/assets/header.js` → **404** (expected — the
header code ships inside each page bundle); all four bundles are
minified and contain the header markers (`sign-out-btn`,
`/api/logout`)
- No CDN rule: `cdn_violations=0` across all four served pages
- `direct_header_script_tags=0`
- → **`SMOKE_PASS`**
## Task 02 — hermetic coverage pin + validation
`tests/integration/test_containerfile_assets.py` (6 tests, no podman /
no network / no DB — parses `Containerfile` + `frontend/` as text):
1. `test_every_html_page_is_copied_into_stage1` — cp set == pages on disk
2. `test_every_local_asset_reference_is_produced` — every `assets/`
src/href has a stage-1 producer
3. `test_page_module_scripts_are_bundled` — HTML module set == esbuild
`--bundle` input set
4. `test_header_module_is_imported_not_directly_loaded` — no direct
header.js tag anywhere; every page script imports `./header.js`
relatively
5. `test_markdown_js_is_a_produced_classic_script` — minified without
`--bundle`; no top-level import/export in the source
6. `test_esbuild_stays_pinned` — concrete `esbuild@X.Y.Z`, no floating
spec
Mutation-checked while authoring: removing `login.html` from the cp
line (→ test 1 fails), `esbuild@latest` (→ test 6 fails), deleting the
`markdown.js` line (→ tests 2+5 fail), deleting the `document.js`
bundle line (→ test 3 fails), re-adding a direct header.js tag (→
tests 3+4 fail). All pins fire.
### Test / lint / coverage results (exact commands)
| command | result |
|---|---|
| `uv run pytest tests/integration/test_containerfile_assets.py -v --no-cov` | 6 passed |
| `uv run pytest tests/e2e/test_smoke.py -v --no-cov` (isolated) | **3 passed** (`e2e_smoke.log`) |
| `uv run pytest tests/e2e/test_shared_header.py -v --no-cov` (isolated) | **6 passed** (`e2e_shared_header.log`) |
| `uv run pytest tests/e2e/test_chat_persistence.py -v --no-cov` (isolated) | **4 passed** (`e2e_chat_persistence.log`) |
| `uv run pytest` | **335 passed** (329 → 335: the six new pins) |
| `uv run pytest --cov=app --cov-report=term-missing` | **335 passed; TOTAL 1109 stmts, 14 missed, 99%** (≥ today's 99%; >90% gate) |
| `uv run ruff check . && uv run pyright` | **All checks passed! / 0 errors, 0 warnings** |
Dev-server behavior unchanged: the tag removal is covered by the three
isolated E2E regressions above (browser-level header init, sign-out
binding exactly once, chat persistence across navigation).
## Decisions
- Relative imports over an esbuild alias (owner-confirmed A4-1).
- Direct header.js tags removed over shipping a raw header.js
(owner-confirmed A4-2 — single module evaluation).
- No Playwright suite for this phase (build/infrastructure): the gate
is the hermetic integration pin + the real `podman build` + image
smoke recorded above + the dev-server E2E regressions (A16 note in
the phase plan).
- No LOCKED anchor changed (A11 No-CDN and A16 honored).
- Build-evidence image `localhost/brain-of-reese/app:phase23test` kept
locally.
@@ -1,97 +0,0 @@
# Phase 25 — Background: No Motion, Only Fading Light
**Owner report (2026-08-25, chat, verbatim):** "It should be smooth,
fluxuating, dimming and brightening, but not moving. Different bright
spots should slowly fade in and out." **Story:**
`.agent/user_stories/background-no-motion.md`.
## Root causes (phase-22 measurements as context)
1. **"Jitters down and to the right every second" = the grid drift.**
`bg-grid-drift` translated the 44px grid 44px per 60s (≈0.73px/s)
diagonally down-right — exactly the reported direction. A 1px line
moved sub-pixel by sub-pixel rasterizes with per-frame stepping, not
smooth motion, so it reads as a once-per-second jitter. Phase 22
had made that drift *visible* (60% line alpha + widened mask;
measured 1.82/765 mean pixel change over 5s in the grid zone vs
0.48 pre-phase-22) — that is precisely why it now reads as jitter.
Confirmed in the `before.png`/`before_4s.png` pair below: a
grid-only region far from every glow changes by 2.98/765 mean over
4s (4.7% of its pixels) — the grid itself is moving.
2. **"Slowly blinks brighter and darker" = the whole-layer breathe.**
`bg-glow-breathe` swung the entire glow layer's opacity 0.85↔1 over
14s (alternate) plus `scale(1)↔scale(1.05)` (a faint zoom). One
synchronized pulse of the whole background reads as a blink; the
before pair's uniform change across both glow corners (mean
2.19/765 over the full frame) is that single pulse.
## Design change (styles.css — pure CSS, zero JS, no blur, palette
untouched; task 01)
| property | phase 22 | phase 25 |
|---|---|---|
| grid (`body::before`) | `bg-grid-drift 60s linear infinite` (0→44px, 0.73px/s) | **static** — animation removed, `bg-grid-drift` deleted (44px cells, 60% `--line` 1px lines, widened mask kept) |
| glow | one whole-layer `bg-glow-breathe` (14s, opacity 0.85↔1 + scale 1↔1.05) | **three independent spot layers, opacity-only fades** |
| spot A `body::after` | indigo + cyan on one layer | indigo `rgb(109 120 242 / 0.14)` 56rem @ 12%/8% — `bg-glow-a` **26s** ease-in-out infinite, low 0.25 |
| spot B `html::before` | — | cyan `rgb(34 211 238 / 0.10)` 60rem @ 88%/92% — `bg-glow-b` **34s** ease-in-out **−12s** infinite, low 0.20 |
| spot C `html::after` | — | indigo `rgb(109 120 242 / 0.09)` 52rem @ 14%/86% — `bg-glow-c` **42s** ease-in-out **−23s** infinite, low 0.15 |
| keyframes | `bg-grid-drift` (background-position), `bg-glow-breathe` (opacity + transform) | `bg-glow-a/b/c` — **opacity only** (0%/100% low → 50% 1) |
| reduced motion | stills the two body layers | stills **all four** layers |
`html::before`/`html::after` join as background layers: `<html>` is
the root stacking context, so their `z-index: -1` pseudo-elements
paint above the `var(--bg)` canvas and below the transparent,
non-stacking `<body>`'s content (verified live — E2E test 6). 26/34/42s
with negative delays (LCM 4641s) keep the cycles out of phase, so the
composite pattern effectively never repeats within a viewing session.
No LOCKED anchor changed (A11 pure CSS / zero JS / no CDN / no new
assets; no `filter: blur`).
## Before / after evidence (1280×800 headless Chromium, shots ~4s
apart; `.agent/screenshots/25_background_no_motion/`)
- `before.png` / `before_4s.png` (captured **pre-change**, task 01):
down-right grid jitter + the uniform whole-layer pulse.
- `after.png` / `after_4s.png` (captured post-change, task 02):
brightness changes; the grid is bit-identical.
Region diffs (pure-stdlib PNG decode; "changed" = pixels with
per-channel Δ sum > 12/765):
| region | before pair: changed / mean\|d\| | after pair: changed / mean\|d\| |
|---|---|---|
| full frame | 42,316 (4.1%) / 2.190 | 8,357 (0.8%) / 1.754 |
| grid-only patch (900,30)–(1250,120) | 1,484 (4.7%) / **2.979 — the grid moves** | 0 (0.0%) / **0.000 — bit-identical** |
| spot A center (154,64) patch | 406 (4.1%) / 3.198 | 4,018 (40.2%) / 6.595 — mid-fade |
| spot C clip (0,500)–(500,800) | 6,469 (4.3%) / 1.274 | 0 / 3.247 — gentle fade, no >12/765 steps |
| spot C center (179,688) patch | 63 (0.6%) / 0.789 | 0 / 6.003 — visibly brightening |
| spot B clip (780,500)–(1280,800) | 5,784 (3.9%) / 2.037 | 0 / 1.211 — slow fade at this phase |
Read: after the change, the only region that moves pixel-by-pixel is
the light itself (spot A's 26s cycle passes through its steepest
mid-fade in the 4s window); every region that the grid drift used to
scrub through is now byte-identical — no positional shift anywhere.
## E2E story gate (task 02) — `tests/e2e/test_background_no_motion.py`
8 tests, green in isolation (see "Results"). The deterministic
no-movement proof is test 3: a live Chromium walk of
`document.styleSheets` shows the set of properties declared across
every frame of every `bg-*` @keyframes rule is exactly `{opacity}`.
Regression suites adapted to the new contract:
`tests/e2e/test_background_animation.py` (grid static + `bg-glow-a`
running + the three spot timelines advance + 4-layer contracts + 360px
pin) and `tests/e2e/test_dark_tech_theme.py`
(`test_animated_background` → static grid + 26/34/42s spots;
`test_reduced_motion_honored` → all four layers stilled).
## UI Structure Check (AGENTS.md rule 5)
- **Layers behind content:** E2E test 6 pins `position: fixed`,
`z-index: -1`, `pointer-events: none`, `inset: 0` on all four
pseudo-layers live, plus the html-canvas / body-transparent
no-occlusion pair (`rgb(10, 14, 23)` / `rgba(0, 0, 0, 0)`).
- **No text/contrast impact:** the change touches only background
layers (no palette token, no text on the layers); the phase-08
contrast suite stays green.
- **No 360px overflow:** E2E test 8 — `scrollWidth <= clientWidth` at
360×740 with all four `fixed; inset: 0` layers live.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 134 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 140 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 80 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 112 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 108 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 96 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 65 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 104 KiB

@@ -1,24 +0,0 @@
== BEFORE (phase-08 CSS) ==
FULL before, 5s apart: 1.436% px>12, mean|d|=1.9793, max=54
top-left: mean|d|=3.0659
top-right: mean|d|=1.2274
bottom-left: mean|d|=0.7014
bottom-right: mean|d|=2.9225
GRIDONLY before, 5s apart (glow hidden): 1.526% px>12, mean|d|=0.4778, max=60
top-left: mean|d|=0.8985
top-right: mean|d|=0.8776
bottom-left: mean|d|=0.0681
bottom-right: mean|d|=0.0671
OCCLUSION PROBE: 3127 red-line px (x 132..1276, y 64..254) -> grid layer IS painted (no occlusion); earlier 0-pixel reading was a too-strict absolute-threshold detector (blended red ≈ rgb(78,27,35))
== AFTER (phase-22 CSS: grid lines 0.6 alpha, mask 140% 110% / 40% / 90%, glow 0.85..1) ==
FULL after, 5s apart: 4.159% px>12, mean|d|=2.3007, max=87
top-left: mean|d|=2.9225
top-right: mean|d|=2.1081
bottom-left: mean|d|=1.6897
bottom-right: mean|d|=2.4826
GRIDONLY after, 5s apart (glow hidden): 6.094% px>12, mean|d|=1.8162, max=74
top-left: mean|d|=2.2062
top-right: mean|d|=2.1563
bottom-left: mean|d|=1.4749
bottom-right: mean|d|=1.4275
@@ -1,13 +0,0 @@
LAYER STATE (computed):
before: {'position': 'fixed', 'zIndex': '-1', 'display': 'block', 'pointerEvents': 'none', 'maskImage': 'radial-gradient(120% 90% at 50% 0%, rgb(0, 0, 0) 25%, rgba(0, 0, 0, 0) 78%)', 'backgroundSize': '44px 44px, 44px 44px', 'animation': '60s linear infinite bg-grid-drift'}
after: {'position': 'fixed', 'zIndex': '-1', 'display': 'block', 'pointerEvents': 'none', 'animation': '14s ease-in-out infinite alternate bg-glow-breathe'}
htmlBg: rgb(10, 14, 23)
bodyBg: rgba(0, 0, 0, 0)
TIMELINE (t_s, body::before background-position, body::after):
t= 0.0s grid-pos='0.855518px 0.855518px, 0.855518px 0.855518px' glow={'opacity': '0.654761', 'transform': 'matrix(1.00068, 0, 0, 1.00068, 0, 0)', 'animationName': 'bg-glow-breathe'}
t= 2.8s grid-pos='2.92103px 2.92103px, 2.92103px 2.92103px' glow={'opacity': '0.708889', 'transform': 'matrix(1.00841, 0, 0, 1.00841, 0, 0)', 'animationName': 'bg-glow-breathe'}
t= 5.6s grid-pos='4.97421px 4.97421px, 4.97421px 4.97421px' glow={'opacity': '0.815653', 'transform': 'matrix(1.02366, 0, 0, 1.02366, 0, 0)', 'animationName': 'bg-glow-breathe'}
t= 8.4s grid-pos='7.02747px 7.02747px, 7.02747px 7.02747px' glow={'opacity': '0.927369', 'transform': 'matrix(1.03962, 0, 0, 1.03962, 0, 0)', 'animationName': 'bg-glow-breathe'}
t= 11.2s grid-pos='9.09298px 9.09298px, 9.09298px 9.09298px' glow={'opacity': '0.990928', 'transform': 'matrix(1.0487, 0, 0, 1.0487, 0, 0)', 'animationName': 'bg-glow-breathe'}
t= 14.0s grid-pos='11.1462px 11.1462px, 11.1462px 11.1462px' glow={'opacity': '0.994963', 'transform': 'matrix(1.04928, 0, 0, 1.04928, 0, 0)', 'animationName': 'bg-glow-breathe'}
t= 16.8s grid-pos='13.1995px 13.1995px, 13.1995px 13.1995px' glow={'opacity': '0.940122', 'transform': 'matrix(1.04145, 0, 0, 1.04145, 0, 0)', 'animationName': 'bg-glow-breathe'}
Binary file not shown.

Before

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 137 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 127 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 273 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 286 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 139 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 136 KiB

-92
View File
@@ -1,92 +0,0 @@
# Story: Admin Sync Button (One-Click Doc Import Sync)
**Phase:** `32_admin_sync_button.md` · **E2E:** `tests/e2e/test_sync_button.py`
## Narrative
As **the admin (owner)**, I don't want to SSH in and hand-run the import
script every time my notes repos move. I want **a button that only I can
see** — on the Sources page — that **triggers a doc import sync by
cloning the relevant repos and then running the import script**, with
live feedback so I always know whether it is running, what it changed,
or why it failed.
- **Given** `BOR_GIT_SOURCES` is set (the git repos) and I am signed in
as the admin
- **When** I click **Sync sources** on the Sources page
- **Then** the app clones/pulls each repo, re-imports with prune (the
`--prune` equivalent — the canonical "mirror the repos" action), and
refreshes the KB overview when anything changed — while the button
reports the whole lifecycle (**Syncing…** → **Synced HH:MM** + counts,
or an error banner naming the failure) and never sits stale or stuck.
Anonymous visitors never see the button, and the sync endpoints answer
them with 403.
## Acceptance criteria
1. **Admin-only visibility.** The button ships `hidden` in
`sources.html` (anonymous-safe) and `header.js` reveals it for the
admin on the SAME cached whoami that reveals `#nav-sources` /
`#nav-tuning` (one fetch, no extra whoami call); anonymous users
never see it.
2. **The sync API (A10 extended, A12 in-process).** `POST /api/sync`
(admin only) starts one background task — `clone_or_pull` each
`BOR_GIT_SOURCES` repo (phase 28, reused) → `import_sources(prune=True)`
→ `regenerate_overview` when the KB changed (phase 31) — and returns
`202`. A second trigger while a run is in flight returns
`409 {"detail": "a sync is already running"}` (one sync at a time).
`GET /api/sync/status` (admin only) reports `idle | running |
success | failed` with ISO-8601 timestamps and the run's `detail` /
`error`. Both endpoints answer anonymous callers with 403.
3. **Loud failure.** An unset/empty `BOR_GIT_SOURCES` fails the sync with
"no git sources configured (BOR_GIT_SOURCES)" (manual `--source` dirs
have no repo to clone); a `GitSyncError` fails the run with git's
stderr (the repo named, credentials masked) before any import.
4. **The §7.4 "never stale" lifecycle.** Click → `202` → button disabled
with **Syncing…** (spinning icon, `aria-busy`) + a 2 s poll of the
status endpoint — the ONLY feedback timer; there is no client-side
hard timeout (a sync can legitimately run for minutes; the server
state is authoritative). Success → enabled, **Synced HH:MM** (local
time of `finished_at`) + the last result in `#sync-result`
(`role="status"` / `aria-live="polite"`; "added" always announced,
zero terms omitted, a no-op run reads `0 added · 1 unchanged`).
Failure → enabled, retry-ready **Sync sources** + the `role="alert"`
banner naming the error. A `409` adopts the in-flight run (never a
second poll loop); a reload mid-sync re-attaches to the running run;
a `403` hides the button (defense in depth).
5. **Idempotent.** Re-syncing an unchanged repo is a fast-forward pull +
sha256 hash skip — nothing re-embedded, the overview left alone
(change-gated), the result `0 added · 1 unchanged`.
6. **Quality gates.** Integration (`tests/integration/test_sync_api.py`):
anonymous 403s; idle → running → success/failed transitions with git +
import + overview mocked; 409 double trigger; `GitSyncError` →
`failed` with the repo named and the import never called; `prune=True`
asserted. Unit (`tests/unit/test_sync_button.py`, frontend-assertion):
the ship-hidden markup, the header reveal, the state machine (2 s
poll, 202/409/403 branches, terminal labels, single-poll guard, no
client timeout), the CSS states (spin + reduced-motion opt-out,
disabled, focus-visible, contrast ≥ 4.5:1, 44 px touch floor).
Coverage `app/` > 90 % (`app/api/sync.py` fully covered).
7. **E2E (this story's gate).** `tests/e2e/test_sync_button.py`, run in
isolation: a real local `file://` git fixture repo (deterministic, no
network — git is a documented environment prerequisite, phase 28) with
the mock LLM proves the admin-only visibility, the full lifecycle
against the REAL clone → import → overview path (including the
idempotent second run and the fresh `kb_overview` row), and the 409
double trigger.
## Playwright Mapping Rule
`tests/e2e/test_sync_button.py` — run in isolation (Chromium +
`podman compose up -d db` + git on PATH; mock LLM, no live aipi). The
module overrides the session app fixture with per-module env
(`BOR_GIT_SOURCES=file://<fixture repo>`, its own `BOR_SOURCES_DIR`) and
truncates the KB tables before each test (the E2E isolation pattern):
1. `test_anonymous_sees_no_button` → AC 1 + 2 (the button never leaves
`hidden`; both endpoints 403).
2. `test_admin_sync_lifecycle` → AC 2 + 4 + 5 (button visible for the
admin; click → **Syncing…** (disabled) → **Synced HH:MM** +
`1 added`; the fixture path `notes/sync-fixture.md` in the Sources
table; the `kb_overview` row fresh and non-empty (DB check); the
idempotent second run → `0 added · 1 unchanged`).
3. `test_double_trigger_409` → AC 2 (second trigger while running → 409
with the exact detail; the single in-flight run still completes).
-128
View File
@@ -1,128 +0,0 @@
# Story: Animated Background That Actually Animates
> **SUPERSEDED (owner direction 2026-08-25):** the phase-22 motion
> design (grid drift + whole-layer breathe) is superseded by the
> owner's "no movement, only fading light" direction — see
> `.agent/user_stories/background-no-motion.md` (phase
> `25_background_no_motion`). The phase-22 history below is preserved
> as-is; the old E2E suite now pins the phase-25 contract as a
> regression.
**Phase:** `22_background_animation` · **E2E:** `tests/e2e/test_background_animation.py`
## Narrative
As **the owner**, I reported (2026-08-24) that the Brain of Reese
background "just blinks". I want the background the phase-08 design
promised — a slow, seamless grid drift plus a gentle glow breathe — to
be *perceived* as smooth motion in a real browser, so the app feels
alive but calm, not like a broken flicker.
- **Given** the phase-08 pure-CSS background (a 60s grid drift masked
into a small top band + a 14s glow opacity swing)
- **When** the page is observed in a real Chromium viewport
- **Then** both layers visibly and smoothly animate — the grid drift
reads as continuous motion and the glow reads as breathing — with no
blink, no static frame, no jank, no new overflow at 360px, and no
impact on text contrast or interactivity.
## Bug report (verbatim, TODO.md L5)
> Fix background animation not working, just blinking
## Owner-confirmed (2026-08-24, roadmap A3)
1. **Intended effect = the phase-08 design comments:** a seamless 60s
grid drift (one cell per loop) + a 14s eased glow breathing. The fix
serves that design — it is not a redesign.
2. **Pure CSS, zero JS** (phase-08 anchor) — no animation JS, no new
assets, no `filter: blur`.
## Root cause (found in task 01 — `.agent/reports/22_background_animation/`)
Both layers *were* animating and nothing occluded them (occlusion
disproven with a red-line probe) — but:
1. **The grid drift was invisible.** 1px lines at 35% `--line` alpha
blend to a ≈10/12/18-level delta over the page bg, and the radial
mask faded the layer to ~zero past the top ~25% of the viewport. A
0.73px/s drift of that faint texture sits below the perception
threshold (measured ≈1.4/765 mean pixel change over 5s in the grid
zone; ≈0 in the bottom half — no grid there at all).
2. **The glow swing was the only visible motion.** The 0.65↔1.0 opacity
swing (35% amplitude, frame-diff concentrated exactly on the two
glow centers) was 100% of the perceived animation → it read as a
blink/pulse, not breathing.
## Fix (styles.css — pure CSS, zero JS, no blur, palette untouched)
| property | phase 08 | phase 22 |
|---|---|---|
| grid line alpha | 35% `--line` ×2 | **60% `--line` ×2** |
| grid mask | `120% 90% at 50% 0%, black 25%, transparent 78%` | **`140% 110% at 50% 0%, black 40%, transparent 90%`** |
| grid drift | 60s, 0→44px | **unchanged** (one-cell seamless loop) |
| glow opacity swing | 0.65↔1 | **0.85↔1** |
| glow period/scale | 14s ease-in-out alternate, 1↔1.05 | **unchanged** |
Both layers keep `position: fixed; inset: 0; z-index: -1;
pointer-events: none`; `<html>` keeps the `var(--bg)` canvas and
`<body>` stays transparent (the no-occlusion contract); the
`prefers-reduced-motion` stilling block is untouched.
## Acceptance criteria
1. In a real Chromium viewport, both background layers run their
animations (grid drift + glow breathe) and the animation timelines
advance — the background is not a static or paused frame.
2. No blink: the glow reads as breathing (0.85↔1), not a pulse
(0.65↔1); the grid drift reads as smooth continuous motion.
3. Layers stay behind content: `position: fixed`, `z-index: -1`,
`pointer-events: none`, full-viewport `inset: 0`; `<html>` keeps
the page background and `<body>` stays transparent (no occlusion).
4. No new horizontal overflow at 360px (the phase-07 pin).
5. Pure CSS, zero JS, no `filter: blur`, no new assets (phase-08
anchor); WCAG AA palette untouched (the layers carry no text).
6. Regressions green in isolation: `test_dark_tech_theme.py` (pins the
live 60s/14s durations + reduced-motion stilling) and
`test_responsive_polish.py`.
7. Unit + integration green, `app/` coverage ≥90%, story E2E green in
isolation, ruff + pyright clean.
## UI Visualization & Structure
- **Grid layer (`body::before`):** 44px cells, 1px lines at 60% of
`--line` (`rgb(38 48 74 / 0.6)`), radial mask fully visible to 40% of
the radius and faded out by 90%, 60s linear infinite drift of
exactly one cell (0→44px) — seamless loop.
- **Glow layer (`body::after`):** indigo `rgb(109 120 242 / 0.14)`
56rem circle at 12%/8% + cyan `rgb(34 211 238 / 0.10)` 60rem circle
at 88%/92%; 14s ease-in-out infinite alternate, opacity 0.85↔1 +
scale 1↔1.05.
- **Stacking:** the `<html>` canvas (`var(--bg)` = `#0a0e17`) sits
under both `z-index: -1` layers; `<body>` stays transparent and
non-stacking, so nothing can paint over the layers.
- **Motion:** compositor-friendly `background-position` / `opacity` /
`transform` only; no blur filter; `prefers-reduced-motion` stills
both layers (phase-08 guard, unchanged).
## Playwright Mapping Rule
**Test Scenario → `tests/e2e/test_background_animation.py`** (the
layers are CSS pseudo-elements — asserted via computed style + the Web
Animations API; Chromium enumerates pseudo-element CSS animations in
`document.getAnimations()`, not `document.body.getAnimations()`):
1. `test_grid_layer_animation_running` — computed style of
`body::before`: `animationName` = `bg-grid-drift`, timing function
`linear`, iteration count `infinite`; a matching entry in the
document animation list with `playState === "running"`.
2. `test_glow_layer_animation_running` — same for `body::after` with
`bg-glow-breathe`; `playState === "running"`.
3. `test_animations_advance` — sample `currentTime` of both layers,
wait ~500ms, assert both advanced — truly running, not paused
(the headless document animation timeline starts ~1s after load, so
the sample polls until the timeline is alive first).
4. `test_background_layers_contracts` — both pseudo-elements:
`position: fixed`, `z-index: -1`, `pointer-events: none`,
`inset: 0`; `document.documentElement` computed background is the
palette bg (canvas stays on `html`); `document.body` computed
background is `rgba(0, 0, 0, 0)` (no occlusion).
5. `test_no_horizontal_overflow_with_layers` — 360px viewport:
`documentElement.scrollWidth <= clientWidth` (the phase-07 pin,
replicated locally).
-181
View File
@@ -1,181 +0,0 @@
# Story: A Background That No Longer Moves — Only Fading Light
**Phase:** `25_background_no_motion` · **E2E:**
`tests/e2e/test_background_no_motion.py`
**Supersedes:** `.agent/user_stories/background-animation.md`
(phase-22 motion design — history preserved there)
## Narrative
As **the owner**, I reported (2026-08-25, chat) that the phase-22
background "jitters down and to the right every second and it slowly
blinks brighter and darker". I want the background to be smooth,
fluxuating, dimming and brightening — but **not moving** — with
**different bright spots** that slowly fade in and out.
- **Given** the phase-22 animated background (a 60s one-cell grid drift
at 0.73px/s diagonally down-right + a 14s whole-layer glow breathe of
opacity 0.85↔1 with scale 1↔1.05)
- **When** the page is observed in a real Chromium viewport
- **Then** nothing moves — the grid is a static texture and no
background keyframe animates anything but `opacity` — while three
independent bright spots each fade in and out on their own slow,
out-of-phase cycles (26s/34s/42s), so the total light fluxuates
smoothly and irregularly: no jitter, no blink, no static frame, no
new overflow at 360px, and no impact on text contrast or
interactivity.
## Owner report (verbatim, 2026-08-25, chat)
> It should be smooth, fluxuating, dimming and brightening, but not
> moving. Different bright spots should slowly fade in and out.
## Owner direction (2026-08-25)
1. **No movement** — no grid drift, no `scale`/`transform`, no
`background-position` animation, anywhere in the background.
2. **Fluxuating brightness** — overall page brightness varies smoothly
and irregularly (not one synchronized pulse).
3. **Different bright spots** — multiple glow spots, each fading in
and out on its own slow cycle.
4. **The static grid stays** — the owner rejected the grid's *motion*,
not the grid; it remains as a still texture. (If the owner later
wants the grid gone, that is a follow-up, not this phase.)
## Root cause (found from the code + phase-22 measurements)
1. **"Jitters down and to the right every second" = the grid drift.**
`bg-grid-drift` moved the 44px grid 44px per 60s (≈0.73px/s)
diagonally down-right — exactly the reported direction. A 1px grid
line translated sub-pixel by sub-pixel is rasterized with per-frame
stepping/shimmer, not smooth motion: it reads as a once-per-second
jitter. Phase 22 had made that drift *visible* (60% line alpha,
wider mask — measured 1.82/765 mean pixel change over 5s in the
grid zone); that is precisely why it now reads as jitter.
2. **"Slowly blinks brighter and darker" = the whole-layer breathe.**
`bg-glow-breathe` swung the ENTIRE glow layer's opacity 0.85↔1 over
14s (alternate) plus `scale(1)↔scale(1.05)` (a faint zoom). One
synchronized pulse of the whole background reads as a blink; the
owner wants independent spots instead.
Phase 22 served the phase-08 design intent (grid drift + whole-layer
breathe). The owner's 2026-08-25 direction supersedes that **design
intent** — no LOCKED anchor changed (A11 stays pure CSS / zero JS / no
CDN / no new assets; the no-`filter: blur` perf anchor is honored).
## Fix (styles.css — pure CSS, zero JS, no blur, palette untouched)
| property | phase 22 | phase 25 |
|---|---|---|
| grid (`body::before`) | `bg-grid-drift 60s linear infinite` (0→44px) | **static** — no animation, `bg-grid-drift` deleted (44px cells, 60% `--line` 1px lines, widened mask kept) |
| glow layer count | one whole-layer breathe | **three independent spot layers** |
| spot A (`body::after`) | indigo + cyan spots, 14s opacity 0.85↔1 + scale 1↔1.05 | **indigo `rgb(109 120 242 / 0.14)` 56rem at 12%/8%**, `bg-glow-a` **26s** ease-in-out infinite, low opacity **0.25** |
| spot B (`html::before`) | — (cyan shared body::after) | **cyan `rgb(34 211 238 / 0.10)` 60rem at 88%/92%**, `bg-glow-b` **34s** ease-in-out **−12s** infinite, low **0.20** |
| spot C (`html::after`) | — | **indigo `rgb(109 120 242 / 0.09)` 52rem at 14%/86%**, `bg-glow-c` **42s** ease-in-out **−23s** infinite, low **0.15** |
| keyframes | `bg-grid-drift` (background-position), `bg-glow-breathe` (opacity + transform) | **`bg-glow-a/b/c` — opacity only** (0%/100% low → 50% 1) |
| reduced motion | stills `body::before/::after` | stills **all four** layers |
All four layers keep `content: ""; position: fixed; inset: 0;
z-index: -1; pointer-events: none`. `<html>` keeps the `var(--bg)`
canvas and `<body>` stays transparent (the no-occlusion contract):
`html` is the root stacking context, so its `z-index: -1`
pseudo-elements paint above the canvas and below the transparent,
non-stacking `<body>`'s content. The 26/34/42s periods with negative
delays (LCM 4641s) keep the cycles out of phase — the composite
pattern effectively never repeats within a viewing session.
## Acceptance criteria
1. **No movement:** the grid is static (`body::before` computed
`animationName: none`; no `bg-grid-drift` in
`document.getAnimations()`; grid texture still painted) and —
audited in real Chromium via `document.styleSheets` — **no `bg-*`
keyframe animates anything but `opacity`** (the deterministic
no-movement proof).
2. **Three distinct bright spots** (`body::after`, `html::before`,
`html::after`) run distinct slow opacity fades (26s/34s/42s,
ease-in-out, infinite, pairwise distinct, out of phase); all three
timelines advance; the layer's computed opacity AND a clipped
screenshot of the bottom-left glow region measurably change within
a few seconds (a real fade, not a frozen frame).
3. **Contracts hold:** all four layers `position: fixed`, `z-index:
-1`, `pointer-events: none`, full-viewport `inset: 0`; `<html>`
keeps the `var(--bg)` canvas (`rgb(10, 14, 23)`) and `<body>` stays
transparent (`rgba(0, 0, 0, 0)`) — no occlusion.
4. **Reduced motion** stills all four layers
(`animationName: none`), the static grid + spot images remain.
5. No new horizontal overflow at 360px (the phase-07 pin).
6. Pure CSS, zero JS, no `filter: blur`, no new assets (A11 + phase-08
perf anchor); WCAG AA palette untouched (the layers carry no text).
7. Regressions green in isolation:
`tests/e2e/test_background_animation.py` (adapted to the phase-25
contract), `tests/e2e/test_dark_tech_theme.py` (grid static +
26/34/42s spots; reduced motion across all four layers),
`tests/e2e/test_responsive_polish.py`.
8. Unit + integration green, `app/` coverage >90%, story E2E green in
isolation, ruff + pyright clean.
## UI Visualization & Structure
- **Grid layer (`body::before`):** 44px cells, 1px lines at 60% of
`--line` (`rgb(38 48 74 / 0.6)`), widened radial mask
(`140% 110% at 50% 0%, black 40%, transparent 90%`) — a STATIC
texture, no animation.
- **Glow spot A (`body::after`):** indigo `rgb(109 120 242 / 0.14)`
56rem circle at 12%/8% (phase-08 position/color); opacity-only fade
0.25↔1 over 26s ease-in-out.
- **Glow spot B (`html::before`):** cyan `rgb(34 211 238 / 0.10)` 60rem
circle at 88%/92% (phase-08 position/color); fade 0.20↔1 over 34s,
−12s delay.
- **Glow spot C (`html::after`):** indigo `rgb(109 120 242 / 0.09)`
52rem circle at 14%/86%; fade 0.15↔1 over 42s, −23s delay.
- **Stacking / no occlusion:** the `<html>` canvas
(`var(--bg)` = `#0a0e17`) sits under all four `z-index: -1` layers;
`<body>` stays transparent and non-stacking, so nothing can paint
over the layers — verified live, not assumed.
- **Motion:** opacity-only keyframes (compositor-friendly); no
`transform`, no `background-position`, no `filter` anywhere in the
background; `prefers-reduced-motion: reduce` stills all four layers
(the static background remains visible).
## Playwright Mapping Rule
**Test Scenario → `tests/e2e/test_background_no_motion.py`** (the
layers are CSS pseudo-elements — asserted via computed style + the Web
Animations API + a live `document.styleSheets` audit; Chromium
enumerates pseudo-element CSS animations in `document.getAnimations()`,
not `document.body.getAnimations()`, and the `html` pseudo-layers'
computed styles come from
`getComputedStyle(document.documentElement, "::before"/"::after")`):
1. `test_grid_layer_is_static` — computed `animationName` of
`body::before` is `"none"`; no `bg-grid-drift` entry in
`document.getAnimations()`; the grid `backgroundImage` is still
present (the static texture survives).
2. `test_three_glow_layers_run_distinct_fades` — `body::after` →
`bg-glow-a` (26s), `documentElement::before` → `bg-glow-b` (34s),
`documentElement::after` → `bg-glow-c` (42s); each `ease-in-out` +
`infinite`, with a matching `playState === "running"` entry in the
document animation list; the three durations are pairwise distinct.
3. `test_no_motion_properties_in_background_keyframes` — walk
`document.styleSheets`; for every `CSSRule.KEYFRAMES_RULE` whose
name starts with `bg-`, collect the declared property names of
every keyframe frame; the set across all frames is exactly
`{"opacity"}` — the deterministic no-movement proof.
4. `test_glow_timelines_advance` — poll until all three timelines
report `currentTime > 0` (headless Chromium starts the document
timeline ~1s after load), sample all three, wait ~500ms, each
advanced ≥ 200ms.
5. `test_background_light_actually_changes` — (a) the computed opacity
of `body::after` changes by ≥ 0.05 within ~8s (a real fade, not a
frozen frame); (b) two clipped screenshots ~4s apart of the
bottom-left glow region (the `html::after` spot at 14%/86%) differ
in bytes — the light visibly changes while nothing moves.
6. `test_background_layers_contracts` — all four pseudo-layers:
`position: fixed`, `z-index: -1`, `pointer-events: none`,
top/right/bottom/left all `0px`; `documentElement` computed
background is `rgb(10, 14, 23)` (canvas stays on `html`);
`document.body` computed background is `rgba(0, 0, 0, 0)` (no
occlusion).
7. `test_reduced_motion_stills_all_layers` —
`reduced_motion="reduce"` context: all four pseudo-layers report
computed `animationName` `"none"` and still carry a
`backgroundImage`.
8. `test_no_horizontal_overflow_with_layers` — 360px viewport:
`documentElement.scrollWidth <= clientWidth` (the phase-07 pin).
-91
View File
@@ -1,91 +0,0 @@
# Story: Cache Busting (Un-stick the Pages)
**Phase:** `33_cache_busting/` · **E2E:** `tests/e2e/test_cache_busting.py`
## Narrative
As **anyone using Brain of Reese**, I want a new deploy to be **visible
without a hard refresh**. Today the five HTML pages reference their CSS/JS
with no version at all, so the browser keeps serving stale assets long
after the app has moved on — the "the pages are too sticky" report. I want
a deploy (a commit) to change what the browser fetches, automatically,
with no new services and no CDN.
- **Given** the app has been redeployed (a new commit)
- **When** I open — or revisit — any page
- **Then** the page itself is always revalidated (never served from cache
unchecked), its assets are fetched from **versioned URLs**
(`?v=<token>` — the git short SHA of the deploy, so each commit changes
them), and the API — the SSE chat stream in particular — is left
byte-for-byte alone.
## Acceptance criteria
1. **HTML always revalidates.** All five pages (`/`, `/index.html`,
`/sources.html`, `/document.html`, `/login.html`, `/tuning.html`) ship
`Cache-Control: no-cache` (the page is the cheap redirector to the
long-cached assets).
2. **Assets are versioned.** Every local `href`/`src` asset reference on
those pages carries `?v=<token>` — rewritten in flight by one
middleware (a small regex over `href`/`src` `assets/…` refs, leading
slash optional, idempotent: an existing query string or fragment is
never double-tagged, and non-asset references pass through). The token
is non-empty and **stable across requests** within a process (one token
per process, computed once).
3. **Immutable assets.** `/assets/*` responses ship
`Cache-Control: public, max-age=31536000, immutable` — safe precisely
because the URL carries the token (a deploy changes the token, hence
the URL). The unversioned path keeps resolving (the static mount
ignores the query string), so old tabs and direct links still work.
4. **The token is the deploy.** `app/core/caching.py::asset_version()`:
a git checkout (the project root next to `frontend/` has a `.git` —
the homelab reality) → `git rev-parse --short HEAD` with a 5 s timeout
(a deploy must not hang a boot); no `.git`, git missing, non-zero
exit, timeout, or empty output → the first **12 hex chars** of SHA-256
over the sorted `relpath:mtime_ns:size` of every regular file under
`frontend/`; a missing/empty static dir → `"dev"`. Cached with
`functools.cache` (== `lru_cache(maxsize=None)`) — zero per-request
git/file cost. A new commit or a frontend content change flips the
token on the next process start.
5. **The API is untouched.** The middleware touches exactly two response
shapes (the page paths: body rewrite + `no-cache`; `/assets/*`:
header only). Everything else — all of `/api/*`, including the SSE
chat stream — passes through with no header changes and no body read
(SSE keeps the `no-cache` its endpoint sets itself). The no-CDN
integration test stays green (the rewritten references are
same-origin).
6. **Quality gates.** Unit (`tests/unit/test_caching.py`): the token's
git path, fallback path (stable across an unchanged tree, flips on a
touch/modify + `cache_clear()`), failure path (git raising → content
hash, no exception), empty-dir `"dev"`; the pure `rewrite_asset_refs`
(versioned `href`/`src`, no-leading-slash refs, module scripts,
idempotency, `#fragment`/existing-query left alone, non-asset refs
untouched). Integration (`tests/integration/test_api.py`): every page
`no-cache` + versioned references, asset headers, no `cache-control`
injected on `/api/health`, SSE chat tests unchanged and green.
Coverage: `app/core/caching.py` 100 %, `app/` > 90 % (TOTAL ≥
pre-change).
7. **E2E (this story's gate).** `tests/e2e/test_cache_busting.py`, run in
isolation: a real (fresh-profile) Chromium asserts the **wire truth**
— the document responses are `no-cache`, the CSS/JS request URLs the
browser actually makes carry one shared token (matching this
checkout's `asset_version()`), the asset responses are immutable for a
year, and `/api/*` — including a live SSE chat turn that streams
deltas and completes with `done` — is unaffected (mock LLM, no live
aipi; `podman compose up -d db` for the chat check).
## Playwright Mapping Rule
`tests/e2e/test_cache_busting.py` — run in isolation (Chromium +
`podman compose up -d db`; mock LLM via the shared `conftest.py` session
app; no per-module overrides needed):
1. `test_html_pages_are_no_cache_and_versioned` → AC 1 + 2 + 3 (the `/`
document response is `no-cache`; the `styles.css` request URL carries
`?v=<token>` and its response is `immutable` + `max-age=31536000`;
the `app.js` request URL carries the **same** token; the served HTML
carries no unversioned `"/assets/styles.css"` reference).
2. `test_other_pages_share_the_token` → AC 1 + 2 (the `/sources.html` and
`/login.html` document responses are `no-cache`; both pages'
stylesheet requests carry the same token).
3. `test_api_responses_unaffected` → AC 5 (`/api/health` has no
`cache-control` injected — the endpoint's baseline headers only; the
SSE chat POST still streams deltas and completes with `done`).

Some files were not shown because too many files have changed in this diff Show More