chore(agent): phase roadmap from TODO.md — 8 phases (40–47), 24 tasks

Converts the 9 TODO items into an executable phase roadmap (Protocol B,
appended after phase 39):

- 40 tuning toggle anonymous flash (TODO L3)
- 41 sync fail-fast + modal when a model is down (TODO L4)
- 42 no reply autoscroll (TODO L5)
- 43 thinking scroll back — user scroll + gated autoscroll (TODO L7)
- 44 markdown tables (TODO L6)
- 45 agent unlimited tool calls behind BOR_AGENT_MAX_ROUNDS (TODO L8)
- 46 mobile hamburger nav (TODO L9)
- 47 quadlet + jinja import formats, A9 revision (TODO L10–L11)

Each phase carries a user story, a dedicated Playwright E2E suite plan,
and owner-locked decisions (R1 A9 format extension, R2 phase-37 budget
revision, A1–A5 scope decisions) confirmed 2026-08-27.

Also records the completed phases 30–39 todo/ -> complete/ moves that
were pending in the working tree. TODO.md is cleared (items now live in
.agent/phases/todo/).
This commit is contained in:
2026-08-27 18:25:53 -04:00
parent 492d8275e7
commit 02c76ad328
66 changed files with 1906 additions and 0 deletions
@@ -1,48 +0,0 @@
# Phase 30 — Document Summaries (lite-model summaries for non-markdown documents)
**Source:** `TODO.md L3 — "One issue I'm having is bad context for the embedder which causes poor retrieval results… we need a small model to analyze non markdown documents and provide a textual summary of those documents with a pointer back to the source… if retrieval == summary, fetch documents referenced by summary… The small model available on aipi.reeseapps.com is 'lite'."`
**Story:** `.agent/user_stories/document-summaries.md`
**Context:** The importer (`app/rag/importer.py::_index_file` — chunk → embed → upsert per file, A9 scope), hybrid retrieval (`app/rag/retriever.py` — A7: cosine ∪ FTS, RRF, chunk→parent-document mapping, full-document context never truncated, phase 24), the locked persona prompts (`app/rag/prompts.py`), the aipi client (`app/rag/llm.py` — A5: `turbo` chat streaming + `embed` embeddings), and the E2E mock LLM (`tests/e2e/mock_llm.py` — deterministic, keys on system-prompt markers like `DEFLECT_MODE` / `<tuning>`).
## Objective
Give every **non-markdown** A9 document (txt, yaml, yml, json, py) a natural-language summary generated at import time by the aipi **`lite`** model. The summary is stored on the document (`documents.summary`) **and indexed as one extra embedded chunk** (`chunks.is_summary`), so hybrid search has a well-embedding natural-language target to hit instead of the badly-formatted raw text. A summary hit resolves to its parent (the source document) — the existing chunk→document mapping then feeds the **full source document** to the LLM, implementing the TODO's "if retrieval == summary, fetch the documents referenced by the summary" step. The per-turn log line records how many summary hits landed in the selected context.
## Dependencies
- `29_tuning_nav_link` (complete) — the latest finished phase (sequencing only).
- Substantively builds on: `02_story_import_documents` / `24_whole_document_context` (import pipeline + full-document context contract), `09_story_retrieval_quality` (A7 hybrid retrieval the summary chunk flows through unchanged), `01_infrastructure` (models/alembic, LLM client, E2E mock).
## Tasks
1. `01_lite_model_client.md` — `BOR_LLM_SUMMARY_MODEL` (default `lite`) + non-streaming `LLMClient.chat()` for the lite model.
2. `02_migration_summary_columns.md` — Alembic 0004: `documents.summary TEXT NULL` + `chunks.is_summary BOOLEAN NOT NULL DEFAULT FALSE`.
3. `03_summarizer_module.md` — `app/rag/summarizer.py`: `SUMMARY_MODE` prompt (capped input), lite call, output validation + deterministic `Source: <source>/<path>` pointer line.
4. `04_importer_summary_integration.md` — importer generates/stores/indexes summaries for non-md files (best-effort fail-soft) + summary counters.
5. `05_pipeline_summary_resolution.md` — `is_summary` through the retriever, `summary_hits` in `TurnPlan` + the per-turn log line; full source document on summary hit (existing mapping, asserted).
6. `06_mock_and_e2e.md` — deterministic `lite` in `mock_llm.py`, sentinel fixture, `tests/e2e/test_document_summaries.py`, story file, README, commit.
## Testing & Quality
- Unit: summarizer (prompt/cap/pointer/errors), importer (summary happy path, md exclusion, fail-soft, replacement on re-import), retriever (`is_summary` through both candidate lists + `fuse`), chat gate (`TurnPlan.summary_hits`), LLM client (`chat()`).
- Integration: migration 0004 up/down.
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`, TOTAL ≥ pre-change number).
- E2E (mandatory, A16): `tests/e2e/test_document_summaries.py` — one story, run **in isolation** (`uv run pytest tests/e2e/test_document_summaries.py -v --no-cov`); proves summary hit → full source document reaches the answer (sentinel in the raw doc, absent from the mock summary).
- All existing E2E suites stay green (new columns are defaulted; all existing chunks have `is_summary=false`).
## Completion Criteria
- [ ] After `uv run python -m scripts.import_docs`, every non-markdown fixture/doc has `documents.summary` set and exactly one `is_summary` chunk (position −1, embedded); markdown docs have neither.
- [ ] A question whose best match is a summary chunk yields an answer grounded in the **full source document** (E2E sentinel) and the per-turn log line shows `summary_hits>=1`.
- [ ] A lite-model failure during import does **not** drop the document — it is indexed without a summary, logged, and counted (`summary_errors`).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
- [ ] `uv run pytest tests/e2e/test_document_summaries.py -v --no-cov` green in isolation; existing suites (`test_chat_rag.py`, `test_retrieval_quality.py`, `test_import_documents.py`, `test_whole_document_context.py`) stay green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/document-summaries.md` exists.
- [ ] `.env.example` + README document `BOR_LLM_SUMMARY_MODEL` / `BOR_SUMMARY_MAX_CHARS` and the summary behavior.
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc`); `.agent/phases/todo/30_document_summaries/` moved to `.agent/phases/complete/`.
## Locked decisions
- **A5 extended, not revised** — the `lite` model is served by the same OpenAI-compatible endpoint (`https://aipi.reeseapps.com/v1`) via a new `BOR_LLM_SUMMARY_MODEL` setting (default `lite`); no new model management, no new package.
- **A7 untouched** — hybrid retrieval logic is unchanged; the summary is an ordinary chunk, so it flows through the existing cosine ∪ FTS ∪ RRF path and the chunk→document mapping. The "fetch the referenced document" step is the existing full-document context contract (phase 24) — never truncated.
- **A9 untouched** — "non-markdown" means every *already-imported* A9 document except `md`/`markdown`. The TODO's quadlet-file example is **out of scope**: `.quadlet` is not an A9 format and `BOR_IMPORT_EXTENSIONS` may only narrow the locked set (flagged at roadmap confirmation; importing quadlet files would require an owner-permission A9 revision).
- **A13** — migration 0004 adds two columns (`documents.summary`, `chunks.is_summary`); no table rework, both reversible.
- **Summary generation is best-effort** — a lite failure logs + counts (`summary_errors`) and the file is still indexed without a summary (same fail-soft spirit as the per-file `EmbeddingError` handling, but weaker: the doc is already committed).
- **Pointer is code-deterministic** — the `Source: <source>/<path>` line is appended by `summarizer.py`, never trusted to the model.
- **A16 honoured** — one dedicated story E2E suite; E2E stays deterministic via the mock LLM's `SUMMARY_MODE` marker.
- **A17 honoured** — one atomic `--no-gpg-sign` commit.
@@ -1,35 +0,0 @@
# Task 06 — Deterministic lite mock + story E2E + docs + commit
**Phase:** `30_document_summaries` · **Source:** `TODO.md:3 — (whole item: bad embedder context for non-markdown docs → lite summaries → summary hits fetch the referenced source; the aipi 'lite' model)`
**Story:** `.agent/user_stories/document-summaries.md`
## Objective
Close the loop: a deterministic `lite` behavior in the E2E mock, a sentinel fixture proving that **a summary hit still delivers the full source document to the LLM**, the story E2E suite, the story file, README docs, and the phase commit.
## Work
1. `tests/e2e/mock_llm.py` — in the chat-completions handler (non-stream and stream paths), **before** the `DEFLECT_MODE` check: if the system prompt contains `SUMMARY_MODE`, return the deterministic digest
`f"This document covers {' '.join(TOKEN_RE.findall(_user(body).lower())[:24])}."`
— the first 24 tokens of the document content (the summarizer puts the content in the *user* message). Byte-stable for a given fixture.
2. E2E fixture — a new non-markdown fixture doc, e.g. `quadlet/qwen-llamacpp.yaml` under the existing E2E fixture KB (follow the import-dependent fixtures' pattern in `tests/e2e/conftest.py` / the fixture dir used by `test_whole_document_context.py`):
- The document **opens** with a header comment line dense in the question tokens (e.g. `# qwen 3.8 llama.cpp optimal parameters deployment notes`) so the mock's 24-token summary digest contains the question's words, followed by ~4–5 k of other yaml content (so the raw chunks dilute their overlap and the summary chunk ranks first — the mock's embeddings are a pure function of tokens, so the ranking is fully deterministic for a fixed fixture; iterate the fixture text until the E2E assertions hold).
- A unique sentinel `RESE-SUMMARY-SENTINEL-7f3a` on the **last line** of the document (outside the 24-token digest, unreachable from the summary).
3. `tests/e2e/test_document_summaries.py` (new, the story gate) — reuse the E2E conftest app/DB fixtures:
- Import the fixture KB (re-import pattern used by import-dependent stories).
- Ask the question (e.g. "What are the optimal parameters for qwen 3.8 on llama.cpp? show the end of your notes" — the phase-24 tail-echo trigger makes the answer quote the **last 160 chars of the document context**).
- Assert the rendered brain answer contains `RESE-SUMMARY-SENTINEL-7f3a` → the full **source** document was in the LLM context (only possible via the summary→parent-document resolution, since the summary digest cannot contain the sentinel).
- Assert the source chip shows the fixture doc's path and `deflected` is false.
- Control: a markdown fixture doc in the same KB gets **no** summary chunk — assert via the Sources table (admin, `#docs-table`) or a direct DB check in the test: markdown doc's chunk count == raw chunks only; the yaml doc has exactly one `is_summary` row.
4. `.agent/user_stories/document-summaries.md` (new) — narrative + acceptance criteria + Playwright mapping rule (story → `tests/e2e/test_document_summaries.py`), matching the style of `.agent/user_stories/git-sources.md`.
5. `README.md` — new "Document summaries" section: what gets summarized (non-markdown A9 docs), the `Source:` pointer, `BOR_LLM_SUMMARY_MODEL` / `BOR_SUMMARY_MAX_CHARS`, fail-soft behavior, and how summary hits appear in the per-turn log.
6. Commit: `git add` the phase's app/script/test/README files; `git commit --no-gpg-sign -m "feat(rag): lite-model document summaries — non-markdown docs summarized at import, summary chunk retrieves and resolves to the full source doc"`; move `.agent/phases/todo/30_document_summaries/` → `.agent/phases/complete/` (`.agent/` is gitignored by design — force-add only if the commit must record the plan change, otherwise leave the move out of git).
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_document_summaries.py -v --no-cov` green **in isolation** (Chromium installed; `podman compose up -d db` up; mock LLM — no live aipi needed).
- Regression: run `test_chat_rag.py`, `test_retrieval_quality.py`, `test_import_documents.py`, `test_whole_document_context.py` in isolation — all stay green.
- Full gate: `uv run pytest` + `uv run pytest --cov=app --cov-report=term-missing` (TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `tests/e2e/test_document_summaries.py` green in isolation; the sentinel assertion proves summary hit → full source document.
- [ ] All regression suites listed above green in isolation.
- [ ] Full test gate + lint/type gate green (per this phase's 00_phase.md).
- [ ] Story file + README + `.env.example` complete; one `--no-gpg-sign` commit made.
@@ -1,45 +0,0 @@
# Phase 31 — KB Overview in the System Prompt (lite-generated knowledge-base outline)
**Source:** `TODO.md L4 — "The system prompt should inject basic categories of everything that's been read so the agent knows roughly what its knowledge base contains before the rag retrieval returns documents. This part of the system prompt should be generated by the lite model and should be stored somewhere so it can be updated whenever we import new documents."`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
**Context:** Phase 15 steering notes (the `<tuning>` prompt section, its char budget, and the **byte-identical-when-absent** convention — `app/rag/prompts.py::build_steering_section`), phase 30 (the `lite` client method `LLMClient.chat`, and per-document summaries that make a much better overview input than raw titles), `scripts/import_docs.py` (the place "whenever we import new documents" happens), and the E2E mock's answer-echo convention (the `(tuning: …)` suffix — `tests/e2e/mock_llm.py`).
## Objective
Store a lite-generated, plain-text outline of the knowledge base's basic **categories** in a single-row `kb_overview` table, inject it into **both** chat prompts (HIGH and LOW) as a `<knowledge_base>` section so the agent knows roughly what the KB contains before retrieval, and regenerate it automatically whenever an import changes the KB.
## Dependencies
- `30_document_summaries` (todo) — `LLMClient.chat` + `BOR_LLM_SUMMARY_MODEL` (task 01) and the stored per-document summaries (task 04) that feed the overview input.
- `15_steering_notes` (complete) — the prompt-section pattern this phase mirrors (budget, marker, byte-identical-when-absent, per-turn load in `app/api/chat.py`).
- `11_long_answers` / README import workflow (complete) — `scripts/import_docs.py`'s `main()` structure, which this phase extends with the post-import regeneration.
## Tasks
1. `01_migration_kb_overview.md` — Alembic 0005: single-row `kb_overview` table + `KbOverview` model.
2. `02_overview_generator.md` — `app/rag/overview.py`: `KB_OVERVIEW_MODE` prompt builder, `load_kb_overview`, `regenerate_overview` (best-effort upsert).
3. `03_prompt_injection.md` — `<knowledge_base>` section in HIGH + LOW prompts (budgeted, byte-identical when absent); `plan_turn`/chat wire it in; `kb_chars` in the per-turn log.
4. `04_import_trigger.md` — `import_docs` regenerates the overview after a KB-changing import (shared with phase 32's sync).
5. `05_mock_and_e2e.md` — deterministic `KB_OVERVIEW_MODE` mock + `(kb: …)` echo, `tests/e2e/test_kb_overview.py`, story file, commit.
## Testing & Quality
- Unit: overview generator (prompt build/cap, load, regenerate upsert/fail-soft/zero-docs), prompts (section present/budgeted/absent → byte-identical, ordering vs `<tuning>`), chat gate (`kb_chars`, prompt carries the section).
- Integration: migration 0005 up/down; `import_docs` regeneration trigger (changed vs unchanged imports, failure isolation).
- Coverage: **>90%** on `app/` (`app/` TOTAL ≥ pre-change).
- E2E (mandatory, A16): `tests/e2e/test_kb_overview.py` — one story, run **in isolation**; the injected section is observable in the mock answer via the `(kb: …)` echo (steering precedent).
## Completion Criteria
- [ ] After a KB-changing import, `kb_overview` holds a fresh outline (log line `overview: regenerated docs=… chars=…`); an unchanged re-import does **not** call the lite model.
- [ ] Every chat turn's system prompt (HIGH and LOW) contains the `<knowledge_base>` section when a row exists; with no row, both prompts are **byte-identical** to the pre-phase text (unit-asserted).
- [ ] The per-turn log line records `kb_chars=<n>`; section overflow beyond `BOR_KB_OVERVIEW_MAX_CHARS` is capped with the shared `[…truncated…]` marker.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
- [ ] `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` green in isolation; existing prompt/steering/chat suites stay green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/kb-overview-prompt.md` exists; `.env.example` + README document `BOR_KB_OVERVIEW_MAX_CHARS` / `BOR_OVERVIEW_INPUT_MAX_CHARS` and the regeneration behavior.
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, <knowledge_base> section in HIGH+LOW prompts`); `.agent/phases/todo/31_kb_overview_prompt/` moved to `.agent/phases/complete/`.
## Locked decisions
- **A13** — migration 0005 adds one single-row table `kb_overview` (`id INTEGER PK DEFAULT 1`, `content TEXT NOT NULL DEFAULT ''`, `updated_at TIMESTAMPTZ`); no other schema change.
- **A5 extended** — the overview is generated by the same `lite` model via the same `BOR_LLM_SUMMARY_MODEL` setting and `LLMClient.chat` (phase 30); no new model or package.
- **Prompt-section convention (phase 15 precedent)** — the section is budgeted by `BOR_KB_OVERVIEW_MAX_CHARS` (default **4000**) with the shared `TRUNCATION_MARKER` overflow; **zero/empty row → prompts byte-identical** to pre-phase text. Section order: `<relevance>` → `<knowledge_base>` → `<tuning>` → mode body.
- **Regeneration is best-effort and change-gated** — runs only when an import added/updated at least one document (or no row exists yet); a lite failure logs and leaves the previous overview intact (an old outline is better than none).
- **Overview input is capped** — `BOR_OVERVIEW_INPUT_MAX_CHARS` (default **40 000**) on the document list (source/path/title/first summary line) sent to the model.
- **No per-turn LLM call** — chat turns only *read* the stored row (one indexed PK lookup); generation happens at import/sync time (phase 32's button triggers the same `regenerate_overview`).
- **A16 / A17 honoured** — one dedicated story E2E suite; one atomic `--no-gpg-sign` commit.
@@ -1,31 +0,0 @@
# Task 05 — Deterministic KB_OVERVIEW_MODE mock + story E2E + commit
**Phase:** `31_kb_overview_prompt` · **Source:** `TODO.md:4 — (whole item: system prompt injects basic categories of everything read, lite-generated, stored, updated on import)`
**Story:** `.agent/user_stories/kb-overview-prompt.md`
## Objective
Make the overview observable end-to-end in a deterministic E2E: the mock generates a `KB_OVERVIEW_MODE` outline and echoes the injected section into its answer (the `(tuning: …)` precedent), plus the story suite, story file, and the phase commit.
## Work
1. `tests/e2e/mock_llm.py`:
- Generation: in the chat-completions handler, if the system prompt contains `KB_OVERVIEW_MODE` → return the deterministic outline `f"Knowledge base outline:\n- {first 8 tokens of _user(body), space-joined}"` (the user message carries the document list).
- Echo: in `compose_answer` (all answer paths), if the system prompt contains a `<knowledge_base>` section, append `(kb: <first bullet line of the section>)` — parse with a regex in the `first_tuning_note` style (skip the intro line, take the first `-` line, strip the dash). This mirrors the steering echo exactly.
2. `tests/e2e/test_kb_overview.py` (new, the story gate):
- Seed the `kb_overview` row directly in the DB (the E2E test has DB access via the conftest fixtures — content with a recognizable first bullet, e.g. `- Kubernetes cluster and node maintenance notes`), so the test exercises the **injection** path deterministically (the CLI trigger path is covered by task 04's integration tests).
- Ask a normal on-topic question (the fixture KB is already imported by the conftest pattern) → assert the rendered brain answer ends with `(kb: Kubernetes cluster and node maintenance notes)`.
- Deflection control: ask an off-topic question (deflection path) → the answer still carries the `(kb: …)` echo (the section is in the LOW prompt too).
- Absence control: delete the row → a fresh question's answer has **no** `(kb: …)` suffix (byte-identical prompt behavior is unit-asserted in task 03; this proves it end-to-end).
3. Integration test (same task): extend `tests/integration/test_chat_api.py` (or a new `test_kb_overview_api.py`) with the capturing-fake-LLM pattern — no row: the system prompt sent to the model equals the pre-phase construction (assert the exact string via the existing `build_high_prompt`/`build_deflect_prompt` with `kb_overview=None`); row present: it contains the section in both HIGH and LOW turns.
4. `.agent/user_stories/kb-overview-prompt.md` (new) — narrative + acceptance criteria + Playwright mapping rule, styled like the other story files.
5. Commit: `git commit --no-gpg-sign -m "feat(rag): lite-generated KB overview in the system prompt — stored single row, regenerated on import, <knowledge_base> section in HIGH+LOW prompts"`; move `.agent/phases/todo/31_kb_overview_prompt/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` green **in isolation**.
- Regression: `test_steering.py`, `test_chat_rag.py`, `test_honest_deflection.py` stay green in isolation (prompt change is additive and defaulted).
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `tests/e2e/test_kb_overview.py` green in isolation (injection, deflection, and absence all asserted).
- [ ] Integration prompt-capture tests green; existing steering/chat suites green.
- [ ] Full test + lint/type gates green (per this phase's 00_phase.md).
- [ ] Story file + `.env.example` + README complete; one `--no-gpg-sign` commit made.
@@ -1,45 +0,0 @@
# Phase 32 — Admin Sync Button (one-click doc import sync)
**Source:** `TODO.md L5 — "Need a button that only the admin can see that triggers a doc import sync by cloning the relevant repos and then running import doc script"`
**Story:** `.agent/user_stories/admin-sync-button.md`
**Context:** Phase 28 (`scripts/git_sync.py::clone_or_pull` — shallow clone / `--ff-only` pull; `BOR_GIT_SOURCES` + `BOR_SOURCES_DIR`; `repo_name` in `scripts/import_docs.py`), phase 31 (`regenerate_overview` — the sync refreshes the KB outline), phase 16 (`require_admin` dependency + the `header.js` `fetchIsAdmin()` reveal gate for admin-only UI like `#nav-sources` / `#nav-tuning`), PLAN §7.4 "never stale" feedback contract (the UI can never sit on a stale button state).
## Objective
Give the admin a **"Sync sources"** button (Sources page, visible to the admin only) that triggers the full document sync in-process — clone/pull every `BOR_GIT_SOURCES` repo, re-import (with prune) so the KB mirrors the repos, and refresh the KB overview — with live, non-stale UI feedback driven by a polled sync-status endpoint.
## Dependencies
- `31_kb_overview_prompt` (todo) — `regenerate_overview(llm)` is the sync's final step; `LLMClient.chat` for it.
- `28_git_based_sources` (complete) — `clone_or_pull` / `GitSyncError` / `BOR_GIT_SOURCES` / `repo_name` (the sync reuses them, does not re-implement git).
- `16_admin_auth` (complete) — `require_admin` for the new endpoints; the `header.js` whoami gate for the button.
- `19_shared_header` / `29_tuning_nav_link` (complete) — the Sources page header actions area where the button lives.
## Tasks
1. `01_sync_api.md` — in-process sync runner: `POST /api/sync` (admin, 409 when running) + `GET /api/sync/status` (admin).
2. `02_ui_button.md` — the admin-only button on Sources with §7.4 feedback states (polling, last-result, error banner) + frontend unit assertions.
3. `03_e2e_and_docs.md` — `tests/e2e/test_sync_button.py` (real `file://` git fixture), README, story file, commit.
## Testing & Quality
- Integration: sync API — anonymous 403s, admin idle/running/success/failed transitions, 409 double-trigger, GitSyncError → `failed` with the repo named (git + import + overview mocked, as `test_import_docs_git.py` does).
- Unit (frontend-assertion style, cf. `tests/unit/test_shared_header.py`): button markup hidden-by-default + labeled; `header.js` reveal; `sources.js` polling/terminal-state logic.
- Coverage: **>90%** on `app/` (`app/api/sync.py` fully covered); `app/` TOTAL ≥ pre-change.
- E2E (mandatory, A16): `tests/e2e/test_sync_button.py` — one story, run **in isolation**; uses a **real** local `file://` git repo fixture (deterministic, no network) with the mock LLM for embeddings.
- UI Structure Check (AGENTS.md rule 5): labeled button, focus-visible, contrast ≥4.5:1, `aria-live` result region, no CDN.
## Completion Criteria
- [ ] Anonymous: the button is not revealed (stays `hidden`) and both endpoints return 403.
- [ ] Admin: clicking "Sync sources" starts the sync (202), the button goes disabled with "Syncing…" while polling `GET /api/sync/status` every 2 s, and on completion shows the last result (`Synced HH:MM` + `N added · M updated`); a failed sync re-enables the button with an error banner (`role="alert"`) naming the failure.
- [ ] A double trigger while running returns 409 and the UI never starts a second poll loop.
- [ ] After a successful sync against the `file://` fixture repo, the newly committed fixture doc appears in the Sources table and the `kb_overview` row is fresh (phase-31 trigger).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
- [ ] `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` green in isolation; `test_admin_auth.py`, `test_shared_header.py`, `test_import_documents.py` stay green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/admin-sync-button.md` exists; README documents the button (behavior, states, prerequisites).
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status`); `.agent/phases/todo/32_admin_sync_button/` moved to `.agent/phases/complete/`.
## Locked decisions
- **A10 extended (recorded, not a revision)** — two new **admin-only** endpoints (`POST /api/sync`, `GET /api/sync/status`) behind the existing `require_admin`; the public API surface stays stateless, the signed cookie remains the only session state (same pattern as `/api/steering`).
- **A12 untouched** — the sync runs **in-process** (one `asyncio` background task + a module-level status object in `app/api/sync.py`). The app is a single instance on the homelab; no Valkey/queue. Status is in memory — a restart mid-sync loses the running state (accepted: the next click re-syncs idempotently).
- **Sync semantics** — the button targets `BOR_GIT_SOURCES` only (manual `--source` dirs have no repo to clone; an unset/empty `BOR_GIT_SOURCES` → the sync fails loudly with "no git sources configured"); the import runs with **`prune=True`** so files deleted upstream leave the index (the button is the canonical "mirror the repos" action — the CLI default of no-prune is unchanged); phase-31's `regenerate_overview` runs after the import when docs changed.
- **Concurrency** — one sync at a time: `POST /api/sync` while running → `409 {"detail": "a sync is already running"}`; the UI reflects the in-flight run (re-attaches on page load while a sync is running).
- **§7.4 adaptation (recorded)** — the 120 s client guard applies to LLM turns; a sync can legitimately run for minutes (clone + embed), so the button has **no client-side hard timeout** — the 2 s status poll is the feedback loop and the server state is authoritative. The button is disabled until the run reaches a terminal state, so it can never be stale *or* stuck: a failed run re-enables it, a running run always shows "Syncing…".
- **A16 / A17 honoured** — one dedicated story E2E suite (real `file://` git fixture — git is a documented environment prerequisite, as in phase 28); one atomic `--no-gpg-sign` commit.
@@ -1,34 +0,0 @@
# Task 03 — Story E2E (real file:// git fixture) + README + commit
**Phase:** `32_admin_sync_button` · **Source:** `TODO.md:5 — (whole item: admin-only button → clone the relevant repos → run the import script)"`
**Story:** `.agent/user_stories/admin-sync-button.md`
## Objective
The story gate: a deterministic E2E that runs the **real** sync path end-to-end (real `git clone` of a local `file://` fixture repo, real import against the mock LLM, real overview regeneration) and verifies both the admin-only visibility and the full button lifecycle, plus README docs, story file, and the phase commit.
## Work
1. `tests/e2e/test_sync_button.py` (new) — module fixtures:
- Build a temp git repo in a `tmp_path` via `subprocess` (`git init -q`, write `notes/sync-fixture.md` containing a unique sentinel `RESE-SYNC-SENTINEL-9b2c`, `git add -A && git -c user.email=e@x -c user.name=t commit -qm one`); set the app's env for this module: `BOR_GIT_SOURCES=file://<repo>`, `BOR_SOURCES_DIR=<tmp_path>/checkouts` (follow the E2E conftest pattern for per-module app env; git is a documented environment prerequisite — phase 28).
- Truncate `query_log` (and the KB tables the fixture needs) per the existing E2E isolation pattern so the run starts clean.
2. Tests (in isolation):
- **`test_anonymous_sees_no_button`** — load `/sources.html` logged out: `#sync-btn` is `hidden` (or absent from the revealed DOM); `POST /api/sync` via `page.request` → 403.
- **`test_admin_sync_lifecycle`** — sign in (reuse `tests/e2e/auth_helpers.py`):
- `#sync-btn` visible with label "Sync sources".
- Click → button disabled, label "Syncing…".
- Wait (poll with Playwright, generous timeout ~60 s — real git + embed against the mock LLM): label becomes `Synced …`, `#sync-result` shows `1 added` (the fixture doc).
- The Sources table (`#docs-tbody`) now contains the fixture path `notes/sync-fixture.md`; the `kb_overview` row is non-empty (phase-31 regeneration ran — DB check in the test).
- Re-click → a second run completes with `0 added · 1 unchanged` (idempotent pull + hash-skip).
- **`test_double_trigger_409`** (integration-level, may live in `tests/integration/test_sync_api.py` if E2E timing is too flaky — the integration test already covers this; include here only if deterministic): start a sync, immediately `POST /api/sync` again → 409.
3. `README.md` — new "Sync from the UI" subsection under the import workflow: what the button does (clone/pull → import `--prune`-equivalent → KB overview refresh), the states (Syncing…/Synced/error), the 409 behavior, and prerequisites (`BOR_GIT_SOURCES` set; git available).
4. `.agent/user_stories/admin-sync-button.md` (new) — narrative + acceptance criteria + Playwright mapping rule.
5. Commit: `git commit --no-gpg-sign -m "feat(admin): one-click sources sync — admin-only button triggers git clone/pull + re-import + KB overview refresh with polled live status"`; move `.agent/phases/todo/32_admin_sync_button/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` green **in isolation** (Chromium + `podman compose up -d db` + git on PATH; mock LLM — no live aipi).
- Regression in isolation: `test_admin_auth.py`, `test_shared_header.py`, `test_import_documents.py`, `test_global_tuning.py`.
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] All E2E tests green in isolation, including the idempotent second run and the `kb_overview` freshness check.
- [ ] Regression suites green; full test + lint/type gates green (per this phase's 00_phase.md).
- [ ] README + story file complete; one `--no-gpg-sign` commit made.
@@ -1,40 +0,0 @@
# Phase 33 — Cache Busting (un-stick the pages)
**Source:** `TODO.md L6 — "We need better cache busting, the pages are too sticky"`
**Story:** `.agent/user_stories/cache-busting.md`
**Context:** `app/main.py` serves the whole `frontend/` directory through one `StaticFiles(html=True)` catch-all mount; the five HTML pages reference assets **without any version** (`href="/assets/styles.css"`, `src="assets/markdown.js"`, `src="/assets/app.js"`, …), so browsers happily keep stale CSS/JS/HTML after a deploy — the "too sticky" report. The no-CDN integration test (`tests/integration/test_api.py::test_html_pages_served_locally_no_cdn`) asserts no `https://` references — appending `?v=` keeps every reference same-origin, so it stays green. SSE/API live under `/api/*` and must be untouched (SSE already ships `Cache-Control: no-cache` itself).
## Objective
A deploy must be visible without a hard refresh: HTML pages are **always revalidated** (`Cache-Control: no-cache`) and reference their assets with a version token (`?v=<token>`); assets are served **immutable for 1 year** (the token in the URL identifies the content, so long caching is safe). Zero new services, zero build-step changes, no CDN.
## Dependencies
- `32_admin_sync_button` (todo) — sequencing only; no shared code (this phase is transport-layer and independent of the RAG work).
## Tasks
1. `01_asset_version_token.md` — `app/core/caching.py::asset_version()`: git short SHA (homelab checkouts have a `.git`), stable mtime+size content-hash fallback, computed once per process.
2. `02_caching_middleware.md` — the response middleware (HTML `no-cache` + `?v=` rewrite; `/assets/*` immutable) wired into `create_app` + integration tests.
3. `03_e2e_and_docs.md` — `tests/e2e/test_cache_busting.py` (Playwright header assertions), README, story file, commit.
## Testing & Quality
- Unit: version token (git path, fallback path, failure path), the asset-reference rewrite (both `href`/`src` and leading-slash-less `assets/…` refs, no double-`?v=`).
- Integration: page headers + rewritten references; asset headers; no-CDN test green; SSE endpoint responses untouched (existing chat SSE tests green).
- Coverage: **>90%** on `app/` (the new `app/core/caching.py` fully covered); TOTAL ≥ pre-change.
- E2E (mandatory, A16): `tests/e2e/test_cache_busting.py` — one story, run **in isolation**; real Chromium asserting the headers and the versioned request URLs the browser actually makes.
## Completion Criteria
- [ ] Every HTML page (`/`, `/sources.html`, `/document.html`, `/login.html`, `/tuning.html`) is served with `Cache-Control: no-cache` and its asset references carry `?v=<token>` (token non-empty, stable across requests, changes when the frontend content changes).
- [ ] `/assets/*` responses carry `Cache-Control: public, max-age=31536000, immutable`.
- [ ] `/api/*` (incl. the SSE chat stream) responses are byte-for-byte header-wise unaffected beyond what they already send; no-CDN integration test green.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number (app/ >90%).
- [ ] `uv run pytest tests/e2e/test_cache_busting.py -v --no-cov` green in isolation; `test_smoke.py` + one RAG E2E stay green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] `.agent/user_stories/cache-busting.md` exists; README documents the caching behavior + how the token changes on deploy.
- [ ] One `--no-gpg-sign` commit staging only this phase's files (e.g. `perf(ui): cache busting — HTML no-cache + versioned asset URLs (?v=) with immutable 1y asset caching`); `.agent/phases/todo/33_cache_busting/` moved to `.agent/phases/complete/`.
## Locked decisions
- **Version token** — `asset_version()`: if the project checkout has a `.git` (the homelab reality), the token is `git rev-parse --short HEAD` (a commit = a deploy, so the token flips on every deploy); otherwise a stable hash of the frontend tree (sorted `relpath + mtime_ns + size`, first 12 hex chars) so dev checkouts still bust. Computed **once per process** (`lru_cache`) — zero per-request git/file cost.
- **Rewrite scope** — only the five known HTML pages are rewritten (a small regex over `href="…assets/…"` / `src="…assets/…"` appending `?v=` when absent). No templating layer, no build step, no changes to the static files themselves (the `Containerfile` esbuild stage is untouched).
- **Asset caching** — `/assets/*` are cached `immutable` for 1 year **because** the URL carries the token; the unversioned path keeps working (StaticFiles ignores the query string), so old tabs and tests referencing `/assets/x.js` directly still resolve.
- **Middleware boundary** — the middleware touches exactly two shapes: the five page paths (body rewrite + `no-cache`) and `/assets/*` (header only). Everything else — all `/api/*` including SSE — passes through byte-identical (SSE keeps its own `no-cache`). Implemented as a Starlette middleware that only rewrites `text/html` responses under the page paths; if `Response.body()` turns out to misbehave on the `FileResponse` streaming path, the fallback is five explicit FastAPI routes that read + rewrite the files (identical observable behavior — the executor picks whichever passes the tests).
- **A11 untouched** — no CDN, no new packages, no new services (A12 untouched).
- **A16 / A17 honoured** — one dedicated story E2E suite; one atomic `--no-gpg-sign` commit.
@@ -1,32 +0,0 @@
# Task 03 — Story E2E (Playwright header assertions) + README + commit
**Phase:** `33_cache_busting` · **Source:** `TODO.md:6 — (whole item: better cache busting — the pages are too sticky)"`
**Story:** `.agent/user_stories/cache-busting.md`
## Objective
The story gate: a real-browser E2E that asserts what the browser actually receives (HTML `no-cache`, versioned asset request URLs, immutable asset headers), plus README docs, story file, and the phase commit.
## Work
1. `tests/e2e/test_cache_busting.py` (new) — collect responses with `page.on("response")`:
- `test_html_pages_are_no_cache_and_versioned` — navigate to `/`:
- the document response's `cache-control` header is `no-cache`;
- the `styles.css` request URL contains `?v=` and the response's `cache-control` contains `immutable` + `max-age=31536000`;
- the `app.js` request URL contains the **same** token value as the CSS one (single token per process);
- the served HTML (`page.content()`) contains no unversioned `/assets/styles.css"` reference.
- `test_other_pages_share_the_token` — navigate to `/sources.html` then `/login.html`: each document response is `no-cache`; both pages' CSS requests carry the same token.
- `test_api_responses_unaffected` — `page.request.get("/api/health")` → no `cache-control: no-cache`/immutable injection (the endpoint's baseline headers only); a chat SSE POST still streams (reuse the minimal chat-request helper from an existing E2E — the stream must complete with `done`).
2. `README.md` — new short "Caching / deploys" section: HTML is always revalidated; assets are cached 1 year immutable and carry `?v=<token>`; the token is the git short SHA (falls back to a content hash in non-git checkouts) and flips on every commit/deploy — no hard refresh needed anymore; API/SSE caching is unchanged.
3. `.agent/user_stories/cache-busting.md` (new) — narrative + acceptance criteria + Playwright mapping rule.
4. Commit: `git commit --no-gpg-sign -m "perf(ui): cache busting — HTML no-cache + versioned asset URLs (?v=) with immutable 1y asset caching"`; move `.agent/phases/todo/33_cache_busting/` → `.agent/phases/complete/`.
5. **Deploy note (post-commit, owner action)**: after this phase lands, the *first* deploy also requires browsers to see the new HTML once (revalidation) — one normal navigation; thereafter every commit is picked up automatically.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_cache_busting.py -v --no-cov` green **in isolation** (Chromium + `podman compose up -d db` for the app boot; the mock LLM keeps the SSE check deterministic).
- Regression in isolation: `test_smoke.py`, `test_chat_rag.py`.
- Full gate: `uv run pytest` + coverage (`app/` TOTAL ≥ pre-change) + `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] All E2E tests green in isolation (headers + token consistency + SSE unaffected).
- [ ] Regression suites green; full test + lint/type gates green (per this phase's 00_phase.md).
- [ ] README + story file complete; one `--no-gpg-sign` commit made.
- [ ] `podman compose up -d` (full app) smoke: a fresh browser profile loads the site and every asset request is versioned (manual confirmation recorded in the commit message or phase notes).
@@ -1,46 +0,0 @@
# Phase 34 — One Navbar on Every Page
**Source:** `TODO.md` L3 — "I want the navbar to be consistent between every page. I don't want buttons to pop in and out of existance. Just keep all those buttons active across all tabs."
**Story:** `.agent/user_stories/nav-consistency.md`
**Context:** The shared header (phase 19, `frontend/assets/header.js`) + the Tuning nav link (phase 29) already standardize nav + auth on chat / sources / tuning — but `document.html` still uses the separate `.doc-header` variant (back + title + actions, **no nav at all**), `login.html` misses the Tuning link, and two functional controls remain page-scoped: the Tuning steering toggle + panel (chat only, logic in `app.js`) and the Sync sources button (Sources only, logic in `sources.js`). Owner confirmation (2026-08-26): the bar must be identical on **all** pages — nav, Tuning toggle, Sync, New chat, and the auth pair all present everywhere; the locked A10 UI revision stays (admin-only controls hidden for anonymous, active for the admin on every tab).
## Objective
Make the header bar **identical on all five pages** (chat, sources, document viewer, tuning, login): one shared markup block, one owner of all functional control behavior (`header.js`), the viewer's back link + title preserved in a second titlebar row, and the phase-12/19 height contract (64px desktop / 58px ≤640px) applied to the standard row on every page.
## Dependencies
- `19_shared_header` (complete) — the `header.js` module, the nav/auth markup + ids, the cached-one-whoami contract, the ship-hidden/reveal-for-admin pattern.
- `29_tuning_nav_link` (complete) — the admin-only `#nav-tuning` reveal pattern this phase completes on the remaining pages.
- `15_steering_notes` + `27_global_tuning` (complete) — the steering toggle/panel logic being moved into the shared module; chat-page behavior must not change.
- `32_admin_sync_button` (complete) — the sync button state machine + `GET/POST /api/sync` being moved into the shared module; Sources-page behavior (result line + error banner) must not change.
- `13_document_back_navigation` (complete) — the `#doc-back` target-resolution behavior the viewer titlebar must preserve.
- `16_admin_auth` (complete) — the whoami gate, the soft-gate pages, the sign-out binding.
## Tasks
1. `01_steering_moves_to_module.md` — the steering toggle + panel logic moves from `app.js` into `header.js` (exported `refreshSteering()`); the chat per-bubble Tune form keeps working.
2. `02_sync_and_chat_moves_to_module.md` — the sync state machine moves from `sources.js` into `header.js` (`bor:sync-status` event); one module-owned New chat binding; the sign-in `?next=` rewrite.
3. `03_full_header_all_pages.md` — all five pages ship the identical header block; `#steering-panel` exists on every page; the viewer becomes standard row + titlebar row; login gains the full header.
4. `04_viewer_titlebar_styles.md` — the two-row viewer header styles, the sync button's failed state on non-Sources pages, theme/contrast/focus preserved.
5. `05_e2e_and_contract_update.md` — the story E2E suite `test_nav_consistency.py`; `test_header_consistency.py` + `test_shared_header.py` updated to the new viewer contract; regression pass; commit.
## Testing & Quality
- Unit/integration: frontend-only — no new `app/` logic; the no-CDN integration test (`tests/integration/test_api.py::test_index_html_served_locally`) must still pass (all new markup is same-origin, no new tags).
- Coverage: **>90%** on `app/` — unchanged by this phase (no Python change).
- E2E (mandatory, A16): `tests/e2e/test_nav_consistency.py` — the story gate, run in isolation; plus the two contract suites updated in task 05 and the regression list below.
## Completion Criteria
- [ ] The same visible header controls appear on **all five pages** in the same order — brand, nav [Chat, Sources, Tuning], Tuning toggle, Sync sources (admin), New chat, exactly one of Sign in / Sign out — verified in `test_nav_consistency.py` for both the admin and the anonymous role.
- [ ] The document viewer shows the standard bar (row 1) + back link and title (row 2); `#doc-back` target resolution (phase 13) unchanged.
- [ ] The login page carries the full header (nav incl. Tuning, Tuning toggle, Sync, New chat, auth pair).
- [ ] Chat page: the steering panel + per-bubble Tune + inline form behave exactly as before; Sources page: the sync button state machine + `#sync-result` line + `#sync-error-banner` behave exactly as before.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged (>90%).
- [ ] Regressions green in isolation: `test_header_consistency.py`, `test_shared_header.py`, `test_document_back_navigation.py`, `test_document_viewer.py`, `test_steering.py`, `test_global_tuning.py`, `test_sync_button.py`, `test_tuning_nav_link.py`, `test_smoke.py`, `test_chat_rag.py`, `test_admin_auth.py`.
- [ ] `uv run ruff check . && uv run pyright` clean (no Python change, but run the gate).
- [ ] UI Structure Check (AGENTS.md rule 5): landmarks / labels / contrast ≥4.5:1 / focus-visible preserved; no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit staging only this phase's files; `.agent/phases/todo/34_consistent_navbar/` moved to `.agent/phases/complete/`.
## Locked decisions
- **A10 UI revision preserved** — admin-only controls (Sources / Tuning nav links, Sync button) ship hidden and are revealed only for the signed-in admin; anonymous visitors get the reduced bar, identically on every page (owner confirmation 2026-08-26 — "hidden for anon, visible for admin").
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages.
- **Phase 19 module contract extended, not replaced** — `header.js` keeps the cached one-whoami-per-page promise; it gains ownership of the controls' behavior, not a second whoami.
- **Viewer bar superseded** — the phase-19 single-row viewer bar (PLAN.md §7.1 "the viewer bar = back + title + the same actions") is replaced by the two-row layout at the owner's request (this TODO). `PLAN.md` is not edited (Protocol B); this phase directory records the revision.
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
@@ -1,33 +0,0 @@
# Task 01 — Steering toggle + panel move into header.js
**Phase:** `34_consistent_navbar` · **Source:** `TODO.md:3 — "I want the navbar to be consistent between every page. I don't want buttons to pop in and out of existance. Just keep all those buttons active across all tabs."`
**Story:** `.agent/user_stories/nav-consistency.md`
## Objective
Make `frontend/assets/header.js` the owner of the steering toggle + panel behavior (today in `frontend/assets/app.js`), so the toggle can sit in every page's header (task 03) with zero page-script duplication. The chat page's behavior — panel open/close, list, count badge, per-note delete, per-bubble Tune form — must be byte-for-byte the same from the user's perspective.
## Work
1. `frontend/assets/header.js` — add the steering logic (runs at module import, like the existing sign-out binding):
- `loadSteering()` — `GET /api/steering`; non-2xx (the anonymous 403, unreachable API) → empty list (the current chat-page anonymous state); render via `renderSteeringPanel(notes)`.
- `renderSteeringPanel(notes)` — newest-first `<li class="steering-note">` rows: the note as `textContent` in a `span.steering-note-text` (XSS contract unchanged — never innerHTML for the note), a per-note Remove `button.steering-delete` with `aria-label="Delete tuning note: …"`; toggle `#steering-empty`'s `hidden` on `notes.length`; set the `#steering-count` badge text.
- `deleteSteeringNote(id, btn)` — disable the row button, `DELETE /api/steering/{id}`, re-load the list, announce through `#steering-announcer` (`role="status"`).
- The `#steering-toggle` click binding — open/close `#steering-panel`, flip `aria-expanded`, move focus into the panel on open (the chat-page a11y contract; read `app.js`'s current implementation first and mirror it exactly, including any close-on-Esc / outside-click behavior it has).
- **Export `refreshSteering()`** (fetch + render) — task 01's `app.js` change wires the per-bubble Tune form's success path to it.
- Update the file's header comment (it now owns the steering controls).
2. `frontend/assets/app.js` — remove the steering **panel** section (the `#steering-toggle` / `#steering-count` / `#steering-panel` / `#steering-list` / `#steering-empty` / `#steering-announcer` refs, `loadSteering`, `renderSteeringPanel`, `deleteSteeringNote`, `announceSteering`, the toggle binding) — **keep** the per-bubble `appendTuneButton` + `openTuneForm` (a chat-specific feature): the inline form's success path calls `refreshSteering()` imported from `./header.js` instead of the removed `loadSteering()`. Keep `TUNE_ICON` and the form's fetch/error handling untouched.
3. Update the comments that describe the panel as chat-page-owned (app.js header comment, index.html steering comments) — the panel now belongs to the shared module; index.html's markup stays for now (task 03 copies it to the other pages).
Notes:
- All elements are looked up null-safe (`querySelector` + guard) — a page that (still) lacks the panel markup is a no-op, mirroring how `initSharedHeader()` already works. This keeps the app functional between tasks.
- Do not change the steering API (`app/api/steering.py`), the panel markup in `index.html`, or the `#steering-panel` styles.
## Testing & Quality
- No Python change; the no-CDN integration test is unaffected.
- Coverage: `app/` gate unaffected (no Python change).
- The moved logic is behavior-verified by the regression suites in task 05 (`test_steering.py`, `test_global_tuning.py`); until then `uv run pytest` (unit + integration) must stay green.
## Completion Criteria
- [ ] `header.js` exports `refreshSteering()` and owns the toggle binding, panel render, per-note delete, count badge, and announcer.
- [ ] `app.js` no longer contains the panel logic; the per-bubble Tune button + inline form remain and call `refreshSteering()` on save.
- [ ] The chat page (`/`) still loads, opens, lists, and deletes steering notes exactly as before (manual smoke via the dev server or the regression suites in task 05).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,37 +0,0 @@
# Task 02 — Sync state machine + New chat + sign-in next move into header.js
**Phase:** `34_consistent_navbar` · **Source:** `TODO.md:3 — "I want the navbar to be consistent between every page. I don't want buttons to pop in and out of existance. Just keep all those buttons active across all tabs."`
**Story:** `.agent/user_stories/nav-consistency.md`
## Objective
Make `header.js` the owner of the Sync button state machine (today in `sources.js`), the single New chat binding (today duplicated across `app.js` / `sources.js` / `tuning.js` / `document.js`), and the sign-in `?next=` derivation — so the same markup on any page (task 03) behaves identically.
## Work
1. `frontend/assets/header.js` — add the sync state machine (read `sources.js`'s sync section first and mirror its contract exactly):
- **Boot (admin only):** `await fetchIsAdmin()` on the cached whoami — non-admins never poll (the status endpoint is admin-only). One `GET /api/sync/status`: `running` → enter the running state + start polling (the phase-32 reload-mid-sync re-attach); terminal → render the last result.
- **Click `#sync-btn`:** `POST /api/sync` → 202 enters running; 409 attaches to the running state (one sync at a time).
- **Poll** `GET /api/sync/status` every 2000 ms — one live timer, stopped on a terminal state. **No client-side hard timeout** (phase-32 locked decision — a sync can outlive the page; the state machine simply keeps polling).
- **Button states (§7.4 never-stale):** idle → label "Sync"; running → `disabled` + `aria-busy="true"` + spinner class (`.sync-icon.is-spinning`) + label "Syncing…"; success → label "Synced HH:MM"; failed → error state with the sanitized error string in the button's `title` + `aria-label` (on non-Sources pages that is where the failure is visible — the Sources page's own banner is driven by the event below).
- **On every state change** dispatch `window.dispatchEvent(new CustomEvent("bor:sync-status", { detail: <status> }))` where `detail` is the `GET /api/sync/status` object — step 2 points the Sources page's banner/result line at it.
2. `frontend/assets/header.js` — **one** New chat binding (module scope, null-safe): if `#messages` exists (chat page) → `window.dispatchEvent(new CustomEvent("bor:new-chat"))` and let the page script act; otherwise `clearChatStorage()` + `location.href = "/"` (the existing non-chat behavior — "new chat" means go to the chat, fresh).
3. `frontend/assets/header.js` — **sign-in `?next=` rewrite:** in `initSharedHeader()` (or the module-scope boot), set `#sign-in-link`'s `href` to `/login.html?next=<current pathname>` (default `/`) — the admin lands back on the page they signed in from.
- ASSUMPTION: on the chat page this changes the static fallback `?next=/sources.html` to `/` at runtime — landing on the page you signed in from ("return to where you were"). The page markup keeps its current href as the no-JS fallback.
4. `frontend/assets/sources.js` — remove the sync state machine (the `#sync-btn` click handler, the 2 s poll loop, the button-state helpers, the boot re-attach). **Keep** `#sync-result` + `#sync-error-banner` rendering, now driven by a `window.addEventListener("bor:sync-status", …)` subscription: `running` → clear the result line, hide the banner; `success` → render the last-result counts in `#sync-result` (reuse the existing formatting, "added" always shown); `failed` → show `#sync-error-banner` with the error text; `idle` → hide the banner, clear the result.
5. `frontend/assets/app.js` — replace the direct `#new-chat-btn` click binding with `window.addEventListener("bor:new-chat", startNewChat)` (the `startNewChat` function itself is unchanged).
6. `frontend/assets/sources.js`, `frontend/assets/tuning.js`, `frontend/assets/document.js` — remove their `#new-chat-btn` click bindings (the module owns them). Update the file-header comments (document.js: the module now owns New chat; sources.js: sync is module-owned, the banner is event-driven).
Notes:
- Null-safe element lookups throughout (a page that doesn't (yet) have `#sync-btn` is a no-op — the app stays functional between tasks).
- The module must keep exactly **one** whoami per page load (the cached promise) — the sync boot may await it but must not add a fetch.
- Do not touch `app/api/sync.py` — the API contract is unchanged.
## Testing & Quality
- No Python change; the no-CDN integration test is unaffected.
- Coverage: `app/` gate unaffected.
- Behavior parity is verified by the regression suites in task 05 (`test_sync_button.py`, `test_shared_header.py`, `test_chat_rag.py`); until then `uv run pytest` must stay green.
## Completion Criteria
- [ ] `header.js` owns: the sync state machine (+ `bor:sync-status` event), the single New chat binding (`bor:new-chat` on chat, clear+navigate elsewhere), and the sign-in `next` rewrite.
- [ ] `sources.js` no longer contains the sync state machine — `#sync-result` / `#sync-error-banner` render off the event; no `#new-chat-btn` binding remains in any page script.
- [ ] On the Sources page the full phase-32 cycle (click → polling → success counts / failure banner, reload re-attach) still works — confirmed in task 05 via `test_sync_button.py`.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,39 +0,0 @@
# Task 03 — The identical full header on all five pages
**Phase:** `34_consistent_navbar` · **Source:** `TODO.md:3 — "I want the navbar to be consistent between every page. I don't want buttons to pop in and out of existance. Just keep all those buttons active across all tabs."`
**Story:** `.agent/user_stories/nav-consistency.md`
## Objective
Ship the **identical header block** on all five pages — brand, nav [Chat, Sources, Tuning], Tuning toggle, Sync sources, New chat, Sign in / Sign out — and the `#steering-panel` section on every page; the document viewer keeps back + title in a second titlebar row; the login page finally carries the full header.
## Work
The canonical block is `index.html`'s current header **plus** the `#sync-btn` copied verbatim from `sources.html` (hidden by default, `#sync-label` + `.sync-icon` inside). Place the Sync button **after** the Tuning toggle and **before** the New chat button on every page.
1. `frontend/index.html` (chat) — add the `#sync-btn` block to the header (the only missing control); everything else already ships. `#steering-panel` stays where it is (after `#kb-banner` in `<main>`).
2. `frontend/sources.html` — add the Tuning toggle block (copied from `index.html`: `#steering-toggle` + `#steering-count`) after the nav; add the `#steering-panel` section (copied from `index.html`, incl. the `#steering-announcer` paragraph) as the **first child of `<main>`**; keep the existing `#sync-btn` where it is.
3. `frontend/tuning.html` — add the Tuning toggle block + the `#sync-btn` block to the header (same order as chat); add the `#steering-panel` section as the first child of `<main>`.
4. `frontend/document.html` — restructure the header:
- **Row 1** becomes the standard `.app-header` / `.header-inner` bar, byte-for-byte the same block as the other pages: brand, `<nav class="app-nav">` with `Chat` + `#nav-sources` (hidden) + `#nav-tuning` (hidden), Tuning toggle, `#sync-btn` (hidden), New chat, Sign in (`?next=/document.html` static fallback) / Sign out.
- **Row 2** — a new `.doc-titlebar` container inside the same `<header>`, carrying the existing `#doc-back` link + `#doc-title` + `#doc-meta` (moved out of the old `.doc-header-inner` title block, markup otherwise unchanged — `renderDocument` addresses them by id, so `document.js` needs no render change).
- ASSUMPTION: **no nav link gets `is-active` / `aria-current` on the viewer** — a document is a detail view reachable from chat or Sources (phase 13's `back` param), so no single nav target is "current". The back link carries the navigation affordance.
- The old `.doc-header-actions` wrapper is dropped — its buttons now live in row 1's standard `.header-inner`.
5. `frontend/login.html` — full header: the nav gains the `#nav-tuning` link (after `#nav-sources`, same hidden-by-default markup as the other pages); add the Tuning toggle + `#sync-btn` + New chat + the Sign in / Sign out pair (sign-in static fallback `?next=/login.html`); add the `#steering-panel` section as the first child of `<main>`.
6. `frontend/assets/header.js` — comment updates only: the viewer now has a nav (its "the viewer has no nav" notes are stale); the module's "missing element is a no-op" contract still holds for any page missing an element. No behavior change — the reveal code already handles `#nav-sources` / `#nav-tuning` / `#sync-btn` / the auth pair wherever they exist.
7. Update the stale HTML comments in the touched headers (phase-19/29 comments describing the old page-specific layouts) to reference this phase + the owner confirmation (2026-08-26).
Rules:
- Keep every existing id / class / aria attribute exactly as it exists today (the E2E suites key off them); only ADD missing blocks and move the viewer's title elements.
- Ship-hidden stays ship-hidden: `#nav-sources`, `#nav-tuning`, `#sync-btn`, and exactly one of the auth pair are `hidden` in the markup on every page — `header.js` reveals at load (one whoami, cached).
- Preserve indentation/markup style so the five headers stay diff-identical (that identity is what task 05's E2E asserts).
## Testing & Quality
- No Python change; the no-CDN integration test must still pass (same-origin markup only).
- Coverage: `app/` gate unaffected.
- Manual smoke before task 05: with the dev server, as admin and as anonymous, each of the five pages shows the full bar (admin) / reduced bar (anonymous) with no console errors.
## Completion Criteria
- [ ] All five page headers contain the same control inventory in the same order: brand, nav [Chat, Sources, Tuning], `#steering-toggle`, `#sync-btn`, `#new-chat-btn`, `#sign-in-link` + `#sign-out-btn`.
- [ ] `#steering-panel` (+ `#steering-announcer`) exists on all five pages (chat: after `#kb-banner`; others: first child of `<main>`).
- [ ] `document.html` = standard row + `.doc-titlebar` row with `#doc-back` / `#doc-title` / `#doc-meta`; no nav link carries `is-active` there.
- [ ] `login.html` carries the full header incl. the `#nav-tuning` link and the auth pair.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,29 +0,0 @@
# Task 04 — Two-row viewer header styles + sync failed state
**Phase:** `34_consistent_navbar` · **Source:** `TODO.md:3 — "I want the navbar to be consistent between every page. I don't want buttons to pop in and out of existance. Just keep all those buttons active across all tabs."`
**Story:** `.agent/user_stories/nav-consistency.md`
## Objective
Style the document viewer's new two-row header (standard row + titlebar row) so row 1 is visually indistinguishable from the other pages' bars, and give the Sync button a visible failed state on pages that have no error banner (every page except Sources).
## Work
1. `frontend/assets/styles.css` — **viewer header:**
- The viewer's `<header>` keeps the sticky app-frame behavior; **row 1** reuses the existing `.app-header` / `.header-inner` rules verbatim (64px desktop, 58px at ≤640px — the phase-12/19 pinned heights apply because row 1 *is* the standard bar).
- New `.doc-titlebar` rules for row 2: `.container`-width inner row with `#doc-back` + the title block; its own height (title line + meta line), `border-top` separator in the existing hairline color, same surface color (`#121a2e`/`#0a0e17` family per the phase-08 palette); `#doc-title` truncates with an ellipsis + `title` attribute instead of the old pill-clipping rule.
- The old `.doc-header` / `.doc-header-inner` / `.doc-header-actions` / title-clipping rules are removed or reduced to the two-row structure (keep class names that task 05's updated contract suites reference — check which selectors the suites use before deleting: `.doc-header` may remain as the header element's class wrapping both rows).
- The `.steering-panel` positioning rules must work from the new placement on the non-chat pages (first child of `<main>`) — the panel is an in-flow section, so this is expected to be a no-op; verify visually and in the E2E.
2. `frontend/assets/styles.css` — **sync failed state (non-Sources pages):** the failed `#sync-btn` gets an error treatment from the phase-08 palette (error ink `#fca5a5` on the error surface `#2d1318`, border `#f59e0b`-free — the error chip uses `#fca5a5`/`#2d1318`, ≈9.1:1) so a failed sync is visible on every page, complementing the `title`/`aria-label` error text set by `header.js`. Reuse the existing `.sync-btn` state classes/styles if phase 32 already defines a failed look; otherwise add it.
3. `frontend/assets/styles.css` — **login page:** the full header needs no new rules (it reuses `.app-header`), but confirm the login card layout still centers correctly with the full bar (no header-height regression at ≤640px).
4. Accessibility checks (AGENTS.md rule 5): `:focus-visible` 3px outline on the new titlebar back link (it already has the existing `.doc-back` styles — preserve); contrast ≥4.5:1 for title/meta text; `prefers-reduced-motion` still stills the sync spinner.
## Testing & Quality
- CSS-only — no Python change; the no-CDN integration test is unaffected.
- Coverage: `app/` gate unaffected.
- Visual pass (dev server, admin + anonymous, desktop + 640px): all five pages, viewer row 1 identical to chat's bar; viewer row 2 shows back + title + meta; the sync button's failed state is visible and readable.
## Completion Criteria
- [ ] The viewer's row 1 renders pixel-consistent with the chat/sources bars (same height 64px / 58px, same paddings, same controls).
- [ ] `.doc-titlebar` renders back + title (ellipsis) + meta badges on one or two tidy lines; sticky with the header.
- [ ] A failed sync shows an error-colored `#sync-btn` on non-Sources pages with the sanitized error in `title` / `aria-label`.
- [ ] No console layout breakage on the login page (card still centered, 58px bar at ≤640px).
- [ ] `uv run pytest` green (no-CDN test included); `uv run ruff check . && uv run pyright` clean.
@@ -1,33 +0,0 @@
# Task 05 — Story E2E + contract-suite updates + regression pass
**Phase:** `34_consistent_navbar` · **Source:** `TODO.md:3 — "I want the navbar to be consistent between every page. I don't want buttons to pop in and out of existance. Just keep all those buttons active across all tabs."`
**Story:** `.agent/user_stories/nav-consistency.md`
## Objective
Prove the contract with the story's dedicated Playwright suite — identical visible header control inventory on all five pages for each role — update the two pre-existing contract suites that encoded the old viewer bar, and run the full regression list.
## Work
1. `tests/e2e/test_nav_consistency.py` (NEW — the story gate, run in isolation). Fixtures: the standard E2E app + DB (see `tests/e2e/conftest.py`); a fixture document for the viewer URL (the `source=docs&path=homelab%2Fkubernetes.md` pattern from `test_shared_header.py`); admin session via `tests/e2e/auth_helpers.py`.
- **Admin inventory (all five pages):** `/`, `/sources.html`, `/document.html?source=…&path=…`, `/tuning.html`, `/login.html` — each header contains, visible: the three nav links (Chat, `#nav-sources`, `#nav-tuning`), `#steering-toggle`, `#sync-btn`, `#new-chat-btn`, and `#sign-out-btn` visible with `#sign-in-link` hidden. Assert the same **id + class inventory and DOM order** of the header controls on every page (normalize: the current-page `is-active` nav marker and the sign-in `?next=` value legitimately differ per page).
- **Anonymous inventory (all five pages):** nav present with Chat visible and `#nav-sources` / `#nav-tuning` hidden (locked A10 UI revision); `#sync-btn` hidden; `#sign-in-link` visible, `#sign-out-btn` hidden; `#steering-toggle` visible.
- **Viewer specifics:** row 1 height equals the chat page's header height (64px desktop / 58px ≤640px); the titlebar row is visible with `#doc-back` + `#doc-title` (rendered document title) + `#doc-meta` badges; clicking `#doc-back` honors the `back` param (phase 13 — one positive + one rejection case).
- **Steering works off-chat:** as admin, on `/tuning.html` — seed zero notes (truncate `steering_notes` via a `SessionLocal` like the other suites), click `#steering-toggle` → `#steering-panel` visible + `aria-expanded="true"` + empty state shown; add a note through the panel? (the panel has no add form — it lists notes; assert the toggle open/close cycle + the count badge reads 0) — keep this deterministic, no chat needed.
- **Sync present, not triggered:** as admin on `/tuning.html` assert `#sync-btn` is visible (do NOT click it — a real sync would clone real repos; the full state machine is `test_sync_button.py`'s job).
2. `tests/e2e/test_header_consistency.py` (UPDATE to the new contract): the viewer assertions change — `.doc-header` is now the two-row header; assert **row 1** (the standard bar) is 64px desktop / 58px ≤640px and identical to the chat/sources bars (the existing `_box_height(page, ".doc-header")` measurement must be pointed at the standard row — use the row-1 selector, e.g. `.app-header .header-inner` inside the viewer header), and assert the titlebar row is present (height > 0). The chat/sources assertions are unchanged.
3. `tests/e2e/test_shared_header.py` (UPDATE to the new contract): the "the viewer has no nav — no Sources link in the DOM" assertions flip — the viewer now carries the same nav contract (`.app-nav` with Chat + hidden `#nav-sources` + hidden `#nav-tuning`, revealed for admin). The auth-pair + New chat assertions for the viewer stay (they move from `.doc-header-actions` to the standard bar — update the selectors).
4. **Regression pass — each in isolation** (`uv run pytest tests/e2e/<file>.py -v --no-cov`): `test_header_consistency.py`, `test_shared_header.py`, `test_document_back_navigation.py`, `test_document_viewer.py`, `test_steering.py`, `test_global_tuning.py`, `test_sync_button.py`, `test_tuning_nav_link.py`, `test_smoke.py`, `test_chat_rag.py`, `test_admin_auth.py`. Fix fallout in the suites above where the old contract is encoded; fix app code where behavior genuinely changed.
5. Full gate: `uv run pytest` (unit + integration), `uv run pytest --cov=app --cov-report=term-missing` (TOTAL unchanged, >90%), `uv run ruff check . && uv run pyright`.
6. **UI Structure Check** (AGENTS.md rule 5) on the five headers + the new titlebar: landmarks (`<header>`, `<nav aria-label>`, `<main>`), labels, contrast ≥4.5:1, focus-visible, no CDN (rule 6 — the no-CDN integration test covers it).
7. **Commit** (A17): stage only this phase's files (`frontend/**`, `tests/e2e/**`), message `feat(ui): one consistent navbar on every page (TODO.md L3)`, always `--no-gpg-sign`. Move `.agent/phases/todo/34_consistent_navbar/` to `.agent/phases/complete/`.
## Testing & Quality
- E2E: `tests/e2e/test_nav_consistency.py` green **in isolation** — the story gate (A16: one story, one file).
- Unit/integration: no new `app/` logic — the existing suite (incl. the no-CDN test) stays green.
- Coverage: **>90%** on `app/` — unchanged (no Python change).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_nav_consistency.py -v --no-cov` green in isolation.
- [ ] `test_header_consistency.py` + `test_shared_header.py` updated and green; every suite in the task 05 regression list green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged (>90%).
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
@@ -1,47 +0,0 @@
# Phase 35 — Admin Page to Add / Remove Git Sources
**Source:** `TODO.md` L4 — "I need a page only the admin can access where I can add and remove git sources for docs"
**Story:** `.agent/user_stories/git-sources-admin.md`
**Context:** Phase 28 introduced git-based sources (`BOR_GIT_SOURCES` env var + `scripts/git_sync.clone_or_pull`) and phase 32 the one-click Sync button (`POST /api/sync`) — but the *list itself* can only be changed by editing `.env` and restarting. This phase makes the list admin-managed: a Postgres-backed table, an admin-only CRUD API, and a dedicated admin page, with the sync pipeline and `import_docs` resolving the effective list from the DB (env var demoted to an empty-table fallback).
## Objective
Deliver a page **only the admin can access** (`/git-sources.html`, soft-gated like Sources) to **add and remove git sources**, stored in a new `git_sources` table; the Sync button (phase 32) and `import_docs` (phase 28) use the stored list, `BOR_GIT_SOURCES` remains a fallback while the table is empty, and phase 32's fail-loud "no git sources configured" is preserved when both are empty.
## Dependencies
- `28_git_based_sources` (complete) — `scripts/git_sync.clone_or_pull`, `repo_name`, the `BOR_GIT_SOURCES` settings + `git_source_list`, the import resolution order (`--source` wins).
- `32_admin_sync_button` (complete) — the `POST /api/sync` / `GET /api/sync/status` pipeline this phase re-points at the DB list; the Sync button the page's hint refers to.
- `16_admin_auth` (complete) — `require_admin` (the router-level pattern from `app/api/sync.py`), the soft-gate page pattern (`sources.html`), the `fetchIsAdmin()` frontend gate.
- `34_consistent_navbar` (todo) — the identical five-page header this phase's admin-only "Git sources" nav link plugs into (phase 29 pattern).
- `29_tuning_nav_link` (complete) — the admin-only ship-hidden nav-link pattern to copy.
## Tasks
1. `01_model_and_migration.md` — `GitSource` model + migration `0006_git_sources.py` (reversible).
2. `02_git_sources_api.md` — admin-only `GET/POST /api/git-sources` + `DELETE /api/git-sources/{id}` with validation, the env-fallback listing, and the integration suite.
3. `03_sync_and_importer_use_db.md` — `effective_git_sources()` shared by `app/api/sync.py` and `scripts/import_docs.py` (DB wins, env fallback, fail-loud unchanged) + test updates.
4. `04_admin_page.md` — `/git-sources.html` + `git-sources.js` (soft-gated, list / add / remove, env note, sync hint) + styles.
5. `05_nav_link.md` — the admin-only "Git sources" nav link on all five pages + the `header.js` reveal.
6. `06_e2e_and_docs.md` — the story E2E suite `test_git_sources_admin.py`, `test_nav_consistency.py` nav-inventory update, README/`.env.example` notes, regressions, commit.
## Testing & Quality
- Unit/integration: `tests/unit/` for `effective_git_sources` (DB-wins / env-fallback / both-empty); `tests/integration/test_git_sources_api.py` for the CRUD contract (403/201/409/422/404, env fallback); the migration up/down test following the 0004/0005 pattern; the existing `test_sync_api.py` + `test_import_docs_git.py` suites stay green with the resolution indirection.
- Coverage: **>90%** on `app/` for the new module + API.
- E2E (mandatory, A16): `tests/e2e/test_git_sources_admin.py` — the story gate, run in isolation.
## Completion Criteria
- [ ] Migration 0006 applied (`uv run alembic upgrade head`); `git_sources` table exists with `url` unique.
- [ ] `GET /api/git-sources` (admin) lists DB rows; while the table is empty it returns the env list with `from_env: true`; anonymous gets 403 on all three routes.
- [ ] `POST` creates (201, trimmed, shape-validated, 409 duplicate without echoing the URL); `DELETE` removes (204/404).
- [ ] `POST /api/sync` and `import_docs` resolve the list via `effective_git_sources` (origin logged `db|env`); both-empty still fails loudly; `--source` override unchanged.
- [ ] `/git-sources.html`: anonymous sees the sign-in gate; the admin sees list + add + remove with a never-stale button and inline errors; the admin-only "Git sources" nav link is visible on all five pages for the admin and hidden for anonymous.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov` green in isolation; regressions (task 06 list) green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
## Locked decisions
- **A3 / A13 honoured** — the list lives in Postgres via Alembic (no JSON file, no new store).
- **A10 extended per the phase-16 pattern** — a new admin-only router behind `require_admin`; the public API surface stays stateless; no new auth mechanism.
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages.
- **Env var demoted, not removed** — `BOR_GIT_SOURCES` keeps working exactly as today while the table is empty (the fallback); once the table has rows it is ignored (the UI is the source of truth). Phase 32's fail-loud empty-config behavior is preserved.
- **Scope boundary** — adding/removing a repo does NOT immediately clone, import, or prune: the existing Sync button performs that (removal prunes on the next sync, `prune=True`). The page's hint says so.
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
@@ -1,46 +0,0 @@
# Task 01 — GitSource model + migration 0006
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
Add the `git_sources` table (one row per admin-managed repo URL) via the model + a reversible Alembic migration, following the exact conventions of migrations 0003–0005.
## Work
1. `app/models.py` — add the model (next to `SteeringNote`, docstring citing this phase + the A13 convention):
```python
class GitSource(Base):
"""One admin-managed git source (phase 35).
The UI-maintained list of repo URLs the Sync button (phase 32) and
import_docs (phase 28) clone/pull. DB rows win over the
BOR_GIT_SOURCES env var, which is a fallback while this table is
empty (see app.rag.git_sources.effective_git_sources).
"""
__tablename__ = "git_sources"
id: Mapped[uuid.UUID] = mapped_column(
UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
)
url: Mapped[str] = mapped_column(Text, unique=True, nullable=False)
added_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now()
)
```
(Import `Text` — already imported in the file; verify.)
2. `alembic/versions/0006_git_sources.py` — new migration:
- Read `alembic/versions/0005_kb_overview.py` first and chain from its actual `revision` id (the filenames are not the revision ids).
- `upgrade()`: `CREATE TABLE git_sources (id UUID PRIMARY KEY, url TEXT NOT NULL, added_at TIMESTAMP WITH TIME ZONE DEFAULT now() NOT NULL)` + `CREATE UNIQUE INDEX uq_git_sources_url ON git_sources (url)` (use `sa.Uuid` / the same column types the other migrations use — mirror their style, including `op.create_table` kwargs and the `UniqueConstraint`-vs-index choice 0003/0004 made).
- `downgrade()`: drop the index + table.
3. Apply it to the dev database: `podman compose up -d db` (if needed) then `uv run alembic upgrade head`.
4. Migration test — follow the existing pattern (see how 0004/0005 are integration-tested — `tests/integration/` migration suite): assert 0006 up creates the table + unique constraint and down drops it (round-trip on the test DB).
## Testing & Quality
- Unit/integration: the migration up/down test above; `uv run pytest` green overall.
- Coverage: model-only for now — the `app/` gate stays >90% (models are thin).
## Completion Criteria
- [ ] `alembic/versions/0006_git_sources.py` exists, chains off 0005's real revision id, and is reversible.
- [ ] `uv run alembic upgrade head` applies cleanly on the dev DB; `git_sources` visible (`\d git_sources` equivalent).
- [ ] The 0006 up/down integration test passes; full `uv run pytest` green.
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,39 +0,0 @@
# Task 02 — Admin-only git sources CRUD API
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
The admin CRUD contract for the stored list: `GET /api/git-sources` (DB rows, or the env fallback while the table is empty), `POST /api/git-sources` (validated create), `DELETE /api/git-sources/{id}` — all behind `require_admin`, exactly like `app/api/sync.py`.
## Work
1. `app/schemas.py` — add (mirroring the steering schemas' style):
- `GitSourceIn` — `url: str = Field(min_length=1, max_length=500)` + a `mode="before"` trim validator (whitespace-only → 422, same trick as `SteeringNoteIn`).
- `GitSourceOut` — `id: uuid.UUID | None`, `url: str`, `added_at: datetime | None` (both nullable: env-fallback rows carry neither).
- `GitSourceList` — `sources: list[GitSourceOut]`, `from_env: bool` (`True` only when the table is empty and the list comes from `BOR_GIT_SOURCES`).
2. `app/api/git_sources.py` (NEW) — `router = APIRouter(prefix="/git-sources", tags=["git-sources"], dependencies=[Depends(require_admin)])` (copy the sync.py pattern + docstring style):
- `GET ""` → `GitSourceList`: DB rows ordered by `(added_at, id)`; if the table is empty → the `get_settings().git_source_list` env URLs as rows with `id=None, added_at=None` and `from_env=True`; `from_env=False` whenever DB rows exist (the env var is then ignored — the phase's locked decision).
- `POST ""` (201) → create:
- Shape validation (module-level `URL_RE = re.compile(r"^(https?://|ssh://|git@)")` with a docstring): the trimmed URL must match — covers the phase-28 real URLs (HTTPS + `git@` SSH); scp-style `host:repo` is deliberately rejected (422).
- ASSUMPTION: the accepted shapes are exactly `http://`, `https://`, `ssh://`, `git@…`; the 422 detail is generic ("not a valid git URL (expected https://, ssh:// or git@…)") and never echoes the input.
- Duplicate (same trimmed URL already stored) → 409 with a generic detail ("a git source with this URL already exists") — **never echo the URL** (URLs may embed `user:pass@` credentials; phase 32's masking discipline).
- Success → insert, commit, return the created `GitSourceOut`.
- `DELETE "/{source_id}"` → 204; unknown id → 404 `git source not found`.
3. `app/main.py` — register the router alongside the existing `include_router` calls (check how `sync` is included and mirror it).
4. `tests/integration/test_git_sources_api.py` (NEW) — follow `tests/integration/test_sync_api.py`'s auth pattern (`_login` via `POST /api/login` with the fixture admin password, admin client as context manager):
- Anonymous → 403 `{"detail": "admin only"}` on GET, POST, and DELETE.
- Admin + empty table + env set (monkeypatch the settings `git_sources`) → GET returns the env rows, `from_env=True`, null ids.
- Admin + empty table + env empty → GET returns `sources=[]`, `from_env=True`.
- POST: valid `https://…` → 201 + the row appears in GET with `from_env` now `False`; a duplicate → 409 and the detail contains no URL; an invalid shape (`not a url`, `host:repo`) → 422; whitespace-only / >500 chars → 422; a `git@github.com:…` URL → 201 (accepted).
- DB rows win over env: seed a row AND set the env → GET returns only the DB rows, `from_env=False`.
- DELETE: known id → 204 + gone from GET (back to env fallback if the table is now empty); unknown id → 404.
## Testing & Quality
- Integration: the suite above — the full CRUD contract, auth split, fallback semantics.
- Coverage: **>90%** on the new module (`app/api/git_sources.py` + schemas).
## Completion Criteria
- [ ] All three routes exist under `/api/git-sources`, admin-only (403 anonymous), registered in `app/main.py`.
- [ ] `tests/integration/test_git_sources_api.py` green; `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
- [ ] No credential-echo path: 409/422 details never contain the submitted URL (a test asserts this).
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -1,38 +0,0 @@
# Task 03 — Sync + import_docs resolve the effective list (DB wins, env fallback)
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
One shared resolver — DB rows win, `BOR_GIT_SOURCES` is a fallback only while the table is empty, fail-loud unchanged when both are empty — used by **both** the in-app sync pipeline (`app/api/sync.py::_run_sync`) and the CLI (`scripts/import_docs.py`), so the admin page's list is what actually gets cloned and indexed.
## Work
1. `app/rag/git_sources.py` (NEW) — the single resolver (importable by both the app and the CLI — `scripts` already imports `app.rag.*`):
```python
def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]:
"""(urls, origin) — DB rows in (added_at, id) order win; while the
table is empty the BOR_GIT_SOURCES env list is the fallback; both
empty → ([], "env")."""
```
- DB rows: `select(GitSource).order_by(GitSource.added_at, GitSource.id)`.
- Fallback: `get_settings().git_source_list` (the phase-28 CSV parse — reuse, don't re-implement).
2. `app/api/sync.py` — `_run_sync` replaces `settings.git_source_list` with the resolver:
- Open a `SessionLocal()` (close in `finally`) around the resolution — the background task has no request session.
- Log the origin: the existing `sync: started repos=N` line gains `origin=db|env`.
- Both-empty: keep the fail-loud `GitSyncError("no git sources configured …")` (extend the message to mention both origins — e.g. `(git_sources table empty and BOR_GIT_SOURCES unset)`; if `tests/integration/test_sync_api.py::test_no_git_sources_configured_fails_loudly` asserts the old text, update that expectation — it is this phase's file to update).
3. `scripts/import_docs.py` — the git-URL resolution branch (today `settings.git_source_list`) resolves via the same function: open a short `SessionLocal()` at resolution time (the import needs the DB anyway — no DB-down fallback to design). The `--source` override still wins (manual mode), the log line records the origin (`git sources: N repo(s) origin=db|env` before the clone loop).
4. **Test updates:**
- `tests/unit/` — NEW unit tests for `effective_git_sources`: DB rows win (seeded table + env set → DB list, origin `db`); env fallback (empty table + env set); both empty → `([], "env")`. (Use a test DB session or a stubbed session following the existing unit-test conventions.)
- `tests/integration/test_sync_api.py` — where it stubs `settings.git_source_list` to drive sync scenarios, keep those scenarios working through the new indirection: either seed the `git_sources` table or monkeypatch `effective_git_sources` (whichever the file's existing fixture style favors); add one scenario asserting a DB row is used over the env when both are set (clone/import mocked — the file already mocks them).
- `tests/integration/test_import_docs_git.py` — same treatment for the CLI path (`--source` override scenario untouched); add the DB-over-env scenario at the CLI level.
## Testing & Quality
- Unit: `effective_git_sources` — all three branches.
- Integration: sync + import_docs suites green with the resolver in the path, incl. the new DB-over-env scenarios and the updated fail-loud expectation.
- Coverage: **>90%** on the new module + the modified call sites.
## Completion Criteria
- [ ] `effective_git_sources` exists in `app/rag/git_sources.py` and is the ONLY place (besides the API's GET fallback, which may call it too) that combines DB + env.
- [ ] `_run_sync` and `import_docs` both resolve through it; origin visible in their logs.
- [ ] Both-empty still raises the fail-loud error (sync) / the CLI's existing no-sources behavior (import_docs) — assertions kept/updated.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
@@ -1,41 +0,0 @@
# Task 04 — The /git-sources.html admin page
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
The page itself: a soft-gated (anonymous → sign-in gate, exactly like `sources.html`) full-width manager with an add form, a sources list with per-row Remove, an env-fallback note, and a hint pointing at the Sync button. Never-stale buttons, inline errors, WCAG 2.1 AA basics, dark tech theme.
## Work
1. `frontend/git-sources.html` (NEW) — modeled on `frontend/sources.html` (same frame, same gate pattern, the standard full header from phase 34):
- Head: same meta/favicon/stylesheet pattern; `<title>Git sources · Brain of Reese</title>`; skip-link; the identical header block (nav [Chat, Sources, Tuning] + Tuning toggle + `#sync-btn` + New chat + auth pair — the "Git sources" nav link itself arrives in task 05, so this file lands without it for now).
- `<main id="main">` — `#steering-panel` first (phase 34 contract), then:
- **Gate** `#git-sources-gate` — the `#sources-gate` soft-gate markup pattern (sign-in card + link `/login.html?next=/git-sources.html`), visible for anonymous, hidden for admin.
- **Content** `#git-sources-content` (hidden until admin):
- `.page-head` — `<h1>Git sources</h1>` + sub: "The repositories the Sync button clones and indexes. Add or remove them here — no `.env`, no restart."
- **Env note** `#git-sources-env-note` (hidden by default; shown when the API returns `from_env: true`): "These sources currently come from `BOR_GIT_SOURCES` in `.env` — adding or removing one here switches management to the database."
- **Add form** `#git-source-form` — visible label (or visually-hidden label per the tuning-page pattern — use a visible `<label for="git-source-url">Add a git source</label>`), input `#git-source-url` (type text, `maxlength="500"`, `autocomplete="off"`, placeholder `https://github.com/you/homelab.git`, `required`), submit button `#git-source-add` ("Add source"), error line `#git-source-error` (`role="alert"`, hidden) — §7.4 never-stale: the button disables + label changes while the POST is in flight, re-enables on success/failure (the form is kept on failure, same as the tuning forms).
- **List** `#git-sources-list` (a full-width table or list per §7.1 — **no skinny single-column list**: use the Sources-page table pattern — columns: URL (mono `<code>`), Added, actions) + empty state `#git-sources-empty` ("No git sources stored yet." — and, with `from_env`, the env note already explains where the active list comes from).
- **Hint box** (`role="note"`): "Use the **Sync sources** button in the header (or on the Sources page) to clone the repos and refresh the index — removing a repository prunes its documents from the index on the next sync."
2. `frontend/assets/git-sources.js` (NEW) — the page module (loaded `type="module"`, imports `./header.js` like its siblings):
- Boot: `const admin = await initSharedHeader()` (one cached whoami) — anonymous → show the gate, stop; admin → hide the gate, `loadSources()`.
- `loadSources()` — `GET /api/git-sources` → render the table rows (`textContent` only — URLs may contain credentials; never innerHTML the URL), the added date (localized, `—` for null), the per-row Remove button (`.git-source-remove`, `aria-label="Remove git source: <url>"`), the env note's `hidden` on `from_env`, the empty state. Non-2xx → the content area shows a `role="alert"` error state with a retry (never a stuck page).
- Add submit — client-side non-empty check; disable `#git-source-add` (label "Adding…"); `POST /api/git-sources` with `{url}`; success → clear the input, re-enable (label "Add source"), `loadSources()`, focus the new row (a11y); failure → `#git-source-error` with the server detail (422 shape-aware like the tuning forms), re-enable, input kept.
- Remove click — `window.confirm("Remove this git source from the list? Its documents stay indexed until the next sync prunes them.")` — cancel → nothing; ok → disable the row button, `DELETE /api/git-sources/{id}`, `loadSources()`; failure → row error state + re-enable.
- Focus management + keyboard: all controls focus-visible (theme CSS covers it), the list rows' buttons are real `<button>`s.
3. `frontend/assets/styles.css` — the `.git-source-*` rules + gate reuse:
- The table: full-width in the 72rem container (the Sources-page table styles are a good starting point — reuse classes where they fit), mono URL cells with horizontal scroll on overflow (long URLs with credentials), rows ≥44px touch targets, `:focus-visible` 3px outline.
- Env note: an info chip in the theme palette (brand-soft `#232b52` surface, brand-ink `#a5b4fc` text ≈6.9:1); hint box: the page-sub styling family; error/alert states reuse the existing `#fca5a5`/`#2d1318` error treatment.
- Gate: reuse the `#sources-gate` styles (the page is the same shape as Sources — one gate visual language).
4. `frontend/assets/header.js` — no change (the page's controls are the standard shared ones; `initSharedHeader` already handles everything that ships in the header).
## Testing & Quality
- Frontend-only — no Python change this task; the no-CDN integration test must still pass (same-origin markup; the E2E in task 06 exercises the page).
- Coverage: `app/` gate unaffected.
- Manual smoke (dev server, signed in): add a real-looking URL → row appears; remove → confirm → row gone; invalid URL → inline 422 error, button re-enabled; signed out → gate only.
## Completion Criteria
- [ ] `/git-sources.html` served at the route; anonymous sees only the gate (list/form absent or inert); admin sees list + form + env note + hint.
- [ ] Add / remove round-trip works against the task-02 API; every in-flight state disables its control and re-enables on resolution (never stale); errors are inline `role="alert"`.
- [ ] UI Structure Check (AGENTS.md rule 5): landmarks, labeled controls, contrast ≥4.5:1, focus-visible; full-width table (no skinny list); no CDN (rule 6).
- [ ] `uv run pytest` green (no-CDN test); `uv run ruff check . && uv run pyright` clean.
@@ -1,32 +0,0 @@
# Task 05 — Admin-only "Git sources" nav link on all five pages
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
Make the new page reachable from everywhere: the admin-only **"Git sources"** nav link in all five identical headers (phase 34's contract), revealed for the admin by `header.js` — the exact phase-29 pattern.
## Work
1. The five page headers — `frontend/index.html`, `frontend/sources.html`, `frontend/document.html`, `frontend/tuning.html`, `frontend/git-sources.html` — inside `<nav class="app-nav" aria-label="Primary">`, **immediately after** the `#nav-sources` link, add (mirroring the `#nav-tuning` markup, comment citing this phase + owner permission 2026-08-26):
```html
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Git sources</a>
```
- Rules (phase-29 contract): `hidden` by default on every page (anonymous-safe ship-hidden); NO `is-active` / `aria-current` on the four pages that aren't the Git sources page.
- **Exception:** on `frontend/git-sources.html` the link carries `class="nav-link is-active"` + `aria-current="page"` (the current page, like Tuning on `tuning.html`).
- Nav order on every page becomes: Chat, Sources, **Git sources**, Tuning.
2. `frontend/assets/header.js` — next to the `navTuning` reveal block, add the same ship-hidden/reveal-for-admin contract:
```js
const navGitSources = document.querySelector("#nav-git-sources");
if (navGitSources) navGitSources.hidden = !admin;
```
Update the file-header comment (the admin-only link list now includes Git sources).
## Testing & Quality
- Frontend-only — no Python change; the no-CDN integration test is unaffected (same-origin `<a>`).
- Coverage: `app/` gate unaffected.
- Manual smoke: admin sees "Git sources" on all five pages → each navigates to `/git-sources.html` (with `is-active` there); anonymous never sees it (ships hidden, no flash).
## Completion Criteria
- [ ] All five headers contain `#nav-git-sources` after `#nav-sources`, `hidden` by default; `git-sources.html`'s carries `is-active` + `aria-current="page"`.
- [ ] `header.js` reveals it for the admin on the cached whoami (no extra request) and hides it for anonymous on every page.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -1,37 +0,0 @@
# Task 06 — Story E2E + nav-inventory update + docs + commit
**Phase:** `35_git_sources_admin` · **Source:** `TODO.md:4 — "I need a page only the admin can access where I can add and remove git sources for docs"`
**Story:** `.agent/user_stories/git-sources-admin.md`
## Objective
Prove the story end-to-end with its dedicated Playwright suite, keep phase 34's nav-consistency contract in sync with the new link, document the env-var demotion, and close the phase with the full gate + one commit.
## Work
1. `tests/e2e/test_git_sources_admin.py` (NEW — the story gate, run in isolation). Fixtures: the standard E2E app + DB (`tests/e2e/conftest.py`); admin session via `tests/e2e/auth_helpers.py`; **no git, no network** — this suite is UI + API only (the clone/import path is mocked at the integration level, task 03).
- **Anonymous:** `/git-sources.html` shows the sign-in gate (list + add form absent/inert); `#nav-git-sources` hidden on the five pages; the API 403s (assert via the page context's `fetch` or `context.request` — follow `test_admin_auth.py`'s pattern for anonymous API assertions).
- **Admin — nav:** `#nav-git-sources` visible on all five pages; clicking it from `/` lands on `/git-sources.html` with the link `is-active`.
- **Admin — list:** seed two rows via the API (or `SessionLocal`) before load → both rows render (mono URL text, added date); the env note is hidden (DB rows exist).
- **Admin — add:** submit `https://example.com/reese/new-repo.git` → the row appears, the input clears, the button re-enables (never stale); submit a duplicate → inline `role="alert"` error, no new row, button re-enabled; submit `not a valid url` → inline 422 error, button re-enabled.
- **Admin — remove:** click a row's Remove → accept the confirm dialog (Playwright `page.on("dialog")`) → the row disappears; cancel a second removal → the row stays.
- **Admin — env fallback:** truncate `git_sources`, set the E2E app's `BOR_GIT_SOURCES` (follow how `test_sync_button.py` controls the env on the app fixture), reload → the env rows render + `#git-sources-env-note` visible.
- The sync-origin behavior (DB over env in the pipeline) is integration-level (task 03) — do not trigger a real sync in this suite.
2. `tests/e2e/test_nav_consistency.py` (phase 34 — UPDATE): the admin nav inventory now includes **"Git sources"** (four links, order Chat, Sources, Git sources, Tuning); the anonymous hidden set gains `#nav-git-sources`; the per-page inventory comparison stays order-sensitive.
3. **Docs:**
- `.env.example` — the `BOR_GIT_SOURCES` comment: now the **empty-table fallback**; the primary management UI is the admin Git sources page (phase 35).
- `README.md` — the import/update workflow section: the git-sources list is managed on the admin page (stored in Postgres); `BOR_GIT_SOURCES` only applies while that list is empty; `--source` still overrides for manual runs.
4. **Regression pass — each in isolation** (`uv run pytest tests/e2e/<file>.py -v --no-cov`): `test_git_sources_admin.py` (new), `test_nav_consistency.py` (updated), `test_sync_button.py`, `test_shared_header.py`, `test_header_consistency.py`, `test_tuning_nav_link.py`, `test_smoke.py`.
5. Full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (>90%), `uv run ruff check . && uv run pyright`.
6. **UI Structure Check** (AGENTS.md rule 5) on the new page (full-width table, labels, contrast, focus-visible, aria-live on the list updates) + no CDN (rule 6).
7. **Commit** (A17): stage this phase's files (`app/**`, `alembic/**`, `frontend/**`, `tests/**`, `README.md`, `.env.example`), message `feat(sources): admin page to add and remove git sources (TODO.md L4)`, always `--no-gpg-sign`. Move `.agent/phases/todo/35_git_sources_admin/` to `.agent/phases/complete/`.
## Testing & Quality
- E2E: `tests/e2e/test_git_sources_admin.py` green **in isolation** (A16: one story, one file).
- Unit/integration: from tasks 01–03 — all green under `uv run pytest`.
- Coverage: **>90%** on `app/` (new API module + resolver fully covered).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov` green in isolation.
- [ ] `test_nav_consistency.py` updated for the fourth nav link and green; every suite in the task 06 regression list green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] README + `.env.example` document the fallback semantics.
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
@@ -1,40 +0,0 @@
# Phase 36 — Document Summary Shown Together With the Original
**Source:** `TODO.md` L5 — "When I click on a document with a summary I should be able to see the summary and the original document together."
**Story:** `.agent/user_stories/summary-in-viewer.md`
**Context:** Phase 30 stores a lite-model summary on `documents.summary` for every non-markdown document (plus an indexed `is_summary` chunk) — but the viewer never shows it: `GET /api/documents/content` omits the field and the shared renderer `renderDocument` (`frontend/assets/document.js`, used by BOTH the full-page viewer `document.html` and the chat/sources modal, phase 26) only renders the raw content. Markdown documents carry no summary (phase 30) and must render exactly as before.
## Objective
When a document **has** a summary, show it and the original content together — a labeled Summary panel above the content, on both viewer surfaces at once (shared renderer); documents without a summary are unchanged.
## Dependencies
- `30_document_summaries` (complete) — the `documents.summary` column (migration 0004), the summarizer, the `summary_kb` E2E fixture + the deterministic mock `SUMMARY_MODE` digest.
- `10_story_document_viewer` + `26_document_modal_viewer` (complete) — the viewer page, the modal, and the shared `renderDocument(doc, {titleEl, metaEl, contentEl})` core both surfaces render through.
- `16_admin_auth` (complete) — the soft rule this phase must not touch: the content endpoint stays public + stateless (catalog gated, viewer public).
## Tasks
1. `01_content_api_summary_field.md` — `DocContent.summary` + the endpoint returns it; integration tests.
2. `02_viewer_summary_panel.md` — the shared renderer draws the `.doc-summary` panel (both surfaces) + theme-matched styles.
3. `03_e2e_and_regression.md` — the story E2E suite `test_summary_in_viewer.py`; regressions; full gate; commit.
## Testing & Quality
- Unit/integration: the content endpoint returns the summary for a summarized non-markdown doc and `null` for a markdown doc; anonymous access unchanged.
- Coverage: **>90%** on `app/` (the touched endpoint stays covered).
- E2E (mandatory, A16): `tests/e2e/test_summary_in_viewer.py` — the story gate, run in isolation.
## Completion Criteria
- [ ] `GET /api/documents/content` returns `summary` (string or null); no auth/shape change beyond the added nullable field; the endpoint is still public.
- [ ] A summarized document shows the labeled Summary panel **above** the original content in the full-page viewer AND the modal; the original content (including content the summary digest doesn't contain) is fully visible.
- [ ] A markdown document (no summary) renders exactly as before on both surfaces — no empty panel.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run pytest tests/e2e/test_summary_in_viewer.py -v --no-cov` green in isolation; regressions (task 03 list) green.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5): the panel is a labeled section, contrast ≥4.5:1, no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
## Locked decisions
- **A7 / A15 untouched** — retrieval, context assembly, and the SSE contract are unchanged; this is a display + API-field phase.
- **Phase 16 soft rule untouched** — `GET /api/documents/content` stays public + stateless (anyone who can open a document sees its summary; the catalog stays admin-gated).
- **Phase 30 untouched** — summaries are still generated at import, still markdown-excluded, still fail-soft (NULL possible); this phase only surfaces the existing field.
- **Shared-renderer principle (phase 26)** — the panel is drawn in `renderDocument`, so the page and the modal can never drift.
- **A11 untouched** — vanilla HTML/CSS/JS, no CDN, no new packages; summary text rendered with `textContent` (XSS contract unchanged).
- **A16 / A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
@@ -1,32 +0,0 @@
# Task 01 — The content endpoint returns the summary
**Phase:** `36_summary_in_viewer` · **Source:** `TODO.md:5 — "When I click on a document with a summary I should be able to see the summary and the original document together."`
**Story:** `.agent/user_stories/summary-in-viewer.md`
## Objective
Surface the existing `documents.summary` field (phase 30) on the viewer's data contract: `DocContent` gains a nullable `summary` and `GET /api/documents/content` returns it — endpoint stays public, stateless, and otherwise byte-identical.
## Work
1. `app/schemas.py` — `DocContent` gains:
```python
#: Lite-model summary (phase 30) — non-markdown A9 docs only; None for
#: markdown documents, pre-phase-30 rows, and the fail-soft path where
#: summary generation failed but the document was still indexed.
summary: str | None = None
```
(Place it after `format` / before `content`, with the docstring mirroring `Document.summary`'s.)
2. `app/api/docs.py` — `get_document_content` returns `summary=doc.summary` in the `DocContent(...)` construction. Nothing else changes (no auth, no query change — `Document` is already selected in full).
3. Integration tests — extend the existing `/api/documents/content` test module (find it in `tests/integration/` — the phase-10 content-endpoint tests):
- A non-markdown document row seeded with `summary="…"` → response JSON carries `summary` verbatim.
- A markdown document row with `summary=None` → `"summary": null`.
- Anonymous (no admin cookie) still gets 200 (the phase-16 soft rule — public viewer) for both.
- The existing assertions (404 on unknown pair, content/format fields) stay green unmodified.
## Testing & Quality
- Integration: the cases above; `uv run pytest` green overall.
- Coverage: **>90%** on `app/` — the touched endpoint stays covered (the new field is exercised by the new assertions).
## Completion Criteria
- [ ] `GET /api/documents/content?source=…&path=…` returns the summary for a summarized doc and `null` for a markdown doc; anonymous access unchanged (200).
- [ ] No other field, status code, or auth behavior of the endpoint changed.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
@@ -1,44 +0,0 @@
# Task 02 — The shared renderer draws the Summary panel (both surfaces)
**Phase:** `36_summary_in_viewer` · **Source:** `TODO.md:5 — "When I click on a document with a summary I should be able to see the summary and the original document together."`
**Story:** `.agent/user_stories/summary-in-viewer.md`
## Objective
`renderDocument` — the single rendering core both the full-page viewer (`document.html`) and the chat/sources modal (`document-modal.js`) go through (phase 26) — draws a labeled Summary panel above the original content whenever `doc.summary` is non-empty; `null`/empty renders nothing, so markdown documents and fail-soft rows are byte-for-byte unchanged.
## Work
1. `frontend/assets/document.js` — in `renderDocument(doc, { titleEl, metaEl, contentEl })`, **after** the meta row is populated and **before** the content is built, append the summary section to `contentEl` (which is then filled with the usual `.doc-md` / `<pre class="doc-raw">` content after it):
```js
if (doc.summary && doc.summary.trim() !== "") {
const section = document.createElement("section");
section.className = "doc-summary";
section.setAttribute("aria-label", "Summary");
const title = document.createElement("h2");
title.className = "doc-summary-title";
title.textContent = "Summary";
const body = document.createElement("p");
body.className = "doc-summary-text";
body.textContent = doc.summary; // text node — XSS contract unchanged
section.append(title, body);
contentEl.appendChild(section);
}
```
- The meta badge row (`#doc-meta` / `#doc-modal-meta`) is untouched; the panel sits between meta and content on **both** surfaces because both call this one function.
- Update the file-header comment (the renderer now also owns the optional summary panel) and the `renderDocument` doc comment.
2. `frontend/assets/styles.css` — `.doc-summary` (dark tech theme, phase-08 palette):
- A clearly-distinct "summary, not content" look: surface `#121a2e` with a 3px brand left border (`#6d78f2`) or a brand-soft (`#232b52`) header strip — pick one and keep it simple; `border-radius` matching the existing content cards; padding ~1rem; `margin-bottom` separating it from the content.
- `.doc-summary-title` — small-caps/label treatment: `#a5b4fc` (brand-ink on the brand-soft chip, ≈6.9:1) or `#a5b4fc` on surface (verify ≥4.5:1 — if short, use the chip).
- `.doc-summary-text` — `var(--ink)` (`#e8ebf4`) on the surface (≈14.5:1); wraps inside the same width the content uses (the ≤46rem centered column for md docs, the raw-content width otherwise — the panel is a child of `contentEl`, so it inherits the column; verify for the `<pre class="doc-raw">` case where the content is wider).
- No animation (nothing for `prefers-reduced-motion` to still); the section is static content — no focusability needed (it carries `aria-label` + heading).
3. Do NOT touch `document-modal.js` (it calls `renderDocument` — the panel comes for free), `markdown.js`, or the page scripts.
## Testing & Quality
- Frontend-only — no Python change; the no-CDN integration test is unaffected.
- Coverage: `app/` gate unaffected.
- Manual smoke (dev server with an imported non-md doc, e.g. a yaml from the fixture KB via the importer + mock, or a hand-seeded `documents.summary` row): modal from the Sources table AND the full page both show panel + content; a markdown doc shows no panel.
## Completion Criteria
- [ ] A document with a summary shows the labeled panel above the original content in **both** the modal and the full-page viewer; the original content is fully visible (nothing hidden/collapsed).
- [ ] A document without a summary (`null` or empty/whitespace) renders exactly as before on both surfaces — no panel, no empty box.
- [ ] The summary text is written with `textContent` (XSS contract); the meta row is unchanged.
- [ ] `uv run pytest` green (no-CDN test); `uv run ruff check . && uv run pyright` clean.
@@ -1,31 +0,0 @@
# Task 03 — Story E2E + regression pass + commit
**Phase:** `36_summary_in_viewer` · **Source:** `TODO.md:5 — "When I click on a document with a summary I should be able to see the summary and the original document together."`
**Story:** `.agent/user_stories/summary-in-viewer.md`
## Objective
Prove the story end-to-end with its dedicated Playwright suite — summary panel + original content visible together on both surfaces, no panel for markdown docs — then the full gate + one commit.
## Work
1. `tests/e2e/test_summary_in_viewer.py` (NEW — the story gate, run in isolation). Reuse `tests/e2e/test_document_summaries.py`'s machinery as closely as possible (its fixture KB + import helpers are the reference):
- Fixtures: the standard E2E app + DB + the deterministic mock LLM (`tests/e2e/mock_llm.py` — its `SUMMARY_MODE` answer is a byte-stable 24-token digest); import `tests/fixtures/summary_kb/` so `quadlet/qwen-llamacpp.yaml` gets its stored summary + `notes/qwen-llamacpp-notes.md` stays summary-less (the existing tail sentinel `RESE-SUMMARY-SENTINEL-7f3a` sits on the yaml's LAST line — **outside** the 24-token digest, so it is a marker for "the original, not the summary").
- **Full page:** open `/document.html?source=summary_kb&path=quadlet%2Fqwen-llamacpp.yaml` → `.doc-summary` visible with the deterministic digest text AND the original content visible with the sentinel (`RESE-SUMMARY-SENTINEL-7f3a` present in the rendered content) — summary and original **together**.
- **Modal:** from the Sources table (admin session via `tests/e2e/auth_helpers.py`), click the yaml's row → the modal shows the same panel + content (sentinel present, digest present); then "Full page" still lands on the dedicated page with the panel (the two surfaces agree).
- **No-summary control:** the markdown doc (`notes/qwen-llamacpp-notes.md`) → no `.doc-summary` element on the full page and in the modal; the content renders as before.
- **API shape (cheap, via the page context's `fetch` or `context.request`):** `GET /api/documents/content` for the yaml carries `summary` (string), for the md doc `null`; anonymous fetch → 200 (soft rule unchanged).
- The E2E must not depend on a real LLM (the mock's digest is deterministic — the same pattern `test_document_summaries.py` relies on).
2. **Regression pass — each in isolation** (`uv run pytest tests/e2e/<file>.py -v --no-cov`): `test_summary_in_viewer.py` (new), `test_document_viewer.py`, `test_document_summaries.py`, `test_document_back_navigation.py`, `test_chat_rag.py` (the source-chip modal path), `test_smoke.py`.
3. Full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (>90%), `uv run ruff check . && uv run pyright`.
4. **UI Structure Check** (AGENTS.md rule 5): the panel is a labeled section (`aria-label` + heading), theme contrast ≥4.5:1, it does not break the centered 46rem chat-column-width content layout, no CDN (rule 6 — the no-CDN integration test covers it).
5. **Commit** (A17): stage this phase's files (`app/schemas.py`, `app/api/docs.py`, `frontend/assets/document.js`, `frontend/assets/styles.css`, `tests/**`), message `feat(viewer): show document summary together with the original (TODO.md L5)`, always `--no-gpg-sign`. Move `.agent/phases/todo/36_summary_in_viewer/` to `.agent/phases/complete/`.
## Testing & Quality
- E2E: `tests/e2e/test_summary_in_viewer.py` green **in isolation** (A16: one story, one file).
- Unit/integration: from task 01 — all green under `uv run pytest`.
- Coverage: **>90%** on `app/` (the endpoint change is covered).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_summary_in_viewer.py -v --no-cov` green in isolation.
- [ ] Every suite in the task 03 regression list green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
@@ -1,39 +0,0 @@
# Phase 39 — Configurable app name (brand)
**Source:** `TODO.md` L12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."
**Story:** `.agent/user_stories/configurable-brand.md`
**Context:** `app/config.py` (`app_name`, already `BOR_APP_NAME`, currently used only for the OpenAPI title at `app/main.py:48`), the five templates (`frontend/*.html` — the `.brand-text` spans, the `<title>`s, the meta descriptions, the index empty-state h1, the `aria-label`s), `frontend/assets/app.js` (status labels ~L107/112, the elapsed-hint aria ~L540), `frontend/assets/document.js` (page titles L147/152), `app/api/health.py` (the public stateless endpoint pattern), `app/core/caching.py` (phase 33: `?v=` rewriting of `assets/…` refs — automatic for any new asset file), the `Containerfile` esbuild stage (explicit per-asset lines, L17–23).
## Objective
Make **one env var** (`BOR_APP_NAME`, default "Brain of Reese") drive the app's display name everywhere — titles, the header brand, the status labels, the aria text, the greeting — via a public `/api/config` endpoint + a small `brand.js`, with zero behavior change when the variable is unset.
## Dependencies
- `34_consistent_navbar` (todo — runs before this phase) — the standard five-page header (the `.brand-text` nodes this phase re-skins).
- `01_infrastructure` (complete) — the public stateless endpoint pattern (`app/api/health.py`).
## Tasks
1. `01_config_endpoint.md` — public `GET /api/config` → `{app_name, version}` + tests.
2. `02_frontend_branding.md` — `brand.js` + the template/JS de-hard-coding + the Containerfile build line.
3. `03_e2e_docs_commit.md` — the story E2E (its own app instance with the overridden name), `.env.example` + README, commit, move the phase dir.
## Testing & Quality
- Unit/integration: the endpoint (200, the values, anonymous access, exactly two keys — no settings may leak later).
- E2E (mandatory, A16): `tests/e2e/test_configurable_brand.py` — a **second** app instance booted with `BOR_APP_NAME` overridden; run in isolation.
- Coverage: **>90%** on `app/`.
- Regression: the default-name behavior is byte-identical — the existing suites (which assert "Brain of Reese" titles/labels against the shared conftest server) stay green **unchanged**.
## Completion Criteria
- [ ] `GET /api/config` (anonymous) → `{"app_name": "Brain of Reese", "version": "0.1.0"}` by default; the response key set is exactly `{app_name, version}`.
- [ ] With `BOR_APP_NAME="Brain of Testy"`: the index title "Brain of Testy"; the header `.brand-text` renders `Brain of <strong>Testy</strong>`; the empty-state h1 "Hey! I'm Brain of Testy."; the `#messages` aria-label "Conversation with Brain of Testy"; the chat status label "Brain of Testy is thinking"; the other pages' titles carry the name; the document viewer title "… · Brain of Testy".
- [ ] With the variable unset: the existing E2E + unit suites green unchanged (no rename leak).
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run pytest tests/e2e/test_configurable_brand.py -v --no-cov` green in isolation.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6).
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
## Locked decisions
- **The existing `app_name` setting is the source of truth (owner permission 2026-08-26)** — `BOR_APP_NAME` (default "Brain of Reese"); no new variable, no setting rename.
- **A11 honoured** — runtime fetch + JS application (no build-time template injection, no Jinja, no CDN); the static templates stay static.
- **A10 honoured** — `/api/config` is public and stateless; the response carries no secrets (exactly `app_name` + `version`).
- **Runtime fetch, brief flash accepted (owner permission 2026-08-26)** — the default name renders immediately and is replaced when `/api/config` answers (LAN latency; no re-paint machinery for in-flight turns — a mid-turn label keeps the previous name for that turn).
- **A16/A17 honoured** — one new story E2E suite + one atomic `--no-gpg-sign` commit.
@@ -1,27 +0,0 @@
# Task 03 — Story E2E + docs + commit
**Phase:** `39_configurable_brand` · **Source:** `TODO.md:12 — "Also need a way to customize the name for 'Brain of'. Should be an env var."`
**Story:** `.agent/user_stories/configurable-brand.md`
## Objective
The story's isolated Playwright suite against an app instance booted with the overridden name, the env docs, and the phase commit.
## Work
1. `tests/e2e/test_configurable_brand.py` (the story gate — one story, one file, run in isolation):
- A **second app instance** — the shared `app_server` conftest fixture keeps the default name (the other suites' title/label assertions depend on it). Copy the conftest `app_server` env block (same DB, the mock-LLM base URL, `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET`, `BOR_STATIC_DIR`, `BOR_RELEVANCE_THRESHOLD`) with two changes: `BOR_APP_NAME="Brain of Testy"` and a distinct port (`APP_PORT + 1` per the conftest convention). A session-scoped fixture **inside the test file**, started after `mock_llm` is available.
- Assertions (custom instance): index `document.title` == `"Brain of Testy"`; the `.brand-text` `innerHTML` == `Brain of <strong>Testy</strong>`; the empty-state h1 text == `"Hey! I'm Brain of Testy."`; the `#messages` `aria-label` == `"Conversation with Brain of Testy"`; the sources page title `"Sources · Brain of Testy"`; the login page title `"Sign in · Brain of Testy"`; one chat turn with a pre-token window (the `think out loud` marker) → the button label shows `"Brain of Testy is thinking"`.
- Default-name assertion (cheap regression in the same file): the shared conftest server's index title still == `"Brain of Reese"`.
- The chat-turn assertion works on the default (possibly empty) KB — a deflected answer is fine; the label assertion is pre-token, so no DB seeding is required.
2. `.env.example` — document `BOR_APP_NAME` in the App section (the display name on all pages; default "Brain of Reese").
3. README — the configuration section: `BOR_APP_NAME` (what it affects: titles, the header brand, the status labels, the aria text; the default; the bold-split rendering rule: names starting "Brain of " bold the remainder, any other name renders in normal weight).
4. Regression pass: `uv run pytest` + the coverage gate (>90%) + the isolated story E2E + the suites that assert brand strings (`test_smoke.py`, `test_shared_header.py`, `test_header_consistency.py`, `test_chat_persistence.py`) green.
5. Commit — one atomic `--no-gpg-sign` Conventional Commits commit for the whole phase (AGENTS.md rule 8), e.g. `feat(brand): configurable app name — BOR_APP_NAME drives /api/config + the frontend brand layer`; move the phase directory to `.agent/phases/complete/`.
## Testing & Quality
- The gates above are this task's quality bar (A16: one story, one isolated E2E file, coverage >90%).
## Completion Criteria
- [ ] The story E2E is green in isolation, deterministic across two consecutive runs (custom-name instance + the default-name assertion).
- [ ] The step-4 regression list green; coverage >90%.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase directory moved to `.agent/phases/complete/`.
@@ -0,0 +1,37 @@
# Phase 40 — Tuning toggle anonymous flash
**Source:** `TODO.md` L3 — "Loading the page briefly shows the 'Tuning' button in the header even when the user isn't authenticated. Only show that if the user is authenticated."
**Story:** `.agent/user_stories/tuning-toggle-flash.md`
**Context:** Phase 15/34 shared header (`frontend/assets/header.js` owns `#steering-toggle` / `#steering-panel` on all six pages; anonymous → `remove()` post-whoami). The admin-only **nav links** already ship `hidden` (phase-19 contract) — the flashing control is the **steering toggle button labeled "Tuning"**, which ships visible in all six pages and is removed only after `/api/whoami` resolves.
## Objective
Kill the anonymous flash: the tuning toggle ships `hidden` in every page's markup and is revealed only when whoami says admin (the exact ship-hidden / reveal-for-admin contract the nav links use), so an anonymous user never sees the "Tuning" button — not for a single frame.
## Dependencies
- `39_configurable_brand` (complete; last existing phase) — current header state: full shared bar on all six pages.
- `19_shared_header` / `16_admin_auth` / `34_consistent_navbar` (complete) — the `fetchIsAdmin()` gate, the ship-hidden nav contract, and the module-owned steering controls this task modifies.
## Tasks
1. `01_toggle_ships_hidden.md` — add `hidden` to `#steering-toggle` in all six pages and reveal-for-admin in `header.js`; pin at source level.
2. `02_flash_e2e_and_regression.md` — story E2E suite (never-visible-for-anonymous, admin reveal, nav-contract regression) + regression pass + commit.
## Testing & Quality
- Unit: new `tests/unit/test_steering_toggle_visibility.py` — `hidden` present on `#steering-toggle` in all six HTML pages; `header.js` unhides for admin (line before `refreshSteering()`) and the anonymous `remove()` path is intact; any existing source-pin test asserting the exact old markup is updated (check `tests/unit/test_shared_header.py`, `test_steering.py`).
- Coverage: frontend-only — the `app/` >90% gate is unaffected (must stay unchanged).
- E2E (mandatory, A16): `tests/e2e/test_tuning_toggle_flash.py`, run in isolation.
## Completion Criteria
- [ ] Anonymous load of every page: the toggle is never visible (MutationObserver records zero visible frames) and is absent from the DOM after load.
- [ ] Admin load: toggle visible, panel opens, count badge correct — admin behavior unchanged.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged.
- [ ] `uv run pytest tests/e2e/test_tuning_toggle_flash.py -v --no-cov` green in isolation.
- [ ] Regression E2E suites green in isolation: `test_shared_header.py`, `test_global_tuning.py`, `test_steering.py`, `test_tuning_nav_link.py`, `test_smoke.py`.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5): no new focus targets; landmarks/contrast unchanged; no CDN.
- [ ] One `--no-gpg-sign` commit; phase dir moved `.agent/phases/todo/` → `.agent/phases/complete/`.
## Locked decisions
- **A10 untouched** — no API change; the fix is pure UI visibility off the existing `/api/whoami` gate.
- **A11 untouched** — no new assets, no CDN.
- **Phase-16 contract preserved** — anonymous still gets "absent, not hidden" (remove-from-DOM); this phase only removes the pre-whoami flash window.
- **A16/A17 honoured** — one story E2E suite, one atomic commit.
@@ -0,0 +1,26 @@
# Task 01 — Toggle ships hidden, admin-only reveal
**Phase:** `40_tuning_toggle_flash` · **Source:** `TODO.md:3` — "Loading the page briefly shows the 'Tuning' button in the header even when the user isn't authenticated. Only show that if the user is authenticated."
**Story:** `.agent/user_stories/tuning-toggle-flash.md`
## Objective
Ship `#steering-toggle` `hidden` in all six pages and unhide it in `header.js` only when whoami says admin — zero flash for anonymous, identical admin UX.
## Work
1. `frontend/index.html`, `frontend/sources.html`, `frontend/document.html`, `frontend/git-sources.html`, `frontend/login.html`, `frontend/tuning.html` — add the `hidden` attribute to the existing `#steering-toggle` `<button>` (the element that ships `aria-expanded="false" aria-controls="steering-panel"`; keep every other attribute, icon, label, and count badge byte-identical). The `#steering-panel` section already ships `hidden` — do not touch it.
2. `frontend/assets/header.js` — in `initSharedHeader()`, in the `if (admin)` branch, add `if (steeringToggle) steeringToggle.hidden = false;` **before** `if (steeringPanel) refreshSteering();`. The anonymous branch (`steeringToggle?.remove(); steeringPanel?.remove();`) stays byte-identical. Update the module docstring: the steering controls are now ship-hidden / reveal-for-admin (2026-08-27, `TODO.md` L3), matching the admin-only nav links.
3. `tests/unit/test_steering_toggle_visibility.py` (new) — source pins in the house style (regex/substring over the HTML + JS files, see `tests/unit/test_sync_button.py`):
- `#steering-toggle` carries `hidden` in **all six** pages;
- `header.js` contains the admin unhide (`steeringToggle.hidden = false`) inside `initSharedHeader`;
- the anonymous removal (`steeringToggle?.remove()`) is still present;
- `#nav-tuning` still ships `hidden` (the contract this phase relies on).
4. Grep the existing suites for exact-markup pins of the toggle (`tests/unit/test_shared_header.py`, `tests/unit/test_steering.py`, `tests/e2e/test_global_tuning.py`, `test_steering.py`) and update any assertion that breaks purely because of the new `hidden` attribute — behavior assertions stay.
## Testing & Quality
- Unit: the new pin file above; full unit suite green.
- Coverage: **>90%** on `app/` (no Python change — TOTAL must be unchanged; run `uv run pytest --cov=app --cov-report=term-missing`).
## Completion Criteria
- [ ] All six pages ship `#steering-toggle` with `hidden`; `header.js` reveals for admin and still removes for anonymous.
- [ ] `uv run pytest` green; coverage TOTAL unchanged.
- [ ] No behavior change in completed work (admin steering flow byte-identical: open/close, count badge, delete, announcer).
@@ -0,0 +1,26 @@
# Task 02 — Flash E2E + regression + commit
**Phase:** `40_tuning_toggle_flash` · **Source:** `TODO.md:3` — "Loading the page briefly shows the 'Tuning' button in the header even when the user isn't authenticated. Only show that if the user is authenticated."
**Story:** `.agent/user_stories/tuning-toggle-flash.md`
## Objective
Prove the flash is gone at the browser level (never visible, not even for a frame) and that the shared-header contract is intact; commit the phase.
## Work
1. `tests/e2e/test_tuning_toggle_flash.py` (new) — mock-only suite (DB up), per the story's Playwright Mapping Rule:
- a helper `install_visibility_observer(page)`: `page.add_init_script` a MutationObserver on `document.documentElement` that appends to `window.__tuningVisibleFrames` every time `#steering-toggle` is added/attribute-changed and is both in the DOM **and** not `[hidden]` (check `el.offsetParent !== null` or `!el.hidden`);
- `test_anonymous_never_sees_toggle` — load `/` anonymously, wait for network idle + header settle (whoami resolved), assert `window.__tuningVisibleFrames` is empty and `#steering-toggle` is absent from the DOM;
- `test_anonymous_other_pages_never_flash` — same on `/sources.html`, `/tuning.html`, `/login.html`;
- `test_admin_toggle_revealed_and_working` — `login()` (e2e.auth_helpers), reload `/`, toggle visible + clickable (opens `#steering-panel`, `aria-expanded="true"`), count badge matches the list;
- `test_nav_contract_regression` — anonymous: `#nav-sources` / `#nav-git-sources` / `#nav-tuning` stay hidden; admin: revealed.
2. Regression pass (isolation runs, per A16): `test_shared_header.py`, `test_global_tuning.py`, `test_steering.py`, `test_tuning_nav_link.py`, `test_smoke.py` — all green; fix only true regressions.
3. `uv run pytest` (unit+integration) green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged; `uv run ruff check . && uv run pyright` clean.
4. Commit (Conventional Commits, `--no-gpg-sign`), e.g. `fix(header): ship the tuning toggle hidden — no anonymous flash`, staging this phase's changed files; move `.agent/phases/todo/40_tuning_toggle_flash/` → `.agent/phases/complete/` (force-add per AGENTS.md rule 8 if the history tracks the tree).
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_tuning_toggle_flash.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`).
- Coverage: **>90%** on `app/` (unchanged — frontend-only phase).
## Completion Criteria
- [ ] The story E2E file passes in isolation; the four regression suites pass in isolation.
- [ ] One atomic `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
@@ -0,0 +1,38 @@
# Phase 41 — Sync fails fast + modal when a model is down
**Source:** `TODO.md` L4 — "If the embedding or lite model is not accessible the sync button should fail fast and there should be a modal error popup explaining that the model isn't available."
**Story:** `.agent/user_stories/sync-model-fail-fast.md`
**Context:** `app/api/sync.py::_run_sync` (phase 32/35/38) runs source resolution → git clones → `import_sources` (embeds) → overview (lite) — with a dead LLM endpoint the run discovers it only mid-import, after slow clones. The sync state machine is module-owned by `frontend/assets/header.js` (`applySyncFailure` → button title/aria + `.is-error` + `bor:sync-status` event; the Sources page renders `#sync-error-banner`). No dialog component exists yet.
## Objective
When `embed` or `lite` is unreachable, the sync fails **before any expensive work** with a message naming the model, and the failure is readable in a **modal dialog** on every page that carries `#sync-btn`.
## Dependencies
- `40_tuning_toggle_flash` (todo) — current shared-header state (sequential; no code overlap, but both touch `header.js` — keep this phase's changes confined to the sync section).
- `32_admin_sync_button` / `35_git_sources_admin` / `38_local_directory_sources` (complete) — the pipeline, the status contract, and the module-owned button lifecycle this phase extends.
## Tasks
1. `01_model_probe_fail_fast.md` — `check_models()` probe in `app/rag/llm.py`, called first in `_run_sync`; unit + integration tests.
2. `02_sync_error_modal.md` — `header.js` modal (built in JS, all pages) + CSS; source pins.
3. `03_model_down_e2e_and_commit.md` — dedicated E2E suite (dead-LLM module app) + phase-32 regressions + commit.
## Testing & Quality
- Unit: probe success/failure paths with a fake LLM client (embed-down, lite-down, both up); the sync task's fail-fast ordering (probe before source resolution — assert no clone call happens).
- Integration: `POST /api/sync` with a stubbed failing client → `GET /api/sync/status` reaches `failed` with the model-naming error; healthy path regression.
- Coverage: **>90%** on `app/` including the new probe code.
- E2E (mandatory, A16): `tests/e2e/test_sync_model_down.py`, run in isolation.
## Completion Criteria
- [ ] With a dead LLM endpoint: sync fails within seconds, **before** any clone, error names the unavailable model; the modal shows it; button settles retry-ready.
- [ ] Modal contract: `role="alertdialog"`, `aria-modal`, text via `textContent`, close via button / `Esc` / backdrop, focus in-and-out.
- [ ] Healthy sync pipeline (clone → import → overview) unchanged — phase-32 suite green in isolation.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
- [ ] `uv run pytest tests/e2e/test_sync_model_down.py -v --no-cov` green in isolation (DB up).
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
## Locked decisions
- **A12 untouched** — still in-process, no queue, no new service; the probe is two cheap model calls.
- **A10 untouched** — no new endpoint; `/api/sync` + `/api/sync/status` keep their shapes (a model failure is just another `failed` state).
- **Phase-32 contract kept** — 2 s poll, 202/409, no client timeout, `bor:sync-status` event, button title/aria affordance, Sources banner (the modal is additive).
- **Owner-locked (2026-08-27, roadmap A4):** probe runs **before** git clones; the modal is the primary failure surface on every page.
@@ -0,0 +1,37 @@
# Task 01 — Model probe: fail fast before any clone
**Phase:** `41_sync_fail_fast_models` · **Source:** `TODO.md:4` — "If the embedding or lite model is not accessible the sync button should fail fast and there should be a modal error popup explaining that the model isn't available."
**Story:** `.agent/user_stories/sync-model-fail-fast.md`
## Objective
The sync run verifies both models it needs (`embed` + the summary `lite`) **first** — before source resolution, before any `clone_or_pull` — and fails the run with a clear, model-naming error when either is unreachable.
## Work
1. `app/rag/llm.py` — add `class ModelUnavailableError(LLMError)` and:
```python
async def check_models(llm: LLMClient) -> None:
"""Verify the models a sync needs (embed + summary) before any
expensive work; raise ModelUnavailableError naming the model."""
```
- `await llm.embed_one("sync model check")` — wrap `EmbeddingError` (and any other exception) in `ModelUnavailableError`: message names the **embedding model** (use `llm.settings.llm_embed_model`, e.g. "The embedding model ('embed') is not available — check the model endpoint and retry.")
- `await llm.chat([{"role": "user", "content": "ping"}])` (defaults to `llm_summary_model`) — wrap `LLMError`/other in `ModelUnavailableError` naming the **summary model** (`llm.settings.llm_summary_model`, e.g. "The summary model ('lite') is not available — check the model endpoint and retry.").
- Docstring notes the probe is deliberately tiny (one short embedding + one 1-token-scale completion) and that the sync sanitizer downstream still masks any embedded credentials.
2. `app/api/sync.py` — in `_run_sync()`, construct `llm = LLMClient()` **before** the DB/source block and call `await check_models(llm)` as the **first** pipeline step (before `effective_sources`, before the clone loop). Update the module docstring's pipeline list (the probe is step 1: "verify `embed` + summary model availability — fail fast before any clone") and renumber. `ModelUnavailableError` falls into the existing `except Exception` → `failed` state with the sanitized error (no special-casing needed — verify the message survives `_sanitize_error` unchanged).
3. `tests/unit/test_sync_model_probe.py` (new) — with a fake LLM client (duck-typed `embed_one`/`chat`, see `tests/fakes.py` `FakeEmbedder` for the shape):
- both up → `check_models` returns, both methods called;
- embed raises → `ModelUnavailableError` mentioning the embed model name, `chat` never called;
- chat raises → `ModelUnavailableError` mentioning the summary model name;
- message content assertions (model name present, "not available" wording).
4. `tests/integration/test_sync_api.py` — extend:
- **fail-fast:** monkeypatch `app.api.sync.LLMClient` (or `check_models`) so the probe raises `ModelUnavailableError`; also monkeypatch `clone_or_pull` to *assert it is never called*; `POST /api/sync` → poll `GET /api/sync/status` until terminal → `state == "failed"`, `error` names the model;
- **ordering:** a spy on `effective_sources` shows the probe ran before it;
- **healthy regression:** the existing success/failure tests stay green (they stub the LLM — the stub must now satisfy the probe: `FakeEmbedder` already implements `chat`; if the existing stub lacks `embed_one`, add it — `FakeEmbedder.embed` exists, so subclass or delegate).
## Testing & Quality
- Unit/integration as above; the probe must be covered (success + both failure modes) to keep `app/` **>90%**.
- `uv run pytest tests/unit/test_sync_model_probe.py tests/integration/test_sync_api.py -v` green; full suite green.
## Completion Criteria
- [ ] `check_models` exists, is called first in `_run_sync`, and names the failing model in `ModelUnavailableError`.
- [ ] A dead-model sync fails **before** any clone (spy-asserted) with a sanitized, model-naming error in the `failed` state.
- [ ] Healthy pipeline behavior unchanged (existing sync integration tests green).
@@ -0,0 +1,29 @@
# Task 02 — Sync error modal (module-owned, every page)
**Phase:** `41_sync_fail_fast_models` · **Source:** `TODO.md:4` — "If the embedding or lite model is not accessible the sync button should fail fast and there should be a modal error popup explaining that the model isn't available."
**Story:** `.agent/user_stories/sync-model-fail-fast.md`
## Objective
A readable, accessible modal dialog for sync failures, built by the shared header module (which owns the sync state machine), so every page carrying `#sync-btn` gets it with zero page-markup changes.
## Work
1. `frontend/assets/header.js` — in the sync section, add:
- `showSyncModal(error)`: lazily create the dialog **once** and append to `document.body` (module-level `let syncModal = null`):
- backdrop `<div class="sync-modal-backdrop">`;
- panel `<div class="sync-modal" role="alertdialog" aria-modal="true" aria-labelledby="sync-modal-title" aria-describedby="sync-modal-error">` with an `<h2 id="sync-modal-title">Sync failed</h2>`, a `<p id="sync-modal-error">` whose text is set via **`textContent`** (the sanitized error — XSS-safe, never `innerHTML`), and a `<button type="button" class="sync-modal-close" aria-label="Close error dialog">` (×);
- opening: add a `.is-open` class (or remove `hidden`), move focus to the close button, remember `document.activeElement` (expected `#sync-btn`);
- closing: reverse (focus returns to the remembered element — `#sync-btn` when present), `Esc` keydown on `document` while open, backdrop click (click on the backdrop element itself, not the panel), and the close button all call the same close function; a second failure while open **updates the error text in place** (no stacking).
- call `showSyncModal(status.error)` from `applySyncFailure(status)` **after** the existing button-title/aria/`.is-error` + `emitSyncStatus` lines (those stay byte-identical — the Sources page's `#sync-error-banner` keeps rendering off the event).
- null-safe: everything guards on `syncBtn`/`document.body`; pages without `#sync-btn` never create the modal (the function is only reachable from the sync state machine).
- Update the module docstring's sync bullet: the failed state now also opens the module-owned error modal (2026-08-27, `TODO.md` L4).
2. `frontend/assets/styles.css` — `.sync-modal-backdrop` (fixed, full-viewport, `rgba` dim over the page, `z-index` above the header) + `.sync-modal` (centered panel, max-width ≈28rem, the dark-theme **error palette** from PLAN §7.2: panel on the error-surface `#2d1318` family, text `#fca5a5`-class ink, 1px error border; title in ink, error text ink-soft-on-error-surface ≥4.5:1); open/close via `.is-open` (visibility/opacity, no motion under `prefers-reduced-motion`); the close button keeps the global `:focus-visible` 3px outline; 44px touch floor.
3. `tests/unit/test_sync_button.py` — add source pins (house style): `header.js` contains `role="alertdialog"`, the `textContent` assignment of the modal error, the `Esc` close binding, the backdrop-click close, focus return to `#sync-btn`, and the `showSyncModal` call inside `applySyncFailure` (after `emitSyncStatus`); `styles.css` carries the `.sync-modal` rules + the reduced-motion stilling. Update any pin that asserts the exact `applySyncFailure` body.
## Testing & Quality
- Unit: the pins above; full suite green (no `app/` change — coverage TOTAL unchanged).
- Coverage: **>90%** on `app/` (unchanged).
## Completion Criteria
- [ ] `applySyncFailure` opens the modal with the sanitized error; button title/aria + `bor:sync-status` event behavior byte-identical.
- [ ] Modal: `role="alertdialog"`, `aria-modal`, labeled, `textContent`-rendered error, close via button/`Esc`/backdrop, focus in-and-out to `#sync-btn`.
- [ ] No page HTML changed (the modal is JS-built); no CDN (A11).
@@ -0,0 +1,28 @@
# Task 03 — Model-down E2E + regressions + commit
**Phase:** `41_sync_fail_fast_models` · **Source:** `TODO.md:4` — "If the embedding or lite model is not accessible the sync button should fail fast and there should be a modal error popup explaining that the model isn't available."
**Story:** `.agent/user_stories/sync-model-fail-fast.md`
## Objective
Prove the whole story in the browser against a **dead model endpoint** — fast failure, readable modal, dismissal, unchanged secondary surfaces, and an untouched healthy pipeline — then commit the phase.
## Work
1. `tests/e2e/test_sync_model_down.py` (new) — mock-only, DB up, git on PATH. Follow the `test_sync_button.py` module-app pattern, but boot **two** module-scoped apps on distinct ports (import `APP_PORT`, `ADMIN_PASSWORD`, `SESSION_SECRET`, `_wait_http` from `e2e.conftest`; use e.g. `APP_PORT + 41` for the dead-model app so the isolated run never clashes with a session app):
- **dead-model app** env: `BOR_LLM_BASE_URL=http://127.0.0.1:9/v1` (closed port — instant connection refused), `BOR_GIT_SOURCES=file://<the test_sync_button fixture repo pattern>` (a local `file://` fixture repo, built the same way `test_sync_button.py` does — a (regressed, non-fail-fast) run would therefore spend real time cloning before failing), its own `BOR_SOURCES_DIR` under `tmp_path`;
- `test_model_down_fails_fast_with_modal` — admin login, click `#sync-btn`; expect (budget ≤ ~10 s, contrast with the 60 s healthy budget) the button settling retry-ready **and** the modal visible: `role="alertdialog"`, title "Sync failed", error text naming the model ("embedding model" / the model id), `aria-modal="true"`;
- `test_modal_dismissal` — one fresh failure, then close via the × button (focus returns to `#sync-btn`); a fresh failure, close via `Esc`; a fresh failure, close via backdrop click;
- `test_sync_error_surfaces_unaffected` — after a failure the button keeps `title` + `.is-error`; on `/sources.html` (same dead-model app) the `#sync-error-banner` renders the error off `bor:sync-status`;
- `test_healthy_sync_still_succeeds` — a **healthy** module app (same port scheme, `BOR_LLM_BASE_URL` = the session mock like `test_sync_button.py`) runs the full pipeline to "Synced HH:MM" (counts + idempotency as in phase 32) — proves the probe didn't break the happy path.
- Module fixture teardown: terminate both apps (the conftest pattern).
2. Regression pass (isolation runs): `tests/e2e/test_sync_button.py` (phase 32 — must stay green unmodified), `test_git_sources_admin.py`, `test_local_directory_sources.py`.
3. `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
4. Commit (Conventional Commits, `--no-gpg-sign`), e.g. `feat(sync): fail fast with a modal when a model is unavailable`, staging this phase's files; move `.agent/phases/todo/41_sync_fail_fast_models/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_sync_model_down.py -v --no-cov` green in isolation.
- Coverage: **>90%** on `app/` (the probe code is fully covered by task 01's tests).
## Completion Criteria
- [ ] The model-down suite passes in isolation: fast fail before clones, modal with model-naming error, all three dismissal paths, secondary surfaces intact, healthy run unaffected.
- [ ] Phase-32/35/38 regression suites green in isolation.
- [ ] One atomic `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
@@ -0,0 +1,36 @@
# Phase 42 — No reply autoscroll
**Source:** `TODO.md` L5 — "Get rid of the chat reply autoscroll, it's breaking things like making it impossible for the user to scroll while a reply generates."
**Story:** `.agent/user_stories/no-reply-autoscroll.md`
**Context:** Phase 18 ("follow-the-bottom", owner choice 2026-08-23) added `NEAR_BOTTOM_PX = 200` / `isNearBottom()` / `scrollReveal(wrap, behavior, force)` in `frontend/assets/app.js`: the page auto-scrolls on every `thinking` / `tool` / `delta` frame while the user is within 200px of the bottom. The owner now finds that fighting their own scroll. The gate and the per-frame scrolls are **removed**; scrolling happens only on explicit user intent (submit, restore landing).
## Objective
The chat page never auto-scrolls during a turn. The viewport moves only when the user submits (their message is revealed) or when a persisted conversation is restored (one-shot landing) — both user-initiated.
## Dependencies
- `41_sync_fail_fast_models` (todo) — sequential execution only (no code overlap).
- `18_follow_bottom_scroll` (complete) — the code being removed; `14_chat_persistence` (complete) — the restore landing that must survive; `17_thinking_display` / `11_long_answers` (complete) — the thinking window-pin and long-answer behavior this phase must not break.
## Tasks
1. `01_remove_autofollow.md` — strip the phase-18 gate + per-frame scrolls from `app.js`; rewrite the unit pin for the new contract.
2. `02_no_autoscroll_e2e_and_commit.md` — replace the phase-18 E2E with the inverse-contract suite + regressions + commit.
## Testing & Quality
- Unit: `tests/unit/test_frontend_scroll.py` **rewritten** — pins the new contract: no `NEAR_BOTTOM_PX` / `isNearBottom` in `app.js`; the scroll helper scrolls unconditionally (smooth / reduced-motion-aware); the user-submit path scrolls; the thinking/tool/delta handlers contain **no** page-scroll call; the restore landing keeps its one-shot forced scroll.
- Coverage: frontend-only — `app/` TOTAL unchanged, >90%.
- E2E (mandatory, A16): `tests/e2e/test_no_reply_autoscroll.py`, run in isolation. `tests/e2e/test_follow_bottom_scroll.py` is **deleted** (behavior intentionally removed by owner direction 2026-08-27).
## Completion Criteria
- [ ] During thinking / tool / answer streaming, `window.scrollY` is stable (±1px) while the viewport is scrolled up.
- [ ] Submit still reveals the user's message; reload still lands one-shot on the latest message.
- [ ] `uv run pytest` green; coverage TOTAL unchanged.
- [ ] `uv run pytest tests/e2e/test_no_reply_autoscroll.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_chat_rag.py`, `test_thinking_display.py`, `test_chat_persistence.py`, `test_long_answers.py`, `test_smoke.py`.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
## Locked decisions
- **Owner direction (2026-08-27, roadmap A1)** revises the phase-18 owner choice (2026-08-23): follow-the-bottom auto-follow is removed; submit-reveal + restore-landing are kept. Recorded in the story file and the `app.js` docstring (PLAN.md §7.4's Scroll row is a PLAN-side revision to be noted by the owner — this phase does not edit PLAN.md).
- **A15 unchanged** — the SSE contract is untouched; this is pure client-side behavior.
- **The thinking window's internal pin** (`textEl.scrollTop`, phase 17) is untouched here — phase 43 reworks it separately.
- **A16/A17 honoured** — one story E2E suite (replacing the removed one), one atomic commit.
@@ -0,0 +1,35 @@
# Task 01 — Remove the auto-follow gate and per-frame scrolls
**Phase:** `42_no_reply_autoscroll` · **Source:** `TODO.md:5` — "Get rid of the chat reply autoscroll, it's breaking things like making it impossible for the user to scroll while a reply generates."
**Story:** `.agent/user_stories/no-reply-autoscroll.md`
## Objective
`app.js` scrolls only on explicit user intent: sending a message and the phase-14 restore landing. No scroll happens anywhere in the streaming path.
## Work
1. `frontend/assets/app.js` —
- **Delete** `export const NEAR_BOTTOM_PX = 200` and `function isNearBottom()`.
- **Simplify** `scrollReveal(wrap, behavior = SCROLL, force = false)` → an unconditional `wrap.scrollIntoView({ behavior })` (keep the `SCROLL` constant: smooth, `auto` under `prefers-reduced-motion`; keep the "Calm, don't remove" comment). Rename the gate comment block: the phase-18 "follow-the-bottom scroll contract" paragraph is replaced by the new contract — *"No reply autoscroll (owner direction 2026-08-27, `TODO.md` L5): the page never auto-scrolls while a turn streams. The only scroll call sites are the user submit (reveal my message) and the phase-14 restore landing (one-shot, load-time)."*
- **`addMessage(who, html, scrollBehavior = SCROLL, force = false)`** → change the signature to `addMessage(who, html, scroll = false)`: the internal `scrollReveal(wrap, scrollBehavior, force)` becomes `if (scroll) scrollReveal(wrap)`. Update the call sites (line numbers are pre-change anchors):
- the **user submit** call (`addMessage("user", renderMarkdown(text))`, ~line 896) → `addMessage("user", renderMarkdown(text), true)` (my message must be revealed — the owner-kept behavior);
- the **phase-14 restore** calls (~lines 772/775: `addMessage("user", …, "auto", true)` / `addMessage("brain", …, "auto", true)`) → keep the one-shot forced scroll under the new signature (e.g. `addMessage("user", renderMarkdown(m.text), true)` — the "auto" (non-smooth) behavior for the landing is preserved by passing it through if the new signature keeps a behavior param, otherwise the default `SCROLL` is acceptable and must be noted in the docstring);
- the brain first-bubble creations in the SSE handlers (`if (!wrap) wrap = addMessage("brain", "")`, ~lines 955/978/995, plus the `"…"` fallback ~1004 and the error fallback ~1046) → `scroll: false` (default) — a streaming turn never scrolls the page;
- `addTyping()` (~line 356): the `scrollReveal(wrap)` after `messagesEl.appendChild(wrap)` is **removed** (a typing bubble must not yank the page).
- **SSE handlers** — remove the page-scroll calls: in the `thinking` frame drop the `scrollReveal(wrap); // page follows only while pinned (phase 18)` line **but keep** `textEl.scrollTop = textEl.scrollHeight;` (the thinking *window* pin — phase 17, reworked in phase 43); in the `tool` frame drop its `scrollReveal(wrap);`; in the `delta` frame drop its `scrollReveal(wrap);`.
- Update the file-top docstring's scroll paragraph (lines ~73–81: "Scroll (phase 18, owner choice…)") to the new contract.
2. `tests/unit/test_frontend_scroll.py` — **rewrite** for the new contract (keep the house style — source pins over `app.js`):
- `NEAR_BOTTOM_PX` / `isNearBottom` are **absent** from `app.js`;
- the scroll helper scrolls unconditionally (no `force`-or-near-bottom condition in its body);
- the user-submit `addMessage` call passes the scroll intent; the brain-bubble creation does not;
- the `thinking` / `tool` / `delta` handler bodies contain no `scrollReveal` call (the thinking handler's `textEl.scrollTop` pin is still present);
- the restore landing still performs its one-shot scroll (pin the marker comment / call);
- the `SCROLL` reduced-motion handling is intact.
- Delete the now-obsolete phase-18 test functions (the band constant, the gate logic) — do not leave dead pins.
## Testing & Quality
- Unit: the rewritten pin file + the full suite green (no `app/` change — coverage TOTAL unchanged).
- Coverage: **>90%** on `app/` (unchanged).
## Completion Criteria
- [ ] No page scroll happens in the streaming path (grep-verifiable + unit-pinned); submit and restore landing still scroll.
- [ ] `uv run pytest` green; the thinking window-pin and all message rendering are byte-identical elsewhere.
@@ -0,0 +1,27 @@
# Task 02 — No-autoscroll E2E (replaces phase 18) + regressions + commit
**Phase:** `42_no_reply_autoscroll` · **Source:** `TODO.md:5` — "Get rid of the chat reply autoscroll, it's breaking things like making it impossible for the user to scroll while a reply generates."
**Story:** `.agent/user_stories/no-reply-autoscroll.md`
## Objective
Prove the inverse of the phase-18 contract in the browser: no streaming autoscroll, submit-reveal and restore-landing intact — then delete the obsolete phase-18 suite and commit.
## Work
1. `tests/e2e/test_no_reply_autoscroll.py` (new) — mock-only, DB up, per the story's Playwright Mapping Rule:
- `test_no_autoscroll_during_long_answer` — `LONG_ANSWER_TRIGGER` question (the mock's ~8 s long answer); once the answer starts streaming, `window.evaluate` a scroll up ~2× the answer's height; sample `window.scrollY` across ≥10 frames (and after `done`): stable within 1px;
- `test_no_autoscroll_during_thinking` — `THINKING_TRIGGER` question; scroll up during the ~4.5 s thinking stream; viewport stable across chunks (no per-chunk page follow);
- `test_submit_reveals_user_message` — in a populated conversation scrolled to the very top, send a question; after send the user's message is in view (its bounding box within the viewport);
- `test_restore_landing_one_shot` — settle a conversation (phase-14 persistence), reload; the page lands on the latest message and stays (no further movement while idle);
- `test_answer_content_intact` — the long answer completes with sources; a thinking turn persists + restores (collapsed block, phase 17).
2. **Delete** `tests/e2e/test_follow_bottom_scroll.py` (its behavior is intentionally removed — owner direction 2026-08-27; the unit pin was rewritten in task 01).
3. Regression pass (isolation runs): `test_chat_rag.py`, `test_thinking_display.py`, `test_chat_persistence.py`, `test_long_answers.py`, `test_smoke.py` — all green; fix only true regressions.
4. `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged; `uv run ruff check . && uv run pyright` clean.
5. Commit (Conventional Commits, `--no-gpg-sign`), e.g. `fix(chat): stop autoscrolling while a reply streams (owner direction)`, staging this phase's files (including the deleted E2E); move `.agent/phases/todo/42_no_reply_autoscroll/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_no_reply_autoscroll.py -v --no-cov` green in isolation.
- Coverage: **>90%** on `app/` (unchanged — frontend-only phase).
## Completion Criteria
- [ ] The new suite passes in isolation; the phase-18 suite is gone; the five regression suites pass in isolation.
- [ ] One atomic `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
@@ -0,0 +1,38 @@
# Phase 43 — Thinking scroll back (user scroll + generate-time autoscroll)
**Source:** `TODO.md` L7 — "Add scrolling back to the thinking block, but have it autoscroll while thinking content is generating."
**Story:** `.agent/user_stories/thinking-scroll-back.md`
**Context:** Phase 17 streams reasoning into the collapsible `.thinking` block with a per-chunk bottom-pin (`textEl.scrollTop = textEl.scrollHeight` in the `thinking` SSE handler). Phase 21 (owner choice 2026-08-24) made `.thinking-text` a no-scroll live tail: `overflow-y: hidden` (the JS pin is the sole scroller). The owner now reverses phase 21: the window is user-scrollable again, and the pin becomes **gated** — follow the tail only while the user is pinned near the window's bottom. This is the window-level successor of the phase-18 pattern (the page-level one is removed in phase 42, which runs first and touches the same `thinking` handler line — order matters).
## Objective
The Thinking block follows its live tail while reasoning is generating **and** the user is at the bottom; a scrolled-up user is never yanked down, and returning to the bottom resumes following.
## Dependencies
- `42_no_reply_autoscroll` (todo) — must run **first**: it strips the page-level scroll from the same `thinking` handler; this phase then reworks the window pin in the cleaned-up handler.
- `17_thinking_display` (complete) — the block, the pin, the auto-collapse on first delta.
- `21_thinking_no_scroll` (complete) — the `overflow-y: hidden` + 320px window being reversed (the 320px clip is kept).
## Tasks
1. `01_window_user_scrollable.md` — CSS: `overflow-y: auto` back, comment replaced (owner direction 2026-08-27).
2. `02_gated_bottom_pin.md` — `app.js`: `THINKING_NEAR_BOTTOM_PX = 32` + gated pin; unit pin rewritten (phase-21 file replaced).
3. `03_thinking_scroll_e2e_and_commit.md` — replace the phase-21 E2E with the new-contract suite + regressions + commit.
## Testing & Quality
- Unit: `tests/unit/test_thinking_no_scroll.py` **deleted**, replaced by `tests/unit/test_thinking_scroll.py` — pins: `overflow-y: auto` + `max-height: 320px` in the `.thinking-text` rule; the 2026-08-27 owner-direction comment; `export const THINKING_NEAR_BOTTOM_PX = 32`; the pin is gated on `isThinkingNearBottom(textEl)` (no unconditional pin).
- Coverage: frontend-only — `app/` TOTAL unchanged, >90%.
- E2E (mandatory, A16): `tests/e2e/test_thinking_scroll.py`, run in isolation. `tests/e2e/test_thinking_no_scroll.py` is **deleted** (behavior intentionally reversed).
## Completion Criteria
- [ ] Wheel/drag/keyboard move `.thinking-text` (frozen-tail state); computed `overflow-y: auto`, `max-height: 320px`.
- [ ] While pinned at the window bottom: each chunk re-pins to the tail (±1px). Scrolled up: no re-pin across chunks. Return to bottom: following resumes.
- [ ] Auto-collapse on first delta, reduced-motion stillness, answer-bubble scroll (phase 11), restored-collapsed block (phase 17) all unchanged.
- [ ] `uv run pytest` green; coverage TOTAL unchanged.
- [ ] `uv run pytest tests/e2e/test_thinking_scroll.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_thinking_display.py`, `test_chat_persistence.py`, `test_no_reply_autoscroll.py`, `test_smoke.py`.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
## Locked decisions
- **Owner direction (2026-08-27, roadmap A2)** reverses the phase-21 owner choice (2026-08-24): the window is user-scrollable again; autoscroll only while pinned near the bottom (32px band). The 320px clip is kept (owner-confirmed).
- **A15 unchanged** — SSE contract untouched; pure client-side.
- **A16/A17 honoured** — one story E2E suite (replacing the removed one), one atomic commit.
@@ -0,0 +1,28 @@
# Task 01 — Window user-scrollable again (CSS + unit pin swap)
**Phase:** `43_thinking_scroll_back` · **Source:** `TODO.md:7` — "Add scrolling back to the thinking block, but have it autoscroll while thinking content is generating."
**Story:** `.agent/user_stories/thinking-scroll-back.md`
## Objective
Restore user scrolling on the Thinking window — `overflow-y: auto`, 320px clip kept, comment updated — and swap the phase-21 unit pins for the new contract so the suite stays green.
## Work
1. `frontend/assets/styles.css` — in the phase-17/21 thinking section, the `details.thinking .thinking-text` rule:
- `overflow-y: hidden;` → `overflow-y: auto;`
- replace the phase-21 comment (*"no user scroll back (owner choice 2026-08-24): the window is a live tail only — the phase-17 JS bottom-pin … is the sole scroller"*) with: *"user-scrollable window (owner direction 2026-08-27, `TODO.md` L7): autoscroll follows the live tail only while the user is pinned near the window's bottom — the phase-17 pin, gated in app.js (task 02: `THINKING_NEAR_BOTTOM_PX`); scrolling up pauses the follow, returning to the bottom resumes it."*
- `max-height: 320px` and **every other declaration in the rule stay byte-identical**; the tightened `p`/`ul` margins rule and the reduced-motion chevron block are untouched.
2. **Delete** `tests/unit/test_thinking_no_scroll.py` (its pins assert the reversed behavior) and create `tests/unit/test_thinking_scroll.py` (house style — source pins, same slicing helpers as the deleted file) with, for now, the CSS contract only:
- the `.thinking-text` rule body contains `overflow-y: auto`, `max-height: 320px`, and the 2026-08-27 owner-direction comment (assert `"owner direction 2026-08-27"` and `"TODO.md L7"`);
- `overflow-y: hidden` / `overflow-y: scroll` are absent from that rule body;
- the phase-17 bottom-pin marker (`textEl.scrollTop = textEl.scrollHeight`) is still present in `app.js` (it becomes gated in task 02 — the pin's existence is asserted now so task 02's diff stays minimal and reviewable).
- The JS-gate pins (`THINKING_NEAR_BOTTOM_PX`, `isThinkingNearBottom`, gated call) are added in task 02 — do not assert them yet.
3. Check `tests/e2e/test_thinking_no_scroll.py` still passes at this checkpoint: it asserts computed `overflow-y: hidden` — **it will fail** (the behavior is intentionally changed). Per the gate, the phase's E2E replacement is task 03; to keep the per-task gate green, **delete** that E2E file in this task as well (its behavior is reversed; task 03 lands the replacement suite). Note the deletion in the final commit message of task 03.
## Testing & Quality
- Unit: `tests/unit/test_thinking_scroll.py` green; full `uv run pytest` green (the deleted E2E file does not run under the unit/integration gate, but the full pytest run must not collect it either — it is gone from the tree).
- Coverage: **>90%** on `app/` (unchanged).
## Completion Criteria
- [ ] `overflow-y: auto` + 320px clip + new comment in the CSS rule; all other declarations byte-identical.
- [ ] Old unit + old E2E phase-21 files deleted; new unit file pins the CSS contract and the surviving pin marker.
- [ ] `uv run pytest` green at this checkpoint.
@@ -0,0 +1,59 @@
# Task 02 — Gated bottom pin (follow while pinned)
**Phase:** `43_thinking_scroll_back` · **Source:** `TODO.md:7` — "Add scrolling back to the thinking block, but have it autoscroll while thinking content is generating."
**Story:** `.agent/user_stories/thinking-scroll-back.md`
## Objective
The phase-17 per-chunk pin becomes a **gate**: the window follows the live tail only while the user is near its bottom; a scrolled-up user is never re-pinned; returning to the bottom re-arms the pin automatically.
## Work
1. `frontend/assets/app.js` —
- add (near the existing `SCROLL` constant, with the phase-18 comment block already removed by phase 42):
```js
/* Thinking-window follow-the-tail contract (owner direction
* 2026-08-27, `TODO.md` L7): the scratchpad autoscrolls to its live
* tail only while the user is pinned near the window's bottom —
* the 32px band is the "window bottom in view" threshold. Scrolling
* up pauses the follow; returning to the bottom resumes it (the
* check runs on every chunk). Exported so the band is unit-pinned
* (same pattern as TURN_TIMEOUT_MS). */
export const THINKING_NEAR_BOTTOM_PX = 32;
function isThinkingNearBottom(textEl) {
return (
textEl.scrollHeight - textEl.scrollTop - textEl.clientHeight <=
THINKING_NEAR_BOTTOM_PX
);
}
```
- in the `thinking` SSE handler, replace the phase-17 block:
```js
if (block.open) {
textEl.scrollTop = textEl.scrollHeight; // pin the stream to the bottom
}
```
(phase 42 already removed the `scrollReveal(wrap)` line there) with the gated pin:
```js
if (block.open && isThinkingNearBottom(textEl)) {
// Follow the live tail only while the user is pinned to the window
// bottom (owner direction 2026-08-27); a scrolled-up reader is
// never re-pinned — returning to the bottom re-arms the pin.
textEl.scrollTop = textEl.scrollHeight;
}
```
- everything else in the handler (acc, sawThinking, clearTurnTimeout, ensureThinkingBlock, `textEl.innerHTML = renderMarkdown(thinkingAcc)`) stays byte-identical.
2. `tests/unit/test_thinking_scroll.py` — extend (from task 01) with the JS pins:
- `app.js` exports `const THINKING_NEAR_BOTTOM_PX = 32`;
- `isThinkingNearBottom` computes `scrollHeight - scrollTop - clientHeight <= THINKING_NEAR_BOTTOM_PX`;
- the thinking handler's pin is gated — the pin line is preceded by `isThinkingNearBottom(textEl)` in the same `if` (assert the combined condition; assert there is **no** unconditional `if (block.open) { textEl.scrollTop = ... }` left);
- `block.open` is still part of the gate (closed blocks never pin);
- the restore path renders collapsed blocks (phase 17) — keep the surviving assertion from task 01.
3. `uv run pytest` green at this checkpoint (E2E not run by the unit gate; the replacement suite lands in task 03).
## Testing & Quality
- Unit: the extended pin file; full suite green.
- Coverage: **>90%** on `app/` (unchanged).
## Completion Criteria
- [ ] The pin fires only when the block is open **and** the window is within 32px of its bottom; scrolled-up users are never re-pinned; the gate re-arms on return (by construction — the check runs per chunk).
- [ ] `uv run pytest` green at this checkpoint.
@@ -0,0 +1,28 @@
# Task 03 — Thinking-scroll E2E (replaces phase 21) + regressions + commit
**Phase:** `43_thinking_scroll_back` · **Source:** `TODO.md:7` — "Add scrolling back to the thinking block, but have it autoscroll while thinking content is generating."
**Story:** `.agent/user_stories/thinking-scroll-back.md`
## Objective
Prove the full contract in the browser — user scroll restored, follow-while-pinned, pause-on-scroll-up, resume-on-return, CSS contract, and the phase-11/17 regressions — then commit the phase.
## Work
1. `tests/e2e/test_thinking_scroll.py` (new) — mock-only, DB up. Reuse the phase-21 determinism machinery (`mock_llm.compose_thinking` is already ~2 700 chars ≈ 4.5 s of paced frames, overflowing the 320px window ~2×; the phase-20 hesitation trigger gives a deterministic 4 s frozen-tail state with the block open). Per the story's Playwright Mapping Rule:
- `test_thinking_window_user_scrollable` — frozen tail: focus `.thinking-text`, wheel up / `Home` / mouse-drag up → `scrollTop` moves and earlier content is visible;
- `test_thinking_window_follows_while_pinned` — live stream: at the window bottom, after the 2nd-to-last and the last chunk the window is pinned to the tail (±1px); the last chunk's text renders inside the visible rectangle;
- `test_thinking_window_stops_on_scroll_up` — mid-stream: scroll up ~half the window; over the next ≥5 chunks `scrollTop` stable (±1px);
- `test_thinking_window_resumes_on_return` — from the paused state, set `scrollTop` to the bottom; on the next chunk the window is re-pinned to the tail (±1px);
- `test_thinking_window_css_contract` — computed `overflow-y: auto`, `max-height: 320px`, `scrollHeight > clientHeight` (real clip);
- `test_answer_bubble_still_scrollable` (phase 11) — long answer: page scrolls, bubble overflow untouched;
- `test_restored_collapsed_thinking_unaffected` (phase 17) — settled thinking turn reloads collapsed with full text.
2. Regression pass (isolation runs): `test_thinking_display.py`, `test_chat_persistence.py`, `test_no_reply_autoscroll.py` (phase 42 — the cleaned `thinking` handler must not have lost the phase-42 contract), `test_smoke.py`.
3. `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged; `uv run ruff check . && uv run pyright` clean.
4. Commit (Conventional Commits, `--no-gpg-sign`), e.g. `feat(chat): thinking window scrolls again, follows the tail only while pinned`, staging this phase's files **including the two deleted phase-21 test files** (`tests/unit/test_thinking_no_scroll.py`, `tests/e2e/test_thinking_no_scroll.py`) and the new unit + E2E files; move `.agent/phases/todo/43_thinking_scroll_back/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_thinking_scroll.py -v --no-cov` green in isolation.
- Coverage: **>90%** on `app/` (unchanged — frontend-only phase).
## Completion Criteria
- [ ] The new suite passes in isolation (all seven tests); the four regression suites pass in isolation.
- [ ] One atomic `--no-gpg-sign` commit covering both deleted and both new test files; phase dir moved to `.agent/phases/complete/`.
@@ -0,0 +1,39 @@
# Phase 44 — Markdown tables (chat, viewer, thinking)
**Source:** `TODO.md` L6 — "Certain markdown formatting isn't working - tables for example don't get rendered as tables in the chat response."
**Story:** `.agent/user_stories/markdown-tables.md`
**Context:** `frontend/assets/markdown.js` is the shared escape-first renderer (no libs, A11): fence protection → escape → inline transforms (`code`, `**bold**`, `*em*`, h1–h3, lists) → paragraph pass → fence restore. It has **no table support** — GFM pipe tables render as one raw `|`-littered paragraph. The renderer serves the chat answer, the document viewer/modal, and the thinking block, so one change covers all three.
## Objective
GFM pipe tables render as semantic, styled, XSS-safe `<table>` elements everywhere the shared renderer runs, with a horizontal-overflow guard for wide tables.
## Dependencies
- `43_thinking_scroll_back` (todo) — sequential only (the thinking block also renders markdown; no shared-file conflict beyond the renderer itself).
- `08_story_dark_tech_theme` (complete) — the palette tokens `.md-table` must use.
- `26_document_modal_viewer` / `10_story_document_viewer` (complete) — the second renderer consumer (viewer/modal).
## Tasks
1. `01_table_renderer_and_styles.md` — table pass in `markdown.js` + `.md-table` CSS.
2. `02_mock_table_trigger.md` — deterministic table answer (incl. a wide table) in `mock_llm.py`.
3. `03_tables_e2e_and_commit.md` — unit pins + story E2E suite + regressions + commit.
## Testing & Quality
- Unit: new `tests/unit/test_markdown_tables.py` — source pins in the house style (regex over `markdown.js` / `styles.css`): the table-protection pass exists and runs **after** the fence pass and **before** the escape pass; cells are escaped + inline-transformed; output carries `class="md-table"`, `<thead>`, `th scope="col"`, and the `.md-table-wrap` wrapper; `styles.css` has the wrapper overflow rule + table borders + reduced-motion-relevant rules. (Behavior is browser-proven by the E2E; unit pins catch silent regressions without a browser — the established frontend pattern.)
- Coverage: frontend-only — `app/` TOTAL unchanged, >90%.
- E2E (mandatory, A16): `tests/e2e/test_markdown_tables.py`, run in isolation.
## Completion Criteria
- [ ] A pipe table in a chat answer renders `<div class="md-table-wrap"><table class="md-table">` with `<thead>`/`<tbody>`, `<th scope="col">` headers, correct cell texts; no raw `|---|` in the bubble.
- [ ] A wide table scrolls inside its wrapper; the 46rem column does not overflow the page.
- [ ] XSS-safe (escaped cells), fences win over tables, lone pipes stay text.
- [ ] The document viewer/modal renders the same table for a fixture document containing one.
- [ ] `uv run pytest` green; coverage TOTAL unchanged.
- [ ] `uv run pytest tests/e2e/test_markdown_tables.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_chat_rag.py`, `test_document_viewer.py`, `test_document_summaries.py`, `test_smoke.py`.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
## Locked decisions
- **A11 untouched** — still the local ~90-line renderer, no library, no CDN.
- **Owner-locked (2026-08-27, roadmap A3):** scope = GFM pipe tables (header + separator + body); links/blockquotes/hr out of scope; alignment colons parsed but rendered left; wide tables get the `overflow-x: auto` wrapper.
- **A16/A17 honoured** — one story E2E suite, one atomic commit.
@@ -0,0 +1,40 @@
# Task 01 — Table pass in the shared renderer + styles
**Phase:** `44_markdown_tables` · **Source:** `TODO.md:6` — "Certain markdown formatting isn't working - tables for example don't get rendered as tables in the chat response."
**Story:** `.agent/user_stories/markdown-tables.md`
## Objective
`renderMarkdown` turns GFM pipe-table blocks into semantic tables (XSS-safe, inline markdown in cells), wrapped in a horizontal-overflow container, styled in the dark-tech palette.
## Work
1. `frontend/assets/markdown.js` — in `renderMarkdown(md)`, between step 1 (fence protection) and step 2 (escape + inline transforms), add a **table protection pass** using the same placeholder mechanism as the fences:
- **Detection** (line-oriented over the fence-protected text): a **table block** starts at a line containing `|` whose **next** line is a **separator** — the separator line consists of ≥1 pipe-separated cells, each matching `^\s*:?-+:?\s*$` (allowing a leading/trailing pipe and inter-cell whitespace). The block then extends over every following line that still contains `|` (the body; zero body rows is a valid table — header only). A maximal such block is one table. Anything else (a single `|` in prose, a separator with no `|`-header line above it, a 1-line "table") is left untouched.
- **Extraction:** for each table run, split each line on `|`, drop the leading/trailing empty entries produced by leading/trailing pipes, `trim()` each cell.
- **Cell rendering:** each cell goes through the same inline pipeline as the rest of the text — `escapeHtml(cell)` first (XSS-safe, invariant of the renderer), then the inline transforms (`` `code` ``, `**bold**`, `*em*` — the exact same `.replace` chain step 2 uses; factor the inline chain into a small local helper if it makes the cell path cleaner, keeping the whole-text path byte-identical in output).
- **Assembly:**
```html
<div class="md-table-wrap">
<table class="md-table">
<thead><tr><th scope="col">h1</th>…</tr></thead>
<tbody><tr><td>…</td>…</tr>…</tbody>
</table>
</div>
```
Rows with fewer cells than the header are padded with empty `<td>`; rows with more are truncated to the header width (defensive — the mock and real answers are well-formed). Alignment colons in the separator are **parsed but ignored** (all cells left — owner decision).
- **Placeholders:** reuse the `\u0000CODEn\u0000` array pattern — e.g. push the table HTML into a second array and emit `\u0000TABLEn\u0000`, restored alongside the code blocks in step 4 (update the restore step accordingly; tables inside the protected span are already final HTML — they must not re-enter the paragraph pass, which the placeholder guarantees).
- Update the file's header comment (the ~60-line no-CDN renderer now also does tables — 2026-08-27, `TODO.md` L6).
2. `frontend/assets/styles.css` — near the markdown/content styling (the chat bubble content rules):
- `.md-table-wrap { overflow-x: auto; }` — the wrapper is the scroller;
- `.md-table { border-collapse: collapse; width: 100%; font-size: 0.9rem; }`;
- `.md-table th, .md-table td { border: 1px solid var(--line); padding: 0.4rem 0.6rem; text-align: left; vertical-align: top; }`;
- `.md-table thead th { background: <surface-darker token>; color: var(--ink); }` — pick the existing token that keeps ≥4.5:1 (PLAN §7.2: ink `#e8ebf4` on surface `#121a2e` is 14.5:1 — use the plain surface family, not brand);
- ensure the rule set is inside or consistent with the reduced-motion constraints (no animation involved — nothing to still).
3. Sanity: run an existing markdown-consuming E2E (e.g. `test_chat_rag.py`) to confirm byte-identical output for non-table content (the inline-chain factor, if done, must not change any existing rendering).
## Testing & Quality
- Unit: `tests/unit/test_markdown_tables.py` (new) source pins per the phase overview (pass ordering, escape-first for cells, output markers, CSS rules). Full suite green.
- Coverage: **>90%** on `app/` (unchanged — frontend-only).
## Completion Criteria
- [ ] `renderMarkdown` handles the shapes in the story's acceptance criteria 1–4 (table, XSS cell, fence-wins, non-tables stay text) — verifiable via the unit pins now and the E2E in task 03.
- [ ] No existing rendering changes for non-table markdown (regression suite from step 3 green).
@@ -0,0 +1,40 @@
# Task 02 — Deterministic table answer in the mock
**Phase:** `44_markdown_tables` · **Source:** `TODO.md:6` — "Certain markdown formatting isn't working - tables for example don't get rendered as tables in the chat response."
**Story:** `.agent/user_stories/markdown-tables.md`
## Objective
The E2E mock serves a byte-stable table answer (plus a deliberately wide table and an XSS cell) on demand, following the existing trigger convention.
## Work
1. `tests/e2e/mock_llm.py` —
- add `TABLE_TRIGGER = "show me a table"` (same case-insensitive-substring convention as `LONG_ANSWER_TRIGGER` / `THINKING_TRIGGER` / `TOOLS_TRIGGER`);
- in `compose_answer(body)`, **before** the default tail-echo branch (and before `DEFLECT_MODE` — a deflection prompt never carries the marker, same reasoning as `SUMMARY_MODE`): when the trigger is in the lowercased user message, return the fixed table answer:
```
Here's the shape, in a table:
| Service | Port | Host |
|---|---|---|
| Caddy | 80 | homelab-gw |
| GitLab | 8929 | homelab-git |
| ntfy | 2087 | homelab-ntfy |
<img src=x onerror=alert(1)>
And the wide one:
| A very long column header to force overflow | Second column with some padding text | Third column | Fourth | Fifth |
|---|---|---|---|---|
| value-one | value-two | value-three | value-four | value-five |
```
(The `<img onerror>` line is the XSS assertion's payload — it must survive the mock byte-for-byte so the E2E can prove the renderer neutralizes it; the wide table guarantees `scrollWidth > clientWidth` inside the 46rem column.)
- keep the answer a plain grounded response (no `DEFLECT_MODE` interplay): the trigger question is asked against an on-topic fixture so the honesty gate is HIGH in the E2E (the suite asserts non-deflection as part of the table test).
2. Update the module docstring's marker list (the file documents every trigger — add the table row).
3. `uv run pytest tests/e2e/mock_llm.py-related unit tests` — run `uv run pytest tests/unit -k "mock" tests/integration -x` (or the mock's existing test file, if any — check `tests/` for mock-specific tests) to prove the new branch breaks no existing flow; the full suite is green (the new branch only fires on the marker).
## Testing & Quality
- Unit/integration: full suite green; the new branch is covered by the E2E (task 03) — if a mock-level unit test file exists, add the table case there so the branch is unit-covered too.
- Coverage: **>90%** on `app/` (mock lives in `tests/` — the gate is unchanged).
## Completion Criteria
- [ ] `TABLE_TRIGGER` returns the fixed table answer (byte-stable), including the XSS line and the wide table; no existing mock behavior changes for marker-less requests.
@@ -0,0 +1,28 @@
# Task 03 — Tables E2E + regressions + commit
**Phase:** `44_markdown_tables` · **Source:** `TODO.md:6` — "Certain markdown formatting isn't working - tables for example don't get rendered as tables in the chat response."
**Story:** `.agent/user_stories/markdown-tables.md`
## Objective
Prove the table contract in the browser — chat, overflow, XSS, viewer, and the two "not a table" regressions — then commit the phase.
## Work
1. `tests/e2e/test_markdown_tables.py` (new) — mock-only, DB up, per the story's Playwright Mapping Rule:
- `test_chat_table_renders` — ask an on-topic question containing `TABLE_TRIGGER` (pick a fixture topic that retrieves HIGH — reuse a question pattern from `test_chat_rag.py`); the brain bubble contains `<div class="md-table-wrap"><table class="md-table">`, a `<thead>` with three `<th scope="col">` (Service/Port/Host), the body cell texts ("Caddy", "8929", …), and **no** `|---|` separator text in the bubble;
- `test_wide_table_scrolls` — in the same answer, the wide table's wrapper has `scrollWidth > clientWidth`; horizontal scrolling (wheel/`scrollLeft`) moves it; the page itself has no horizontal overflow (`document.documentElement.scrollWidth <= clientWidth`);
- `test_table_xss_safe` — the `<img src=x onerror=…>` line renders as visible text (no `<img>` element inside the bubble; `onerror` can never fire — assert `page.evaluate` found zero injected img nodes and the tag text is present);
- `test_viewer_table_renders` — add a fixture document (extend `tests/fixtures/docs/homelab/` with a small `.md` file containing a pipe table — e.g. `tables.md` with a 3×3 table; re-import per the `test_document_documents.py`/`test_import_documents.py` fixture pattern), open it from the Sources table (admin) in the modal; the modal content renders `<table class="md-table">`;
- `test_fence_not_a_table` — a question/fixture whose content puts `|`-heavy lines inside a ``` fence (existing fixtures have fenced blocks — pick/extend one) renders `<pre><code>` with no `<table>`;
- `test_plain_pipe_stays_text` — an off-trigger grounded answer containing a single `|` in prose (assert via an existing deterministic answer or a minimal new fixture) renders as text, no `<table>`.
- Assert non-deflection (`.is-deflected` absent) in the table tests — the honesty gate interplay is part of the contract.
2. Regression pass (isolation runs): `test_chat_rag.py`, `test_document_viewer.py`, `test_document_summaries.py` (the renderer is shared — summaries render through it too), `test_smoke.py`.
3. `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged; `uv run ruff check . && uv run pyright` clean.
4. Commit (Conventional Commits, `--no-gpg-sign`), e.g. `feat(chat): render markdown tables in answers, viewer, and thinking`, staging this phase's files; move `.agent/phases/todo/44_markdown_tables/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_markdown_tables.py -v --no-cov` green in isolation.
- Coverage: **>90%** on `app/` (unchanged — frontend-only phase).
## Completion Criteria
- [ ] The story E2E suite passes in isolation (all six tests); the four regression suites pass in isolation.
- [ ] One atomic `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
@@ -0,0 +1,42 @@
# Phase 45 — Agent makes as many tool calls as it wants
**Source:** `TODO.md` L8 — "Allow the LLM to make as many tool calls as it wants, remove the restrictions, they're causing problems getting correct answers"
**Story:** `.agent/user_stories/agent-unlimited-tools.md`
**Context:** Phase 37 shipped the grounded-turn agent loop (`app/rag/agent.py::run_agent`) with per-turn budgets — `agent_list_calls` / `agent_read_calls` (default 1 each, `BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS`), "budgets-as-kill-switch" locked decision. The exhaustion refusals (`LIST_EXHAUSTED` / `READ_EXHAUSTED`) are where correct multi-document answers die. Owner direction (2026-08-27): remove both budgets; the loop keeps one guard — a configurable **round cap** that also doubles as the no-tools kill switch (`0`).
## Objective
`list_documents` / `read_document` can be called as many times as the model needs (re-lists included), bounded only by `BOR_AGENT_MAX_ROUNDS` (default 10; `0` = no tools, byte-identical to the pre-phase-37 path).
## Dependencies
- `44_markdown_tables` (todo) — sequential only (no shared files: this phase is `app/` + tests + mock).
- `37_agent_document_tools` (complete) — the loop, the `tool` SSE event, the UI tool lines, the per-turn `tool_calls=N` log field, and the phase-37 locked decision being revised.
- `31_kb_overview_prompt` (complete) — the `lite` one-shot path is untouched by this phase.
## Tasks
1. `01_config_round_cap.md` — the server core, atomically: `agent_max_rounds` replaces the budgets in `app/config.py` + `app/rag/agent.py`, unit + integration rewrites, `.env.example` (one task so the per-task gate stays green).
2. `02_mock_multi_read_flow.md` — the E2E mock's deterministic multi-read (list → read #1 → read #2 → answer) flow.
3. `03_unlimited_tools_e2e_and_commit.md` — story E2E + phase-37 regression + PLAN.md revision note + commit.
## Testing & Quality
- Unit: `tests/unit/test_agent.py` rewritten around the round cap (always-calling mock LLM: N tool rounds then a forced `tools=None` final answer; `max_rounds=0` → exactly one request with `tools=None`; rejected-call spam — unknown tool / already-in-context — is bounded by the cap, not by budgets; re-lists execute and count in `tool_calls`); `tests/unit/test_config.py` (default 10, `BOR_AGENT_MAX_ROUNDS` override, `0`, the budget env vars are gone).
- Integration: `tests/integration/test_chat_api.py` — the `agent_list_calls=0, agent_read_calls=0` fixtures become `agent_max_rounds=0`; the tool SSE event shape and the `done.sources` extension assertions stay.
- Coverage: **>90%** on `app/` — `agent.py` + `config.py` fully covered.
- E2E (mandatory, A16): `tests/e2e/test_agent_unlimited_tools.py`, run in isolation.
## Completion Criteria
- [ ] `BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` are gone (config, `.env.example`, agent, tests); no exhaustion refusal remains.
- [ ] A multi-read turn (list + 2 reads) streams three tool lines, answers non-deflected, and `done.sources` lists the retrieved doc(s) + both reads deduped.
- [ ] `agent_max_rounds=0` → single `tools=None` request (kill switch); at the cap the loop forces a final no-tools answer (log warning kept).
- [ ] `tool` SSE event shape and `tool_calls=N` per-turn log field unchanged.
- [ ] `.agent/PLAN.md` carries the phase-37 revision note (owner permission 2026-08-27, `TODO.md` L8) — the only PLAN edit in this phase.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
- [ ] `uv run pytest tests/e2e/test_agent_unlimited_tools.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_agent_document_tools.py`, `test_chat_rag.py`, `test_smoke.py`.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
## Locked decisions
- **Owner-locked revision (2026-08-27, roadmap R2):** the phase-37 "budgets-as-kill-switch" decision is **revised** — both per-tool budgets removed; `BOR_AGENT_MAX_ROUNDS` (default 10) is the only loop guard and the kill switch (`0`). Recorded as a PLAN.md revision note (the established owner-permission pattern, like the A10/A7/A9/A15 notes) — a recorded revision, not a silent deviation (AGENTS.md rule 3).
- **A15 extension unchanged** — the `tool` SSE event shape, the `done` shape, and the per-turn log line (`tool_calls=N`) are untouched; the revision note amends the phase-37 note's budget wording only.
- **Rejections kept:** `Unknown tool.`, `MISSING_READ_ARGS`, `Already in your context.` (non-budget rejections; the cap bounds their pathological repetition).
- **A17 honoured** — one atomic commit.
@@ -0,0 +1,52 @@
# Task 01 — Server core: round cap replaces the budgets (config + loop + unit/integration)
**Phase:** `45_agent_unlimited_tools` · **Source:** `TODO.md:8` — "Allow the LLM to make as many tool calls as it wants, remove the restrictions, they're causing problems getting correct answers"
**Story:** `.agent/user_stories/agent-unlimited-tools.md`
## Objective
One coherent server-side change, landed atomically so the suite is green at the checkpoint: `agent_max_rounds` (`BOR_AGENT_MAX_ROUNDS`, default 10; `0` = no tools) replaces both per-tool budgets in config, the agent loop, and every test that pins them.
## Work
1. `app/config.py` —
- **delete** the `agent_list_calls` and `agent_read_calls` fields (with docstrings);
- **add** in their place, same "RAG tuning" section:
```python
#: Hard cap on the agent tool rounds per grounded turn (phase 45,
#: revising phase 37's per-tool budgets — owner permission
#: 2026-08-27, TODO L8: "allow the LLM to make as many tool calls
#: as it wants"). Every tool call the model emits consumes a
#: round; at the cap the loop forces one final no-tools answer.
#: ``0`` disables the tools entirely — the turn is a single
#: request with ``tools=None`` (the pre-phase-37 path — the kill
#: switch).
agent_max_rounds: int = 10
```
- optional: a `field_validator` rejecting negative values (note it in the docstring if added).
2. `app/rag/agent.py` —
- `run_agent`: `max_rounds = settings.agent_max_rounds`; `tools = AGENT_TOOLS if max_rounds > 0 else None` (the kill switch — at 0 the loop makes exactly one request with `tools=None`, byte-identical to the pre-phase-37 path);
- delete `list_left` / `read_left` and the budget-driven `tools = None if (list_left == 0 and read_left == 0) else AGENT_TOOLS` transition — `tools` stays `AGENT_TOOLS` while rounds remain;
- after each executed call: `rounds += 1`; the existing cap branch becomes the **only** forced-exit: `if rounds >= max_rounds:` → the `logger.warning("agent round cap reached …")` + final `chat_stream(messages, tools=None)` (update the warning text: it is no longer belt-and-braces — it is the cap);
- `_execute_tool(db, call, seed_docs, holder)`: drop the `list_left` / `read_left` parameters and the `LIST_EXHAUSTED` / `READ_EXHAUSTED` early returns; keep the `ALREADY_IN_CONTEXT`, `UNKNOWN_TOOL`, `MISSING_READ_ARGS` rejections (non-budget — a repeated rejected call still consumes a *round* in the loop, so a pathological stream is bounded by `max_rounds`); return type simplifies to `str`;
- delete the `LIST_EXHAUSTED` / `READ_EXHAUSTED` constants;
- `AGENT_TOOLS`: `read_document` description "Add the full content of exactly one more indexed document to your context" → "Add the full content of one more indexed document to your context";
- module docstring: the budget paragraph (points 1, 3, 4) rewritten for the round cap (owner revision 2026-08-27, `TODO.md` L8); `run_agent` docstring updated (`seed_docs` note unchanged); `AgentHolder` unchanged (`tool_calls` still counts executed calls — now including re-lists);
- the per-call `logger.info("agent tool=… budget list_left=… read_left=…")` line becomes `logger.info("agent tool=%s args=%s round=%d/%d", …)` (or equivalent — the per-turn `tool_calls=N` field in `app/api/chat.py` is untouched).
3. `tests/unit/test_agent.py` — **rewrite** the budget tests around the round cap (keep the file's fake-LLM harness):
- an always-`list_documents`-calling mock with `agent_max_rounds=3`: exactly 3 tool rounds execute, then one forced `tools=None` request streams the answer; `holder.tool_calls == 3`;
- `agent_max_rounds=0`: exactly one request, `tools=None`, no tool lines, `holder.tool_calls == 0` (kill switch);
- an always-`read_document`-with-unknown-path mock (every call rejected — `No document at …`): the loop runs to `max_rounds` and forces the final answer (rejections no longer end the loop early via budgets, the cap bounds them);
- the existing rejections tests (`Unknown tool.`, `MISSING_READ_ARGS`, `Already in your context.`) keep passing — update their `_settings(...)` calls (`agent_max_rounds=…` instead of the budget kwargs);
- a **re-list** test: `list_documents` called twice in one turn executes both (the second returns the catalog again) and counts 2 in `holder.tool_calls`.
4. `tests/unit/test_config.py` — default 10; `BOR_AGENT_MAX_ROUNDS=0` / `=5` overrides; (negative validator, if added); delete the old budget assertions.
5. `tests/integration/test_chat_api.py` — the `agent_list_calls=0, agent_read_calls=0` fixture kwargs (~line 661) become `agent_max_rounds=0`; any other budget kwarg in the file the same; the tool SSE-event and `done.sources` assertions stay untouched.
6. `.env.example` — the two `BOR_AGENT_*_CALLS` lines become one: `# BOR_AGENT_MAX_ROUNDS=10 # hard cap on agent tool rounds per turn (0 = no tools)` (file's optional-setting comment style).
7. Grep the repo for `agent_list_calls|agent_read_calls|BOR_AGENT_(LIST|READ)_CALLS|LIST_EXHAUSTED|READ_EXHAUSTED` — zero hits outside `.agent/phases/complete/**` (history).
## Testing & Quality
- Unit + integration: full `uv run pytest` green at this checkpoint (the mock/E2E multi-read flow lands in task 02 — the existing 3-step mock flow still works unmodified, so `test_agent_document_tools.py` E2E is not yet run by the gate).
- Coverage: **>90%** on `app/` — `agent.py` + `config.py` fully covered by the rewritten tests.
## Completion Criteria
- [ ] No per-tool budgets anywhere in `app/` or `tests/`; `agent_max_rounds` is the single knob (default 10, `0` = kill switch).
- [ ] Re-lists execute; non-budget rejections intact; the cap bounds pathological streams; `tool_calls=N` log field and `tool` SSE event unchanged.
- [ ] `uv run pytest` + coverage gate green at this checkpoint.
@@ -0,0 +1,27 @@
# Task 02 — Mock: deterministic multi-read tool flow
**Phase:** `45_agent_unlimited_tools` · **Source:** `TODO.md:8` — "Allow the LLM to make as many tool calls as it wants, remove the restrictions, they're causing problems getting correct answers"
**Story:** `.agent/user_stories/agent-unlimited-tools.md`
## Objective
The E2E mock gains a deterministic **multi-read** agent flow (list → read #1 → read #2 → answer) so "as many tool calls as it wants" is provable statelessly, without disturbing the existing 3-step flow.
## Work
1. `tests/e2e/mock_llm.py` —
- the existing phase-37 flow (documented in the module docstring and `_tool_flow`): marker `TOOLS_TRIGGER` ("use your tools") + `<tools>` system section → step classification **statelessly from the messages**: no tool results yet → `list`; one `tool`-role message with the catalog prefix → `read` (first catalog doc, parsed from the listing via the `rsplit("/", 1)` convention); one `tool`-role message with the `Document <source/path>:` prefix → forced answer.
- add a **multi-read variant**: when the user message contains **both** `TOOLS_TRIGGER` and a new marker `MULTI_READ_TRIGGER = "read two documents"`, the classifier reads the *count* of `tool`-role messages whose content starts with `"Document "` (the read-result prefix, `app.rag.agent`'s `_execute_tool` output):
- 0 read results (+ no catalog yet) → `list`;
- 0 read results (catalog present) → `read` the **first** catalog doc;
- 1 read result → `read` the **second** catalog doc (the listing minus the already-read doc — parse the catalog lines the same way the existing read step does, skipping the path already read);
- 2 read results → forced answer: the existing answer shape (tail echo) plus a deterministic line naming **both** read paths (e.g. `"I read <path1> and <path2>."` — byte-stable) so the E2E can assert the model actually used both;
- the single-read flow (no `MULTI_READ_TRIGGER`) stays byte-identical — the variant must be a strict superset (the existing `test_agent_document_tools.py` E2E keeps passing unmodified).
- update the module docstring's tool-flow documentation (the multi-read steps + the marker).
2. `uv run pytest` green (mock-only change; the existing 3-step E2E is not run by the unit gate but must stay conceptually intact — the regression run in task 03 proves it).
## Testing & Quality
- Unit: full suite green; if a mock-specific unit test file exists (check `tests/unit/`), add the multi-read classification case there (catalog → read #1 → read #2 → answer) so the new branch is unit-covered; otherwise the E2E (task 03) covers it.
- Coverage: **>90%** on `app/` (unchanged — `tests/` only).
## Completion Criteria
- [ ] `TOOLS_TRIGGER` + `MULTI_READ_TRIGGER` → deterministic 4-step flow (list, read #1, read #2, answer naming both paths); the 3-step flow is unchanged for marker-less turns.
- [ ] Full unit/integration suite green.
@@ -0,0 +1,39 @@
# Task 03 — Unlimited-tools E2E + PLAN revision note + commit
**Phase:** `45_agent_unlimited_tools` · **Source:** `TODO.md:8` — "Allow the LLM to make as many tool calls as it wants, remove the restrictions, they're causing problems getting correct answers"
**Story:** `.agent/user_stories/agent-unlimited-tools.md`
## Objective
Prove the multi-tool turn end to end, record the phase-37 decision revision in PLAN.md, run the regressions, and commit the phase.
## Work
1. `tests/e2e/test_agent_unlimited_tools.py` (new) — mock-only, DB up (the grounded-turn prerequisite: the fixture KB imported, per the `test_agent_document_tools.py` fixture pattern):
- `test_multi_read_turn` — a grounded question carrying `TOOLS_TRIGGER` + `MULTI_READ_TRIGGER`: the turn streams **three** tool lines (`.tool-call` rows: one `list_documents` — "is listing documents" — and two `read_document` — "is reading <source/path>") in order, then a final non-deflected answer containing the mock's "I read <path1> and <path2>." line;
- `test_done_sources_include_reads` — the source chips under the answer list the retrieval doc(s) **plus both** read documents, deduped (the phase-37 `done.sources` extension contract, now with 2 reads);
- `test_relist_allowed` — the listing tool ran without a "No listing budget left" refusal: assert no refusal text anywhere in the bubble/tool lines (the old refusal strings must be gone — grep the app for them is task 01's job; here assert the UI never shows one);
- `test_single_tool_flow_regression` (phase 37) — the original 3-step flow (marker without the multi-read trigger) still answers after exactly one read with its single tool pair (this may be a targeted re-assertion; the full suite `test_agent_document_tools.py` runs in the regression pass).
2. `.agent/PLAN.md` — **the only PLAN edit in this phase** (owner-locked revision, roadmap R2): in the §4 SSE-revision block, after the phase-37 revision note, add a new note in the established style:
> **SSE revision (phase 45, owner permission 2026-08-27):** the phase-37
> per-turn tool budgets are **removed** (owner: "allow the LLM to make
> as many tool calls as it wants — `TODO.md` L8): `BOR_AGENT_LIST_CALLS`
> / `BOR_AGENT_READ_CALLS` no longer exist; `BOR_AGENT_MAX_ROUNDS`
> (default 10) caps the tool rounds and `0` disables the tools
> entirely (the pre-phase-37 path). The `tool` event shape and the
> `done` shape are unchanged — a recorded revision of the phase-37
> note's budget wording, not a silent deviation.
Also update the phase-37 note's budget clause if it reads as current
truth ("budgeted by `BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS`")
by appending "(removed in phase 45 — see the revision note below)".
Touch **nothing else** in PLAN.md (Protocol B: no roadmap-table edit for appended phases).
3. Regression pass (isolation runs): `test_agent_document_tools.py` (phase 37 — must pass **unmodified**), `test_chat_rag.py`, `test_smoke.py`.
4. `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
5. Commit (Conventional Commits, `--no-gpg-sign`), e.g. `feat(rag): unbounded agent tool calls behind a round cap (owner revision)`, staging this phase's files **including the force-added `.agent/PLAN.md`** (AGENTS.md rule 8: `git add -f .agent/PLAN.md`) and the phase dir move `.agent/phases/todo/45_agent_unlimited_tools/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_agent_unlimited_tools.py -v --no-cov` green in isolation.
- Coverage: **>90%** on `app/` (task 01's rewritten tests carry it).
## Completion Criteria
- [ ] The multi-read E2E suite passes in isolation; the phase-37 suite passes unmodified in isolation.
- [ ] PLAN.md carries the phase-45 revision note (owner permission 2026-08-27) and nothing else changed.
- [ ] One atomic `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
@@ -0,0 +1,40 @@
# Phase 46 — Mobile hamburger nav
**Source:** `TODO.md` L9 — "The navbar on mobile is way too squished. Make it a hamburger dropdown menu with a nice animation"
**Story:** `.agent/user_stories/mobile-hamburger-nav.md`
**Context:** The shared bar (phase 19/34 — the same header block on all six pages) carries brand + up to four text nav pills (Chat, `#nav-sources`, `#nav-git-sources`, `#nav-tuning` — the latter three ship `hidden`, revealed for admin by `header.js`) + four action controls (steering toggle, `#sync-btn`, `#new-chat-btn`, sign in/out). At ≤640px the phase-34/35 squeeze rules (0.72rem pills, 0.05rem gaps) leave a bar that is squished and hard to hit. The fix: on mobile the nav links move into an animated hamburger dropdown; the action pills stay in the bar.
## Objective
At ≤640px the nav links live in a `#nav-toggle`-opened dropdown menu (slide+fade, reduced-motion-still) with comfortable targets and the same auth visibility; at >640px the bar is byte-identical to today.
## Dependencies
- `45_agent_unlimited_tools` (todo) — sequential only (no shared files).
- `34_consistent_navbar` / `35_git_sources_admin` (complete) — the six-page bar contract, the admin-only link reveal, and the mobile squeeze rules being superseded.
- `07_story_responsive_polish` (complete) — the ≤640px conventions (44px targets, safe areas).
## Tasks
1. `01_hamburger_markup_all_pages.md` — `#nav-toggle` + `id="app-nav"` on all six pages; the mobile CSS (dropdown, animation, reduced-motion).
2. `02_toggle_behavior.md` — `header.js` open/close behavior (aria, Esc, link-close, resize-close) + source pins.
3. `03_hamburger_e2e_and_commit.md` — story E2E suite (mobile + desktop + reduced motion) + regressions + commit.
## Testing & Quality
- Unit: new `tests/unit/test_hamburger_nav.py` — `#nav-toggle` (with `aria-controls="app-nav"`, `aria-expanded`, `aria-label="Menu"`) present in **all six** pages and absent-visible on desktop (CSS `display: none` outside the media query); `<nav class="app-nav" id="app-nav">` in all six; the mobile CSS block carries the dropdown rules + `.is-open` state + the 180ms transition + the reduced-motion override; the old nav-pill squeeze rules are gone/superseded; `header.js` carries the toggle binding (click, Esc, delegated link close, matchMedia close).
- Coverage: frontend-only — `app/` TOTAL unchanged, >90%.
- E2E (mandatory, A16): `tests/e2e/test_mobile_hamburger_nav.py`, run in isolation.
## Completion Criteria
- [ ] 375px: hamburger visible (44px target), inline nav hidden, no horizontal bar overflow; menu opens with animation, closes via link/Esc/outside; anonymous sees only "Chat" in the menu, admin sees all four links.
- [ ] `reducedMotion: "reduce"`: no transition, open/close still instant and correct.
- [ ] >640px: no hamburger, inline pills exactly as today (phase-34/35 contract intact).
- [ ] `uv run pytest` green; coverage TOTAL unchanged.
- [ ] `uv run pytest tests/e2e/test_mobile_hamburger_nav.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_nav_consistency.py`, `test_header_consistency.py`, `test_shared_header.py`, `test_responsive_polish.py`, `test_tuning_nav_link.py`.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] UI Structure Check (AGENTS.md rule 5): the toggle is a labeled 44px button with `aria-expanded`/`aria-controls`; the menu keeps `nav aria-label="Primary"`; focus-visible on the new control; contrast ≥4.5:1; no CDN.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
## Locked decisions
- **Owner-locked (2026-08-27, roadmap A5):** nav links only in the menu; action pills stay in the bar; slide-down + fade 180ms; `prefers-reduced-motion` stills it; the existing ≤640px breakpoint (no new one).
- **A11 untouched** — no new assets/CDN; the menu reuses the existing `<nav>` element (no duplicated links, so the whoami reveal keeps working unchanged).
- **Phase-34 contract kept** — the same bar on every page; the auth visibility rules apply inside the menu exactly as before.
- **A16/A17 honoured** — one story E2E suite, one atomic commit.
@@ -0,0 +1,42 @@
# Task 01 — Hamburger markup (six pages) + mobile dropdown CSS
**Phase:** `46_mobile_hamburger_nav` · **Source:** `TODO.md:9` — "The navbar on mobile is way too squished. Make it a hamburger dropdown menu with a nice animation"
**Story:** `.agent/user_stories/mobile-hamburger-nav.md`
## Objective
All six pages carry the identical `#nav-toggle` button + `id="app-nav"`, and the ≤640px stylesheet turns the nav into an animated full-width dropdown — desktop untouched.
## Work
1. **Markup — all six pages** (`frontend/index.html`, `sources.html`, `document.html`, `git-sources.html`, `login.html`, `tuning.html`), each in its `<header class="app-header">` block:
- insert the toggle button **immediately before** the `<nav>`:
```html
<!-- Phase 46 (owner permission 2026-08-27, `TODO.md` L9): the
mobile hamburger — visible ≤640px only (CSS); opens the nav as
an animated dropdown. Behavior: assets/header.js. -->
<button type="button" class="nav-toggle" id="nav-toggle"
aria-expanded="false" aria-controls="app-nav" aria-label="Menu">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"><path d="M4 7h16M4 12h16M4 17h16"/></svg>
</button>
```
- give the existing nav the id: `<nav class="app-nav" id="app-nav" aria-label="Primary">` (every other attribute/child byte-identical — the links keep their `hidden` attributes; **no links duplicated**).
- keep the six pages visually/structurally identical (the phase-34 contract) — the toggle is part of the shared bar block, positioned the same on every page.
2. `frontend/assets/styles.css` —
- **global (outside media queries):** `.nav-toggle { display: none; }` (desktop: absent);
- **inside the existing `@media (max-width: 640px)` block:**
- `.nav-toggle { display: inline-flex; align-items: center; justify-content: center; width: 44px; height: 44px; padding: 0; color: var(--ink); background: none; border: 0; border-radius: var(--radius-sm); cursor: pointer; }` + `:focus-visible` inherits the global 3px outline + a hover state matching the other pills (the `.steering-toggle:hover` family);
- `.app-nav` becomes the dropdown: `position: absolute; top: 100%; left: 0; right: 0; flex-direction: column; gap: 0; background: var(--surface); border-bottom: 1px solid var(--line); box-shadow: <existing shadow token or 0 8px 24px rgba(0,0,0,.4)>; padding: 0.5rem 0; z-index: <above the header content — check the header's z-index and use header+1>;` — note the containing block is the sticky `.app-header` (`.header-inner` is not positioned), so the menu spans the header's full width, edge to edge — intended on mobile;
- **closed state (default):** `visibility: hidden; opacity: 0; transform: translateY(-8px); pointer-events: none; transition: opacity 180ms ease, transform 180ms ease, visibility 0s linear 180ms;`
- **open state:** `.app-nav.is-open { visibility: visible; opacity: 1; transform: none; pointer-events: auto; transition: opacity 180ms ease, transform 180ms ease, visibility 0s; }`
- **menu rows:** `.app-nav .nav-link { padding: 0.75rem 1.25rem; font-size: 1rem; }` (comfortable 44px+ targets, readable — this **supersedes** the ≤640px pill-squeeze rules for `.nav-link` and `.app-nav` gap in that block: delete/replace the `.nav-link { padding: 0.3rem 0.25rem; font-size: 0.72rem; }` and `.app-nav { gap: 0.05rem; }` rules, keeping the rest of the block);
- **reduced motion:** inside the file's existing `@media (prefers-reduced-motion: reduce)` block (the one covering the 640px rules — or a new one after it): `.app-nav { transition: none; }`;
- the **900px tablet block is untouched** (inline nav still in use at 641–900px); the action-pill rules in the 640px block are untouched; the header height (`--header-h: 58px`) is untouched.
- verify at 360px: brand (clipped clean as today) + hamburger + the four icon action pills fit without horizontal overflow (the old four text pills are gone from the bar — there is now room; if the bar still overflows, the brand ellipsis target absorbs it exactly as before).
## Testing & Quality
- Unit: the new `tests/unit/test_hamburger_nav.py` starts here with the markup + CSS pins (the JS pins land in task 02): toggle markup (attributes) in all six pages; `id="app-nav"` in all six; `.nav-toggle { display: none }` outside media queries; the mobile block carries the dropdown, `.is-open`, the 180ms transition pair, the reduced-motion override, and the superseded squeeze rules are gone; full suite green (behavior not yet wired — the menu is closed by default and CSS-inert without JS, so no E2E regressions at this checkpoint).
- Coverage: **>90%** on `app/` (unchanged).
## Completion Criteria
- [ ] Six identical toggles + `id="app-nav"`; desktop rendering byte-identical (`.nav-toggle` hidden, nav inline as before).
- [ ] Mobile: closed dropdown is invisible and non-interactive; `.is-open` (added by task 02's JS) will be the only opener.
- [ ] Full suite green at this checkpoint.
@@ -0,0 +1,63 @@
# Task 02 — Toggle behavior in the shared header module
**Phase:** `46_mobile_hamburger_nav` · **Source:** `TODO.md:9` — "The navbar on mobile is way too squished. Make it a hamburger dropdown menu with a nice animation"
**Story:** `.agent/user_stories/mobile-hamburger-nav.md`
## Objective
The menu opens and closes with correct ARIA state, Esc/link/outside dismissal, and desktop-resize cleanup — one module-owned binding, like the existing sign-out/steering bindings in `header.js`.
## Work
1. `frontend/assets/header.js` — a new module-import binding (same pattern as the sign-out binding: look up at import, guard null-safe, no page-script involvement):
```js
/* ---------- mobile hamburger (phase 46; module-owned) ----------
* ≤640px only (CSS hides the button elsewhere): #nav-toggle opens the
* nav as a dropdown (#app-nav .is-open — the animated state, task 01
* CSS). One binding for all six pages; a page without either element
* is a no-op, like the rest of this module. The nav LINKS keep their
* ship-hidden whoami contract (hidden links stay hidden inside the
* menu) — this binding only toggles the container. */
const navToggle = document.querySelector("#nav-toggle");
const appNav = document.querySelector("#app-nav");
function setNavMenu(open) {
if (!appNav || !navToggle) return;
appNav.classList.toggle("is-open", open);
navToggle.setAttribute("aria-expanded", open ? "true" : "false");
}
if (navToggle && appNav) {
navToggle.addEventListener("click", () =>
setNavMenu(!appNav.classList.contains("is-open")));
// A link click navigates (or closes same-page) — shut the menu.
appNav.addEventListener("click", (e) => {
if (e.target.closest("a")) setNavMenu(false);
});
// Esc closes while open (document-level; the menu is the only
// document-level overlay this module owns).
document.addEventListener("keydown", (e) => {
if (e.key === "Escape" && appNav.classList.contains("is-open")) {
setNavMenu(false);
navToggle.focus(); // focus returns to the opener
}
});
// Resize back to desktop: the inline nav reappears — no stale open
// state (the .is-open class is scoped by the ≤640px CSS anyway, but
// dropping it keeps aria-expanded honest).
const mq = window.matchMedia("(max-width: 640px)");
const onMqChange = () => { if (!mq.matches) setNavMenu(false); };
if (mq.addEventListener) mq.addEventListener("change", onMqChange);
else mq.addListener(onMqChange); // older engines, defensive
}
```
- update the module docstring: add the hamburger bullet (phase 46, owner permission 2026-08-27, `TODO.md` L9).
2. `tests/unit/test_hamburger_nav.py` — extend (from task 01) with the JS pins: `header.js` contains the `#nav-toggle` binding, `setNavMenu` (or equivalent) syncing **both** `.is-open` and `aria-expanded`, the delegated `a`-click close, the `Escape` close (with focus return), and the `matchMedia("(max-width: 640px)")` change-close; assert the binding is null-safe (`navToggle && appNav` guard).
3. `uv run pytest` green at this checkpoint.
## Testing & Quality
- Unit: the extended pin file; full suite green.
- Coverage: **>90%** on `app/` (unchanged — frontend-only).
## Completion Criteria
- [ ] One module-owned binding: click toggles (aria-expanded in sync), Esc closes + refocuses the toggle, a link click closes, desktop resize closes; pages lacking the elements are a no-op.
- [ ] The auth visibility contract is untouched — the binding toggles the container only; `hidden` links stay hidden.
- [ ] Full suite green at this checkpoint.
@@ -0,0 +1,28 @@
# Task 03 — Hamburger E2E + regressions + commit
**Phase:** `46_mobile_hamburger_nav` · **Source:** `TODO.md:9` — "The navbar on mobile is way too squished. Make it a hamburger dropdown menu with a nice animation"
**Story:** `.agent/user_stories/mobile-hamburger-nav.md`
## Objective
Prove the full mobile contract in the browser (visibility, contents per auth state, navigation, dismissal, animation, reduced motion, desktop regression) and commit the phase.
## Work
1. `tests/e2e/test_mobile_hamburger_nav.py` (new) — mock-only, DB up. Mobile tests use a 375×812 page (new page per test, or `page.set_viewport_size` — the conftest `page` fixture is 1280×800, so create mobile pages via the `browser` fixture); per the story's Playwright Mapping Rule:
- `test_mobile_hamburger_visible_and_bar_roomy` — 375px: `#nav-toggle` visible (box ≥44px in both dimensions), `aria-expanded="false"`, the inline nav links are **not** visible in the bar (menu closed — bounding boxes outside the header band or opacity 0), and no horizontal overflow (`document.documentElement.scrollWidth <= window.innerWidth`);
- `test_anonymous_menu_contents` — anonymous at 375px: click `#nav-toggle` → `aria-expanded="true"`, exactly **one** visible link in `#app-nav` ("Chat"); `#nav-sources` / `#nav-git-sources` / `#nav-tuning` remain `hidden` inside the menu;
- `test_admin_menu_contents` — login (e2e.auth_helpers) at 375px: open → Chat / Sources / Git sources / Tuning all visible (the whoami reveal works inside the menu);
- `test_link_click_navigates_and_closes` — admin at 375px: open, click "Sources" → URL becomes `/sources.html` and on the arrival page the menu is closed (`aria-expanded="false"`, no `.is-open`);
- `test_esc_and_outside_close` — open, press `Escape` → closed **and** focus is back on `#nav-toggle`; open again, click a neutral point (e.g. the page footer/main) → closed. (If the outside-click close is not implemented per task 02's contract — it is not: only Esc/link/resize close — assert instead that the menu stays open on an outside click and **note the accepted behavior** in the test docstring; the story's AC 4 lists Esc + link + resize, not backdrop click. Do NOT add a backdrop-close — it is out of the locked scope.)
- `test_animation_and_reduced_motion` — motion allowed: computed `transition-duration` on `#app-nav` includes `0.18s` (opacity/transform pair); open → the class/aria flip; `reducedMotion: "reduce"` (new context via the `browser` fixture): computed transition is `none`/`0s` and open/close still works;
- `test_desktop_unchanged` (regression) — 1280×800: `#nav-toggle` not visible (`display: none`), the inline nav renders in the bar exactly as before (admin: all four links visible inline).
2. Regression pass (isolation runs): `test_nav_consistency.py`, `test_header_consistency.py`, `test_shared_header.py`, `test_responsive_polish.py`, `test_tuning_nav_link.py`.
3. `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL unchanged; `uv run ruff check . && uv run pyright` clean.
4. Commit (Conventional Commits, `--no-gpg-sign`), e.g. `feat(header): hamburger dropdown nav on mobile (owner permission)`, staging this phase's files; move `.agent/phases/todo/46_mobile_hamburger_nav/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_mobile_hamburger_nav.py -v --no-cov` green in isolation.
- Coverage: **>90%** on `app/` (unchanged — frontend-only phase).
## Completion Criteria
- [ ] The story E2E suite passes in isolation (all seven tests); the five regression suites pass in isolation.
- [ ] One atomic `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
@@ -0,0 +1,42 @@
# Phase 47 — Import quadlet + jinja files
**Source:** `TODO.md` L10–L11 — "Add '.container', '.network', '.volume' and other quadlet files to the list of allowed/parsed files" / "Add '.j2' jinja files to the list of allowed/parsed files"
**Story:** `.agent/user_stories/quadlet-jinja-import.md`
**Context:** A9 (LOCKED, revised 2026-08-21): default import formats `md, markdown, txt, yaml, yml, json, py`; `app/config.py::_ALLOWED_IMPORT_EXTENSIONS` bounds `BOR_IMPORT_EXTENSIONS` (narrow-only); `app/rag/chunker.py::_FORMAT_CHUNKERS` maps suffix → chunker (unknown suffix → `chunk_text` fallback). The owner-locked revision (2026-08-27, roadmap R1): ten new formats join the allowed **and** default set — the full Podman quadlet family (`container, network, volume, image, pod, kube, swap, os, endpoint`) plus `j2` — chunked as plain text.
## Objective
Quadlet unit files and Jinja templates are indexed like any other A9 format: allowed + default in config, dispatched to plain-text chunking, and provable end to end (import → catalog → viewer → retrieval).
## Dependencies
- `46_mobile_hamburger_nav` (todo) — sequential only (no shared files: this phase is `app/` + `scripts/` + tests + fixtures + docs).
- `38_local_directory_sources` / `28_git_based_sources` (complete) — the import path the new formats ride (`import_sources` / sync).
- `02_story_import_documents` (complete) — the A9 format machinery (walk, title, delta, prune) the new formats inherit unchanged.
## Tasks
1. `01_config_formats.md` — allowed + default extension sets, `.env.example`, config unit tests.
2. `02_chunker_dispatch_fixtures.md` — chunker dispatch for the ten suffixes + the fixture files.
3. `03_importer_integration.md` — importer walk/delta/prune parity + integration test.
4. `04_quadlet_e2e_and_docs_commit.md` — story E2E + README + PLAN A9 revision note + commit.
## Testing & Quality
- Unit: `test_config.py` (allowed set contains all ten; the default CSV carries them after the original seven; the env validator accepts the new names and still rejects unknown ones; `import_extension_set` dotted form); `test_chunker.py` (dispatch for **every** new suffix → `chunk_text` semantics: a quadlet TOML fixture and a jinja fixture chunk under `HARD_MAX_CHARS`, paragraph packing behaves); `test_importer.py` (a directory walk with the new files indexes them; hidden dirs + exclusions still filter).
- Integration: import over a temp tree with quadlet+j2 files → `documents` + `chunks` rows, delta re-import idempotent.
- Coverage: **>90%** on `app/` — config/chunker/importer changes fully covered.
- E2E (mandatory, A16): `tests/e2e/test_quadlet_jinja_import.py`, run in isolation.
## Completion Criteria
- [ ] A default-extensions import indexes `.container` / `.network` / `.volume` / `.image` / `.pod` / `.kube` / `.swap` / `.os` / `.endpoint` / `.j2` files (fixture-proven); no env configuration needed.
- [ ] The Sources table lists them; the viewer shows a `.container` file's TOML content with the stem as title.
- [ ] A question containing a `.j2` sentinel is non-deflected with the `.j2` doc as a source chip (A8 FTS-OR honesty gate).
- [ ] `BOR_IMPORT_EXTENSIONS` still rejects truly unknown extensions (validator intact).
- [ ] README + `.env.example` + PLAN.md A9 revision note record the extended set.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%.
- [ ] `uv run pytest tests/e2e/test_quadlet_jinja_import.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_import_documents.py`, `test_sync_button.py`, `test_git_sources_admin.py`.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.
## Locked decisions
- **Owner-locked revision (2026-08-27, roadmap R1):** A9's format set is **revised** — ten formats added to the allowed + default set (the full quadlet family + `j2`); plain-text chunking (no TOML/Jinja-aware splitters); recorded as a PLAN.md A9 revision note with owner permission — a recorded revision, not a silent deviation (AGENTS.md rule 3).
- **A9 invariants kept:** hidden (dot) directories still skipped; the exclusion list unchanged; narrow-only `BOR_IMPORT_EXTENSIONS` validator; sha256 delta / prune unchanged; `HARD_MAX_CHARS` (1200) honored by the `chunk_text` dispatch.
- **A17 honoured** — one atomic commit.
@@ -0,0 +1,29 @@
# Task 01 — Config: the ten new formats (allowed + default)
**Phase:** `47_quadlet_jinja_import` · **Source:** `TODO.md:10–11` — "Add '.container', '.network', '.volume' and other quadlet files to the list of allowed/parsed files" / "Add '.j2' jinja files to the list of allowed/parsed files"
**Story:** `.agent/user_stories/quadlet-jinja-import.md`
## Objective
`app/config.py` allows and imports the ten new formats by default; the env validator keeps rejecting truly unknown extensions.
## Work
1. `app/config.py` —
- `_ALLOWED_IMPORT_EXTENSIONS`: add `"container", "network", "volume", "image", "pod", "kube", "swap", "os", "endpoint", "j2"` (with a comment: A9 revised 2026-08-27, owner permission — the full Podman quadlet family + Jinja templates);
- `import_extensions` default: `"md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2"` (the original seven first, the ten appended — order is cosmetic, the set is what matters; keep the field's docstring and the `mode="after"` validator **unchanged** — it already normalizes/lowercases/dedups and rejects unknowns, so it accepts the new names automatically);
- update the module-level comment on `_ALLOWED_IMPORT_EXTENSIONS` (it cites A9 revised 2026-08-21 — append the 2026-08-27 revision).
2. `.env.example` — the commented `BOR_IMPORT_EXTENSIONS` line updates to the new default CSV (it currently documents the old default).
3. `tests/unit/test_config.py` —
- the allowed set contains all seventeen formats;
- the default `import_extensions` includes the ten new names (assert each);
- `import_extension_set` returns the dotted lowercased set (`.container`, `.j2`, …);
- the validator **accepts** a `BOR_IMPORT_EXTENSIONS` containing the new names (e.g. `md,container,j2`) and **still rejects** an unknown one (e.g. `md,xyz`) — the never-widen contract with the widened base set.
4. `uv run pytest tests/unit/test_config.py -v` green; full unit suite green (the chunker/importer don't know the new suffixes yet — task 02; no test at this checkpoint imports a new-format file).
## Testing & Quality
- Unit: as above; coverage **>90%** on `app/` (config covered).
- No behavior change for existing formats (the default CSV only grows — every previously-imported file still matches).
## Completion Criteria
- [ ] Default settings import all seventeen formats; env narrowing/widening rules behave (new names allowed, unknowns rejected).
- [ ] `.env.example` documents the new default.
- [ ] Full suite green at this checkpoint.
@@ -0,0 +1,39 @@
# Task 02 — Chunker dispatch + fixture files
**Phase:** `47_quadlet_jinja_import` · **Source:** `TODO.md:10–11` — "Add '.container', '.network', '.volume' and other quadlet files to the list of allowed/parsed files" / "Add '.j2' jinja files to the list of allowed/parsed files"
**Story:** `.agent/user_stories/quadlet-jinja-import.md`
## Objective
Every new suffix dispatches to plain-text paragraph packing (`chunk_text`), and the fixture tree carries realistic quadlet + jinja files for the unit/integration/E2E layers.
## Work
1. `app/rag/chunker.py` —
- `_FORMAT_CHUNKERS`: add the ten entries (owner decision R1: plain-text packing — no TOML/Jinja-aware splitter):
```python
".container": chunk_text, ".network": chunk_text, ".volume": chunk_text,
".image": chunk_text, ".pod": chunk_text, ".kube": chunk_text,
".swap": chunk_text, ".os": chunk_text, ".endpoint": chunk_text,
".j2": chunk_text,
```
- update the `_FORMAT_CHUNKERS` comment (currently "A9, revised: md, markdown, txt, yaml, yml, json, py") and the module docstring's per-format list (add: "**container / network / volume / image / pod / kube / swap / os / endpoint / j2** (A9 revised 2026-08-27) — quadlet unit files (TOML) and Jinja templates; plain-text paragraph packing (`chunk_text`) — no format-specific splitter (owner decision).").
- `chunk_document`'s unknown-suffix fallback stays as-is (belt-and-braces).
2. **Fixture files** (under `tests/fixtures/docs/homelab/` — the tree the import E2E imports; hidden dirs are skipped, so no dot-dirs):
- `quadlet/compose.container` — realistic quadlet TOML (≥ ~1 500 chars to exercise sub-splitting past a single paragraph pack): `[Unit]` (Description/Wants), `[Service]` (Restart=always), `[Container]` (Image, Ports, Environment, Network, Volume mounts, Exec), comments. Include a unique sentinel token on its own line, e.g. `# RESE-QUADLET-SENTINEL-77aa`.
- `quadlet/lan.network` — small: `[Unit]` + `[Network]` (Driver=bridge, IPAMDriver, Subnets) + sentinel `# RESE-NETWORK-SENTINEL-11bb`.
- `quadlet/cache.volume` — small: `[Unit]` + `[Volume]` (Driver, Device) + sentinel `# RESE-VOLUME-SENTINEL-22cc`.
- `templates/deploy.j2` — a Jinja snippet with `{% for %}` / `{{ var }}` / `{# comment #}` constructs (realistic: an ansible-style service template) + sentinel `RESE-JINJA-SENTINEL-33dd` (no `#` prefix — it lives in a `{% set %}` line or a comment the importer keeps).
- keep the existing fixture files byte-identical (other E2E suites import this tree — `test_import_documents.py` asserts exact chunk counts: **run that suite's expectations check**: the tree grew by 4 files, so the phase-02 import E2E's document/chunk count assertions will change — update `tests/e2e/test_import_documents.py`'s count constants in this task, or fold that update into task 04's E2E work; whichever you choose, the full E2E regression pass in task 04 must be green. Prefer updating the constants here so task 03's integration test and task 04 share the same fixture state.)
3. `tests/unit/test_chunker.py` —
- dispatch: for **each** of the ten suffixes, `chunk_document(content, "x/<name>.<suffix>")` produces the same chunks as `chunk_text(content, …)` (parametrize over the suffix list);
- the `.container` fixture (read the file in the test, house pattern) chunks into ≥2 chunks, every chunk ≤ `HARD_MAX_CHARS`, and the sentinel token survives in some chunk;
- the `.j2` fixture chunks; Jinja braces are just text (no special handling — assert a `{{` line appears verbatim in a chunk);
- the unknown-suffix fallback is unchanged (a `.whatever` file still chunk-paragraphs).
4. `uv run pytest tests/unit/test_chunker.py -v` green; full unit suite green.
## Testing & Quality
- Unit: as above; coverage **>90%** on `app/` (chunker dispatch covered).
- No behavior change for the seven original formats (their chunker bindings are untouched).
## Completion Criteria
- [ ] Ten dispatch entries; docstrings/comments cite the A9 revision; fixtures exist with their sentinels and the ≥1 200-char container file.
- [ ] `test_import_documents.py` count constants updated (or explicitly deferred to task 04 — state the choice in the task's completion note).
@@ -0,0 +1,30 @@
# Task 03 — Importer parity: walk, delta, prune, titles
**Phase:** `47_quadlet_jinja_import` · **Source:** `TODO.md:10–11` — "Add '.container', '.network', '.volume' and other quadlet files to the list of allowed/parsed files" / "Add '.j2' jinja files to the list of allowed/parsed files"
**Story:** `.agent/user_stories/quadlet-jinja-import.md`
## Objective
Prove the new formats ride the existing import machinery unchanged: the default walk picks them up, delta detection re-imports on change, prune drops them on removal, and titles fall back to the file stem.
## Work
1. `tests/unit/test_importer.py` — extend (house style — the walk function `iter_importable_files` is pure and unit-tested against temp trees):
- a temp tree containing one file per new extension (all ten) + one unknown (`.xyz`) + one hidden-dir file (`.esphome/x.container`) + one exclusion (`node_modules/y.container`): the default-extensions walk returns exactly the ten new files — unknown/hidden/excluded filtered;
- the seven original extensions still walk (regression in the same test);
- title extraction: a `.container` file with no H1 gets the stem title (`extract_title` fallback — via the importer's title path, whatever the house test asserts titles through).
2. `tests/integration/` — a new `tests/integration/test_import_quadlet_jinja.py` (or extend `test_importer_e2e.py` if that file's harness fits better — choose and note):
- build a temp source dir with a `.container`, a `.volume`, and a `.j2` file (reuse the fixture files' content or small inline variants);
- run `import_sources([dir], fake_llm, prune=False)` with the house fake (`tests/fakes.py::FakeEmbedder` — it already implements `embed` + `chat`; give it an `embed_one` delegate if the import path calls one — check the `Embedder` protocol in `app/rag/importer.py` and satisfy exactly what it names):
- the three docs land in `documents` (source/path/title — stem titles) with non-zero `chunks` rows;
- **delta:** re-run with the `.j2` file's content changed → that doc `updated` (hash changed), the others `unchanged`;
- **prune:** delete the `.volume` file, re-run with `prune=True` → pruned count 1, the row gone, its chunks cascade-deleted.
- use the existing DB integration harness (the conftest app/db fixtures in `tests/conftest.py` — same pattern as `test_importer_e2e.py`).
3. `uv run pytest tests/unit/test_importer.py tests/integration/test_import_quadlet_jinja.py -v` green; full suite green (DB up for the integration part: `podman compose up -d db`).
## Testing & Quality
- Unit + integration as above; coverage **>90%** on `app/` (the importer is unchanged code — the new coverage comes from exercising it with the new formats; if TOTAL dips from task 01's config growth, add asserts — but no `app/` code change is expected in this task).
- No `app/` code change expected: if a gap is found (e.g. the walk already accepts any dotted suffix and the config set was the only gate), record that in the task completion note — the tests then prove the gate's location.
## Completion Criteria
- [ ] Default walk indexes the ten new formats; hidden-dir/exclusion/unknown filtering unchanged; stem titles.
- [ ] Delta + prune parity for the new formats (integration).
- [ ] Full suite green at this checkpoint.
@@ -0,0 +1,30 @@
# Task 04 — Quadlet/jinja E2E + docs (README, PLAN A9 revision) + commit
**Phase:** `47_quadlet_jinja_import` · **Source:** `TODO.md:10–11` — "Add '.container', '.network', '.volume' and other quadlet files to the list of allowed/parsed files" / "Add '.j2' jinja files to the list of allowed/parsed files"
**Story:** `.agent/user_stories/quadlet-jinja-import.md`
## Objective
Prove the story end to end (import → catalog → Sources table → viewer → FTS retrieval), record the A9 revision in the docs, run the regressions, and commit the phase.
## Work
1. `tests/e2e/test_quadlet_jinja_import.py` (new) — mock-only, DB up, per the story's Playwright Mapping Rule (the import happens out-of-band against the session mock, exactly like `test_import_documents.py`: truncate the KB, `import_sources([FIXTURES], LLMClient(settings))` in a thread — reuse that file's helpers/pattern; the task-02 fixtures are already in the tree):
- `test_quadlet_and_jinja_indexed` — after the module import, `GET /api/docs` lists `quadlet/compose.container`, `quadlet/lan.network`, `quadlet/cache.volume`, `templates/deploy.j2`, each with a non-zero chunk count and the stem as title;
- `test_sources_table_shows_them` — admin: `/sources.html` renders rows for the four files (path links present, `.doc-link`);
- `test_container_content_viewable` — open `compose.container` from the Sources table (modal, phase 26): the content area contains the `[Container]` section text and the `RESE-QUADLET-SENTINEL-77aa` sentinel; the title is the stem (`compose`);
- `test_jinja_retrievable_not_deflected` — ask a question containing `RESE-JINJA-SENTINEL-33dd` (the A8 gate: an FTS hit among the candidates keeps it honest-positive — LOW requires **zero** FTS hits): the brain bubble is **not** `.is-deflected` and a source chip names `templates/deploy.j2` (the mock's answer shape is deterministic; the assertion is on the gate + the chips, not the prose).
- module fixture: truncate `documents`/`chunks`/`query_log` per test module (the house E2E pattern) and re-import — note: this file's re-import changes the KB for the session; it is run in **isolation** (A16), so no cross-suite interference.
2. **Docs:**
- `README.md` — wherever the import format list is documented (the "Import & update" section mirrors PLAN §11 / A9), extend it with the ten new formats (the 2026-08-27 A9 revision, plain-text chunking);
- `.agent/PLAN.md` — **the only PLAN edit in this phase** (owner-locked revision R1): in the §2 anchors table, the A9 row's decision text gains the extension — append to the A9 row (keep the original wording, mark the revision in the row's notes/status or in a revision note under the table, the established style): "**A9 revision (phase 47, owner permission 2026-08-27):** the format set extends with the Podman quadlet family (`container, network, volume, image, pod, kube, swap, os, endpoint`) and `j2` (Jinja templates) — plain-text chunking (`chunk_text`), owner: `TODO.md` L10–L11. The narrow-only `BOR_IMPORT_EXTENSIONS` rule and the hidden-dir/exclusion invariants are unchanged." Update PLAN §5's chunking-policy format line and §11's workflow line to list the extended set (same note style). Touch **nothing else** in PLAN.md.
3. Regression pass (isolation runs): `test_import_documents.py` (task 02's count-constant update must hold — the tree now has four more files), `test_sync_button.py`, `test_git_sources_admin.py`.
4. `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
5. Commit (Conventional Commits, `--no-gpg-sign`), e.g. `feat(import): index quadlet unit files and jinja templates (A9 revision)`, staging this phase's files **including the force-added `.agent/PLAN.md`** (AGENTS.md rule 8) and the phase dir move `.agent/phases/todo/47_quadlet_jinja_import/` → `.agent/phases/complete/`.
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_quadlet_jinja_import.py -v --no-cov` green in isolation.
- Coverage: **>90%** on `app/` (tasks 01–03 carry it).
## Completion Criteria
- [ ] The story E2E suite passes in isolation (all four tests); the three regression suites pass in isolation.
- [ ] README + `.env.example` (task 01) + PLAN.md (A9 row + §5 + §11) record the extended set; no other PLAN change.
- [ ] One atomic `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/`.