feat(sources): real-time file progress for sync and upload — background upload with success toast
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
# Phase 61 — Stale UI Copy (retire the homelab-era text)
|
||||
|
||||
**Source:** `TODO.md` L4 — "Clean up the UI, there's text that references old features or talks about the way BOR used to work. For example the page-sub text in knowledge base references !/Homelab and ~/Deployments but those aren't the only sources and sometimes aren't included."
|
||||
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-01)
|
||||
**Context:** Verified copy inventory (audited 2026-09-01, all line numbers as of this writing):
|
||||
- `frontend/index.html` L6 — meta description "Ask Brain of Reese anything about the homelab and deployments."; L199–201 — empty-state sub "I've read through the homelab and deployment notes — ask me anything, and I'll point you at the exact doc. You've got this."; L231 — composer `placeholder="Ask me about the homelab…"`.
|
||||
- `app/config.py` L265–270 — the DEFAULT suggestion chips: 3 of 4 are owner-corpus-specific ("How is my Kubernetes cluster set up?", "What's my backup strategy?", "How do I deploy a new service?", "What's currently running in the homelab?"). Rendered as empty-state chips by `app.js` from `GET /api/suggestions`.
|
||||
- `frontend/sources.html` L110–114 — the KB page-sub cited in the TODO: "Every `*.md` file indexed from `~/Homelab` and `~/Deployments`. Press **Sync sources** to clone the repos and re-import." Stale on three counts: sources are now git repos + local directories + uploaded archives (phases 28/38/49); import extensions are configurable, not only `*.md` (phase 56); Sync pulls the configured git repos and imports local dirs — it does not clone "the repos" in general.
|
||||
- `frontend/git-sources.html` L175 — form example `placeholder="https://github.com/you/homelab.git"`.
|
||||
- ALL 9 pages — footer `<span>Powered by Reese's self-hosted models</span>` (`index.html` L245, `sources.html` L186, `tuning.html` L148, `document.html` L150, `git-sources.html` L243, `history.html` L173, `login.html` L131, `shared.html` L147, `doc-edit.html` L118).
|
||||
- OUT OF SCOPE (owner-locked A3): code/HTML *comments* mentioning "homelab" (`frontend/assets/app.js` L180/L1398/L1419, `frontend/assets/history.js` L39/L244, `frontend/index.html` L176) — not user-visible; left untouched to keep the diff minimal.
|
||||
- The phase-39 brand layer (`frontend/assets/brand.js`) already rewrites every occurrence of the literal "Brain of Reese" (titles, `.brand-text`, text nodes, aria/placeholder/meta attributes) — so strings that embed the BRAND name (e.g. the visually-hidden label "Ask Brain of Reese a question", `index.html` ~L227) stay as-is: the layer keeps them in sync when `BOR_APP_NAME` is set.
|
||||
- No existing test pins the exact default chip text (`tests/integration/test_api.py` L75–81 asserts only shape: a list, `len >= 3`, non-empty strings).
|
||||
- `.agent/` is UNTRACKED (owner instruction, commit 281f355) — the phase commit stages `frontend/ tests/ app/` only.
|
||||
- The E2E shared conftest server boots with the default env — this phase's E2E asserts the NEW neutral copy against that shared server.
|
||||
|
||||
## Objective
|
||||
Every user-visible string in the app describes how BOR works TODAY (any configured sources — git, local directories, uploads — with configurable extensions) instead of the owner's original `~/Homelab` + `~/Deployments` deployment: accurate, neutral defaults on all pages, with the exact replacement copy locked below.
|
||||
|
||||
## Dependencies
|
||||
- `60_sticky_navbar` (complete, preceding — no functional dependency; ordering by number)
|
||||
|
||||
## Tasks
|
||||
1. `01_chat_page_copy.md` — neutralize the chat page (`index.html` meta description, empty-state sub, composer placeholder) + the default suggestion chips in `app/config.py`.
|
||||
2. `02_sources_pages_copy.md` — rewrite the KB page-sub + review the remaining page-subs and the git-sources example URL against current behavior.
|
||||
3. `03_footer_defaults.md` — all 9 footers → neutral default text, tagged `class="footer-text"` (the hook phase 62's env var drives).
|
||||
4. `04_tests_and_commit.md` — unit text pins, dedicated E2E `tests/e2e/test_stale_ui_copy.py`, regression suites, one atomic commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit pins (house style, `tests/unit/test_frontend_brand.py` pattern — read the HTML/`config.py` as text): the old literals are GONE, the locked replacements are present, all 9 pages carry `.footer-text`.
|
||||
- E2E (mandatory, house rule): `tests/e2e/test_stale_ui_copy.py`, run in isolation against the shared conftest server — no visible page text contains "homelab"/"deployment" (case-insensitive), placeholder/meta/footer/chips match the locked copy.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate — this phase changes one Settings default, so the gate is meaningful, not ceremonial).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] No user-visible "homelab"/"deployment(s)" text on any of the 9 pages (case-insensitive; code comments excepted per A3) — proven by the E2E body-text scan on chat + sources pages and the DOM assertions.
|
||||
- [ ] KB page-sub describes the CURRENT source model (git repos + local directories + uploaded archives; Sync pulls/imports).
|
||||
- [ ] All 9 footers read the locked neutral default inside `<span class="footer-text">`.
|
||||
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_stale_ui_copy.py -v --no-cov` green in isolation (DB up).
|
||||
- [ ] Regression E2E suites green in isolation: `test_smoke.py`, `test_configurable_brand.py`, `test_header_consistency.py`, `test_dark_tech_theme.py`.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/` (`.agent/` stays untracked).
|
||||
|
||||
## Locked decisions
|
||||
- **Owner-locked (2026-09-01, roadmap confirmation, A1):** exact replacement copy —
|
||||
- meta description (`index.html` L6): `Ask anything about your indexed documents — every answer cites the exact doc.`
|
||||
- empty-state sub (`index.html` L199–201): `I've read through your documents — ask me anything, and I'll point you at the exact doc. You've got this.`
|
||||
- composer placeholder (`index.html` L231): `Ask me anything…`
|
||||
- KB page-sub (`sources.html` L110–114): `Every file indexed from your configured sources — git repositories, local directories, and uploaded archives. Press <strong>Sync sources</strong> to pull the latest and re-import.`
|
||||
- footer (all 9 pages): `Powered by self-hosted models`
|
||||
- **Owner-locked (2026-09-01, roadmap confirmation, A2):** the four default suggestion chips become —
|
||||
1. `What documents are in the knowledge base?`
|
||||
2. `Which source does each answer come from?`
|
||||
3. `How do I add a new source?`
|
||||
4. `Summarize the most recent document.`
|
||||
|
||||
(operators keep `BOR_SUGGESTIONS` for their own corpus).
|
||||
- **Owner-locked (2026-09-01, roadmap confirmation, A3):** the git-sources example becomes `https://github.com/you/your-repo.git`; code/HTML comments mentioning "homelab" are left untouched (not user-visible).
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add frontend/ tests/ app/ && git commit --no-gpg-sign -m "fix(web): retire the stale homelab-era copy — neutral, accurate defaults on every page"
|
||||
```
|
||||
@@ -0,0 +1,39 @@
|
||||
# Task 01 — Neutralize the chat page copy + default suggestion chips
|
||||
|
||||
**Phase:** `61_stale_ui_copy` · **Source:** `TODO.md:4` — "Clean up the UI, there's text that references old features or talks about the way BOR used to work…"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
The chat page (the app's front door) and the default onboarding chips describe the app generically — "your documents / your knowledge base" — instead of the owner's homelab, so a first-time deployer sees copy that fits THEIR corpus from the first render.
|
||||
|
||||
## Work
|
||||
1. `frontend/index.html`:
|
||||
- L6 — the `<meta name="description">` content becomes exactly (A1):
|
||||
`Ask anything about your indexed documents — every answer cites the exact doc.`
|
||||
- L199–201 — the `.empty-state-sub` paragraph text becomes exactly (A1):
|
||||
`I've read through your documents — ask me anything, and I'll point you at the exact doc. You've got this.`
|
||||
(keep the surrounding markup; the "You've got this." closer survives by design).
|
||||
- L231 — the `#message-input` `placeholder` becomes exactly (A1): `Ask me anything…`
|
||||
- Leave UNTOUCHED: the visually-hidden `<label>` "Ask Brain of Reese a question" (~L227) and any string embedding the literal "Brain of Reese" — the phase-39 brand layer (`brand.js` attribute pass) keeps those in sync with `BOR_APP_NAME`.
|
||||
2. `app/config.py` L265–270 — the `suggestions` default list becomes exactly (A2):
|
||||
```python
|
||||
suggestions: list[str] = [
|
||||
"What documents are in the knowledge base?",
|
||||
"Which source does each answer come from?",
|
||||
"How do I add a new source?",
|
||||
"Summarize the most recent document.",
|
||||
]
|
||||
```
|
||||
(the `BOR_SUGGESTIONS` JSON env override is unchanged — operators still override per corpus).
|
||||
3. No other file changes in this task (sources pages and footers are tasks 02/03; the unit pins land in task 04).
|
||||
|
||||
## Testing & Quality
|
||||
- No existing test pins the old default chip text (`tests/integration/test_api.py` L75–81 asserts shape only: list, `len >= 3`, non-empty strings — the new 4-chip default satisfies it).
|
||||
- `uv run pytest` green (full unit + integration); `uv run ruff check . && uv run pyright` clean.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `grep -n "homelab" frontend/index.html` matches ONLY the L176 code comment (A3) — no user-visible homelab text remains in the chat template.
|
||||
- [ ] `grep -n "Ask me anything" frontend/index.html` shows the new placeholder (exactly one user-visible occurrence).
|
||||
- [ ] `app/config.py` default chips == the four locked (A2) strings; `uv run pytest tests/integration/test_api.py -v` green.
|
||||
- [ ] Full suite green, lint + types clean, coverage >90%.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Task 02 — Rewrite the Knowledge-base copy to the current source model
|
||||
|
||||
**Phase:** `61_stale_ui_copy` · **Source:** `TODO.md:4` — "…the page-sub text in knowledge base references !/Homelab and ~/Deployments but those aren't the only sources and sometimes aren't included."
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
The Sources/KB page — the page the TODO cites by name — and the remaining page-subs describe the source model as BOR implements it TODAY: configured git repositories, local directories, and uploaded archives (phases 28/38/49), with import extensions set by `BOR_IMPORT_EXTENSIONS` (phase 56) — not "every `*.md` in `~/Homelab` and `~/Deployments`".
|
||||
|
||||
## Work
|
||||
1. `frontend/sources.html` L110–114 — the `<p class="page-sub">` becomes exactly (A1), keeping the `<strong>` around Sync sources:
|
||||
```html
|
||||
<p class="page-sub">
|
||||
Every file indexed from your configured sources — git repositories,
|
||||
local directories, and uploaded archives. Press <strong>Sync sources</strong>
|
||||
to pull the latest and re-import.
|
||||
</p>
|
||||
```
|
||||
(line-wrapped as in the original; the `<code>` tags for `~/Homelab` / `~/Deployments` are deleted, not reworded).
|
||||
2. `frontend/git-sources.html` L175 — the repo-URL form example becomes (A3):
|
||||
`placeholder="https://github.com/you/your-repo.git"`
|
||||
3. Verification read-through (fix only what is factually stale; do not reword accurate copy):
|
||||
- `frontend/sources.html` — the sync result/error copy (`#sync-result` announcer strings in `frontend/assets/sources.js`, the `#sync-error-banner` text): confirm it matches what Sync actually does (pulls configured git repos, imports local directories, re-imports; "no sources configured" loud failure). The phase-35+ error strings already name the real failure modes — expect no change; record the outcome in the commit message body only if something was fixed.
|
||||
- The remaining page-subs — `tuning.html` L104–108 (Global Tuning), `history.html` L106–110 (Saved chats), `doc-edit.html` L38–42 (Edit doc), `git-sources.html` L135–139 (Git sources): each describes current behavior (verified at audit time) — expect no change; if any is stale against current behavior, fix it with neutral, accurate wording and note it in the commit message.
|
||||
- The anonymous sources gate copy (`frontend/sources.html` `#sources-gate`) — confirms what stays public vs. locked; accurate — expect no change.
|
||||
4. Do NOT touch code/HTML comments mentioning "homelab" anywhere (A3).
|
||||
|
||||
## Testing & Quality
|
||||
- No logic changes in this task — the full suite stays green unchanged; `uv run ruff check . && uv run pyright` clean.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
- The text pins for the new copy land in task 04 (unit + E2E).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `grep -n "Homelab\|Deployments" frontend/sources.html` is empty.
|
||||
- [ ] `grep -n "homelab" frontend/git-sources.html` is empty.
|
||||
- [ ] The KB page-sub reads the locked (A1) text, including `<strong>Sync sources</strong>`.
|
||||
- [ ] The verification read-through is done and its outcome (changed / no change per page) is recorded in the task completion report.
|
||||
- [ ] Full suite green, lint + types clean.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 03 — Neutral footer default on all 9 pages (+ the `.footer-text` hook)
|
||||
|
||||
**Phase:** `61_stale_ui_copy` · **Source:** `TODO.md:4` — "Clean up the UI, there's text that references old features or talks about the way BOR used to work…"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
The footer line — duplicated verbatim on all 9 pages — stops naming the owner ("Powered by **Reese's** self-hosted models") and reads the locked neutral default; each footer span is tagged `class="footer-text"` so phase 62's `BOR_FOOTER_TEXT` env var has one stable selector to drive on every page.
|
||||
|
||||
## Work
|
||||
1. On ALL 9 pages, replace the identical footer span (audit-verified locations):
|
||||
```html
|
||||
<span>Powered by Reese's self-hosted models</span>
|
||||
```
|
||||
with exactly (A1):
|
||||
```html
|
||||
<span class="footer-text">Powered by self-hosted models</span>
|
||||
```
|
||||
Locations: `frontend/index.html` L245 · `frontend/sources.html` L186 · `frontend/tuning.html` L148 · `frontend/document.html` L150 · `frontend/git-sources.html` L243 · `frontend/history.html` L173 · `frontend/login.html` L131 · `frontend/shared.html` L147 · `frontend/doc-edit.html` L118.
|
||||
- The `<span class="footer-version" id="app-version"></span>` siblings (where present) are untouched.
|
||||
- No CSS change: `.footer-inner > span` is unstyled inline text today; adding the class is selector-only.
|
||||
2. Sanity: `grep -rn "Reese's" frontend/` must be empty after this task (the possessive form existed ONLY in these footers — the brand layer's literal is "Brain of Reese", which is a different string and stays).
|
||||
|
||||
## Testing & Quality
|
||||
- No logic changes — full suite green unchanged; `uv run ruff check . && uv run pyright` clean.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
- The 9-page `.footer-text` pin lands in task 04's unit test.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] All 9 pages carry `<span class="footer-text">Powered by self-hosted models</span>` (exactly 9 occurrences: `grep -rc 'class="footer-text"' frontend/*.html`).
|
||||
- [ ] `grep -rn "Reese's" frontend/` is empty.
|
||||
- [ ] The phase-39 brand layer is unaffected: `uv run pytest tests/unit/test_frontend_brand.py tests/e2e/test_configurable_brand.py -v --no-cov` — the unit file green; (the E2E needs DB up — run it in task 04's regression pass instead if the DB is down).
|
||||
- [ ] Full suite green, lint + types clean.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Task 04 — Unit text pins + dedicated E2E + regression pass + atomic commit
|
||||
|
||||
**Phase:** `61_stale_ui_copy` · **Source:** `TODO.md:4` — "Clean up the UI, there's text that references old features or talks about the way BOR used to work…"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
The new neutral copy is pinned at two layers so it can never silently regress: unit text pins (no browser, house style) and one dedicated Playwright suite asserting what a visitor actually SEES on the shared default server — then one atomic commit.
|
||||
|
||||
## Work
|
||||
1. `tests/unit/test_stale_ui_copy.py` (NEW — house pattern `tests/unit/test_frontend_brand.py`: read files as text, assert substrings):
|
||||
- Negative pins (GONE): `"the homelab and deployments"` (meta), `"the homelab and deployment notes"` (empty-state sub), `"Ask me about the homelab"` (placeholder), `"~/Homelab"` and `"~/Deployments"` (KB page-sub), `"Reese's self-hosted models"` (footers), the four old default chips incl. `"What's currently running in the homelab?"` — checked against `frontend/index.html`, `frontend/sources.html`, `frontend/git-sources.html`, and `app/config.py` where they lived.
|
||||
- Positive pins (PRESENT, exactly once where asserted): the five locked (A1) strings in their files; the four locked (A2) chips in `app/config.py`; `class="footer-text"` in ALL 9 HTML pages (iterate the same `HTML_PAGES` tuple as `test_frontend_brand.py`).
|
||||
- The old chip strings are asserted absent from `app/config.py` (they may still appear NOWHERE else — they are retired, not relocated).
|
||||
2. `tests/e2e/test_stale_ui_copy.py` (NEW — one story, one file, run in isolation, DB up; shared conftest server, default env — no second instance needed):
|
||||
- Chat page (`/` or `/index.html`):
|
||||
- `#message-input` placeholder === `Ask me anything…`;
|
||||
- `document.querySelector('meta[name="description"]').content` === the locked (A1) meta string;
|
||||
- the first `.footer-text` textContent === `Powered by self-hosted models`;
|
||||
- `document.body.innerText.toLowerCase()` contains neither `homelab` nor `deployment` (the rendered empty-state sub AND the four rendered suggestion chips are covered by this scan — chips come from `GET /api/suggestions`, default list);
|
||||
- the four rendered `.suggestions` chips' texts === the locked (A2) list, in order.
|
||||
- Sources page (`/sources.html`): the `.page-sub` innerText contains neither `Homelab` nor `Deployments` (case-insensitive) and DOES contain `configured sources`; (the page is anonymously viewable — the catalog gate hides the table, not the page-head).
|
||||
- Git sources page (`/git-sources.html`): the repo-URL input's `placeholder` attribute === `https://github.com/you/your-repo.git` (read from the DOM — the element exists even while `#git-sources-content` is `hidden` for anonymous; no sign-in needed).
|
||||
- Determinism: all assertions are settled-state (static HTML + one fetch of /api/suggestions); Playwright `expect` retries ride out chip rendering.
|
||||
3. Regression E2E pass (each in isolation, DB up): `test_smoke.py`, `test_configurable_brand.py`, `test_header_consistency.py`, `test_dark_tech_theme.py`.
|
||||
4. One atomic commit (`.agent/` untracked — owner instruction, commit 281f355):
|
||||
```bash
|
||||
git add frontend/ tests/ app/ && git commit --no-gpg-sign -m "fix(web): retire the stale homelab-era copy — neutral, accurate defaults on every page"
|
||||
```
|
||||
Commit body: the verification read-through outcome from task 02 (which pages needed no change / what was fixed) — one line each.
|
||||
|
||||
## Testing & Quality
|
||||
- `uv run pytest` green (unit + integration); coverage TOTAL **>90%** (validate.sh gate).
|
||||
- `uv run pytest tests/e2e/test_stale_ui_copy.py -v --no-cov` green in isolation.
|
||||
- `uv run ruff check . && uv run pyright` clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `tests/unit/test_stale_ui_copy.py` + `tests/e2e/test_stale_ui_copy.py` exist and are green (E2E in isolation, DB up).
|
||||
- [ ] The four regression E2E suites are green in isolation.
|
||||
- [ ] Full suite green, coverage >90%, lint + types clean.
|
||||
- [ ] Exactly one new commit on `main`, staged paths `frontend/ tests/ app/` only, `--no-gpg-sign`.
|
||||
- [ ] Phase dir moved to `.agent/phases/complete/61_stale_ui_copy/` (`.agent/` stays untracked).
|
||||
@@ -0,0 +1,54 @@
|
||||
# Phase 62 — UI Customization (deploy with your own look)
|
||||
|
||||
**Source:** `TODO.md` L3 — "Allow UI customization. This is brain of reese, but I want anyone to be able to deploy it with their name. The project should support an environment variable to call it 'Brain of <name>' along with custom message-input placeholder, custom footer-inner text, custom color themes, and anything else that an end-user might want to customize."
|
||||
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-01; builds on the COMPLETED phase-39 capability, whose dir was never filed under `complete/` — the code + `tests/e2e/test_configurable_brand.py` exist, commit `fe55be0`)
|
||||
**Context:** The app-name half of this TODO item is DONE (phase 39): `BOR_APP_NAME` → `Settings.app_name` (`app/config.py` L43) → `GET /api/config` (`app/api/config.py`, public, returns `{app_name, version, docs_repo_configured}`) → the classic `frontend/assets/brand.js` layer (parse-time `window.BOR_BRAND`, one `cache: "no-store"` fetch whose never-rejecting promise is `window.BOR_CONFIG_PROMISE`, DOM passes for title / `.brand-text` / text nodes / aria-label+placeholder+meta attributes; fetch failure ⇒ default stands + `console.warn`, the page never breaks). This phase extends that EXACT mechanism to three more settings — no new network surface, no new script, no templating engine (the frontend stays `StaticFiles(html=True)`, `app/main.py` L101):
|
||||
- Composer placeholder: `#message-input` exists only on `frontend/index.html` (the chat page).
|
||||
- Footer text: after phase 61, ALL 9 pages carry `<span class="footer-text">…</span>` inside `.footer-inner`.
|
||||
- Theme: the entire palette is `:root` custom properties in `frontend/assets/styles.css` L7–25 (`--bg, --surface, --ink, --ink-soft, --line, --brand, --brand-soft, --brand-ink, --accent-bg, --accent-ink, --accent-line, --ok-bg, --ok-ink, --err-bg, --err-ink, --err-line`) — a theme file overriding those variables is a drop-in; no component CSS references literal colors except via the variables (spot-verify while authoring the example theme).
|
||||
- No-CDN rule (AGENTS.md rule 6): theme CSS is served from the FastAPI static dir (`frontend/` in dev via `BOR_STATIC_DIR`; `/app/static` in the container).
|
||||
- `Containerfile` stage 1 esbuild/minify chain + per-page HTML copy does NOT ship a new `frontend/assets/themes/` dir — one line is added (task 03, A7).
|
||||
- Contract tests that must be updated when `/api/config` grows: `tests/integration/test_api.py` L21–46 (exact key set, two tests) and `tests/e2e/test_configurable_brand.py` L150 + L158 (exact key set); `tests/unit/test_frontend_brand.py` pins the brand layer.
|
||||
- The phase-39 E2E boots its own second instance (`testy_server`, `APP_PORT + 1`, `BOR_APP_NAME` override) — phase 62's E2E reuses that pattern on a distinct port (task 05).
|
||||
- `.agent/` is UNTRACKED (owner instruction, commit 281f355) — the phase commit stages `app/ frontend/ tests/ .env.example README.md Containerfile`.
|
||||
|
||||
## Objective
|
||||
Anyone can deploy BOR under their own identity with env vars alone: the app name (already shipped) plus a custom composer placeholder (`BOR_INPUT_PLACEHOLDER`), custom footer line (`BOR_FOOTER_TEXT`), and a custom color theme (`BOR_THEME=indigo.css` → `frontend/assets/themes/indigo.css`). Unset ⇒ byte-identical to the phase-61 neutral defaults.
|
||||
|
||||
## Dependencies
|
||||
- `61_stale_ui_copy` (todo, preceding — FUNCTIONAL: phase 61's neutral strings become these Settings' DEFAULTS, and its `.footer-text` class is the footer selector this phase drives; the "unset ⇒ byte-identical" contract is measured against phase-61 copy)
|
||||
|
||||
## Tasks
|
||||
1. `01_config_settings.md` — three new `Settings` fields (+ fail-loud `theme` filename validator) + three new `GET /api/config` keys; update the exact-key-set contract tests (integration + the phase-39 E2E).
|
||||
2. `02_brand_layer_customization.md` — `brand.js` applies placeholder / footer text / theme link from the same boot fetch; unit pins.
|
||||
3. `03_theme_assets.md` — `frontend/assets/themes/indigo.css` example theme (WCAG-AA palette) + `themes/README.md` authoring guide + the one `Containerfile` line.
|
||||
4. `04_docs.md` — `.env.example` + README env table + "Customizing the look" section.
|
||||
5. `05_e2e_customization.md` — `tests/e2e/test_ui_customization.py` (own instance, all three vars set; default instance byte-identical; malformed-theme fail-loud), regression suites, one atomic commit.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `theme` validator (empty ok; bare filename ok; path/`..`/non-`.css` rejected naming the value); `GET /api/config` returns the six keys with defaults AND with overrides; brand-layer JS pins (house style).
|
||||
- Integration: the config contract tests updated to the six-key set (task 01).
|
||||
- E2E (mandatory, house rule): `tests/e2e/test_ui_customization.py`, run in isolation — overridden instance shows custom placeholder/footer/computed theme color on the pages; the shared default server is byte-identical (default placeholder/footer, NO theme link, built-in `--brand`).
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `GET /api/config` returns exactly `{app_name, version, docs_repo_configured, input_placeholder, footer_text, theme}` — proven by integration + E2E.
|
||||
- [ ] With `BOR_INPUT_PLACEHOLDER` / `BOR_FOOTER_TEXT` / `BOR_THEME=indigo.css` set: the chat composer placeholder, the footer line on multiple pages, and the computed `:root --brand` all reflect the overrides; a `<link href="/assets/themes/indigo.css">` is present.
|
||||
- [ ] With nothing set: placeholder `Ask me anything…`, footer `Powered by self-hosted models`, no theme link, `--brand` = built-in `#f43f5e` — the phase-39/61 no-op contract holds.
|
||||
- [ ] `BOR_THEME=../evil.css` (or any malformed value) refuses startup, naming the value (phase-56 house style).
|
||||
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_ui_customization.py -v --no-cov` green in isolation (DB up).
|
||||
- [ ] Regression E2E suites green in isolation: `test_configurable_brand.py`, `test_stale_ui_copy.py`, `test_smoke.py`, `test_dark_tech_theme.py`.
|
||||
- [ ] `frontend/assets/themes/` ships in the container image (Containerfile line) and serves in dev (static dir).
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/` (`.agent/` stays untracked).
|
||||
|
||||
## Locked decisions
|
||||
- **Owner-locked (2026-09-01, roadmap confirmation, A4):** env var names — `BOR_INPUT_PLACEHOLDER`, `BOR_FOOTER_TEXT`, `BOR_THEME` (theme value = a FILENAME, e.g. `indigo.css`).
|
||||
- **Owner-locked (2026-09-01, roadmap confirmation, A5):** the theme mechanism — `BOR_THEME` names a file under `frontend/assets/themes/` served statically by FastAPI (no-CDN intact); the file overrides the `:root` palette variables; the example theme is `indigo.css` (dark indigo/slate, every text/background pair ≥4.5:1); a MALFORMED `BOR_THEME` (anything not matching `^[a-z0-9_-]+\.css$`) fails startup loudly, naming the value (phase-56 house style); a theme file MISSING at runtime degrades to the built-in theme (`console.warn`, the page never breaks — the loadHealth/brand.js house style).
|
||||
- **Owner-locked (2026-09-01, roadmap confirmation, A6):** the TODO's open-ended "anything else that an end-user might want to customize" is scoped to these three vars + the existing `BOR_APP_NAME` / `BOR_SUGGESTIONS`; the settings → `/api/config` → `brand.js` path is the extension point for future items (no further surface invented now).
|
||||
- **Owner-locked (2026-09-01, roadmap confirmation, A7):** the Containerfile ships themes via `cp -r ./assets/themes /out/assets/themes` in stage 1 (no per-file esbuild minify — a future theme file needs no Containerfile edit).
|
||||
|
||||
## Commit
|
||||
```bash
|
||||
git add app/ frontend/ tests/ .env.example README.md Containerfile && git commit --no-gpg-sign -m "feat(web): customizable placeholder, footer text, and color theme via BOR_* env vars"
|
||||
```
|
||||
@@ -0,0 +1,48 @@
|
||||
# Task 01 — Three new Settings + `/api/config` keys (+ validator)
|
||||
|
||||
**Phase:** `62_ui_customization` · **Source:** `TODO.md:3` — "…an environment variable to call it 'Brain of <name>' along with custom message-input placeholder, custom footer-inner text, custom color themes…"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
The backend half of the customization layer: three new `BOR_`-prefixed settings with phase-61's neutral strings as DEFAULTS, surfaced through the existing public `GET /api/config` boot fetch — with the existing exact-key-set contract tests moved to the new six-key set so the suite stays green from this task on.
|
||||
|
||||
## Work
|
||||
1. `app/config.py` — in the `# --- App ---` block (after `static_dir`, ~L47), add:
|
||||
```python
|
||||
# --- UI customization (phase 62, TODO L3) ---
|
||||
# Defaults are the phase-61 neutral copy — UNSET => byte-identical UI.
|
||||
input_placeholder: str = "Ask me anything…"
|
||||
footer_text: str = "Powered by self-hosted models"
|
||||
#: Theme file NAME under frontend/assets/themes/ (e.g. "indigo.css");
|
||||
#: empty = the built-in dark-tech palette. Validated: bare filename
|
||||
#: only — no paths, no ".." (no-CDN: served from the static dir).
|
||||
theme: str = ""
|
||||
```
|
||||
plus a `@field_validator("theme")` (pydantic v2, `mode="after"`): empty string passes; the value must match `^[a-z0-9_-]+\.css$` (ASCII, lowercase, bare filename) or the validator raises `ValueError` naming the offending value and the allowed shape (phase-56 fail-loud house style — a typo in `.env` must kill startup, not silently 404).
|
||||
- `input_placeholder` / `footer_text`: no validation beyond being strings (empty ⇒ the template default stands, handled by brand.js treating empty as "skip").
|
||||
2. `app/api/config.py` — the `GET /api/config` response grows to exactly six keys:
|
||||
```python
|
||||
return {
|
||||
"app_name": settings.app_name,
|
||||
"version": settings.app_version,
|
||||
"docs_repo_configured": settings.docs_configured,
|
||||
"input_placeholder": settings.input_placeholder,
|
||||
"footer_text": settings.footer_text,
|
||||
"theme": settings.theme,
|
||||
}
|
||||
```
|
||||
(values passed through verbatim — the frontend brand layer decides "empty = keep the template default"; NO new secrets surface: these are display strings, same public posture as `app_name`). Update the module + endpoint docstrings to cite phase 62.
|
||||
3. Contract-test updates (MUST land with this task or the suite goes red):
|
||||
- `tests/integration/test_api.py` L21–46: both exact-key-set assertions become `{"app_name", "version", "docs_repo_configured", "input_placeholder", "footer_text", "theme"}`; assert the DEFAULT values (`"Ask me anything…"`, `"Powered by self-hosted models"`, `""`); extend `test_config_follows_overridden_app_name`'s pattern with a `test_config_serves_ui_customization_overrides` test (fresh app with `Settings(input_placeholder=…, footer_text=…, theme="indigo.css")` → the three keys reflect the overrides).
|
||||
- `tests/e2e/test_configurable_brand.py` L150 + L158: the two exact-key-set assertions grow to the six-key set (values untouched — that suite's instance has no phase-62 overrides, so the new keys are their defaults).
|
||||
4. Unit tests — `tests/unit/test_config.py` (existing file, add cases): `theme` validator — `""` ok; `"indigo.css"` ok; `"Indigo.css"` rejected; `"../evil.css"` rejected; `"a/b.css"` rejected; `"indigo"` (no extension) rejected — each rejection names the value. Plus one test that `GET /api/config` (integration) default response carries the three new keys with the locked defaults.
|
||||
|
||||
## Testing & Quality
|
||||
- `uv run pytest` green (unit + integration); `uv run ruff check . && uv run pyright` clean.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `Settings` exposes `input_placeholder` / `footer_text` / `theme` with the locked defaults; `BOR_THEME=../evil.css` in the env fails app startup, naming the value (verified once via a unit/validator test — the E2E boots-check lands in task 05).
|
||||
- [ ] `GET /api/config` returns exactly the six keys (defaults + overrides) — integration tests green.
|
||||
- [ ] `tests/e2e/test_configurable_brand.py`'s key-set assertions updated (suite itself re-run in task 05's regression pass, DB up).
|
||||
- [ ] Full suite green, lint + types clean, coverage >90%.
|
||||
@@ -0,0 +1,35 @@
|
||||
# Task 02 — `brand.js` applies the placeholder, footer text, and theme
|
||||
|
||||
**Phase:** `62_ui_customization` · **Source:** `TODO.md:3` — "…custom message-input placeholder, custom footer-inner text, custom color themes…"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
The frontend half: the EXISTING `brand.js` boot-fetch pass (no new script, no new network call) applies the three new config values to the DOM — each empty ⇒ skipped, so an unset deployment stays byte-identical.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/brand.js` — inside the SAME settled-config `.then((cfg) => …)` that `applyBrand()` runs (after the existing `app_name` block — INDEPENDENT of it: the three new keys are applied even when `app_name` is the default/empty):
|
||||
- **Placeholder** — `cfg.input_placeholder` non-empty → `document.querySelector("#message-input")?.setAttribute("placeholder", cfg.input_placeholder)` (chat page only; every other page no-ops via the null guard).
|
||||
- **Footer text** — `cfg.footer_text` non-empty → `document.querySelectorAll(".footer-text").forEach((el) => { el.textContent = cfg.footer_text; })` (all 9 pages, phase-61 hook; `textContent` — no escaping needed, an operator string can't inject markup).
|
||||
- **Theme** — `cfg.theme` non-empty → insert a stylesheet link IMMEDIATELY AFTER the existing `<link rel="stylesheet" href="/assets/styles.css">` (find it by `rel="stylesheet"` + `href` ending in `styles.css`; append after it so the theme overrides win by cascade order):
|
||||
```js
|
||||
const link = document.createElement("link");
|
||||
link.rel = "stylesheet";
|
||||
link.href = "/assets/themes/" + cfg.theme; // filename validated server-side (task 01) — no path input can reach here via /api/config
|
||||
link.id = "theme-override";
|
||||
link.onerror = () => console.warn("brand: theme " + cfg.theme + " did not load — the built-in theme stands.");
|
||||
stylesLink.insertAdjacentElement("afterend", link);
|
||||
```
|
||||
Idempotent: skip insertion if `#theme-override` already exists (the brand pass runs once, but keep the guard — the loadHealth house style: never break the page, never double-apply).
|
||||
- Update the file-header contract comment: a "Phase 62 (owner-locked 2026-09-01, TODO L3)" section documenting the three keys, the empty-skip no-op contract, and the theme degradation path (A5).
|
||||
- Do NOT touch the existing app_name passes (1–4) or the `BOR_DOCS_REPO_CONFIGURED` flag logic — byte-identical behavior for them.
|
||||
2. `tests/unit/test_frontend_brand.py` — add pins (house style, read `brand.js` as text): markers for the three applications are present — `#message-input` placeholder application, the `.footer-text` selector, the `assets/themes/` link insertion with the `theme-override` id + `onerror` warn; the `app_name` literal default pin still holds.
|
||||
|
||||
## Testing & Quality
|
||||
- No Python logic in this task; `uv run pytest` green (unit pins + everything else unchanged); `uv run ruff check . && uv run pyright` clean.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
- Browser-level proof lands in task 05's E2E.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `brand.js` applies the three keys from the SAME settled config (no second fetch); empty values skip every application (byte-identical no-op).
|
||||
- [ ] Theme link is inserted after `styles.css`, guarded by `#theme-override`, and degrades with a `console.warn` on 404 (A5).
|
||||
- [ ] Unit pins green; full suite green; lint + types clean.
|
||||
@@ -0,0 +1,53 @@
|
||||
# Task 03 — Example theme (`indigo.css`) + authoring guide + Containerfile line
|
||||
|
||||
**Phase:** `62_ui_customization` · **Source:** `TODO.md:3` — "…custom color themes…"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
A working example theme a deployer can point `BOR_THEME` at out of the box — plus the documentation that makes "write my own theme" a 15-minute job, and the one Containerfile line that ships the directory in the container image (A7).
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/themes/indigo.css` (NEW) — overrides ONLY the 8 identity variables in a single `:root` block; the semantic families (accent/ok/err) are deliberately UNTOUCHED (they encode states — deflection amber, success green, error red — and are already AA in the built-in theme; a theme that keeps them stays honest):
|
||||
```css
|
||||
/* Phase 62 example theme — dark indigo/slate.
|
||||
Overrides the :root identity palette from styles.css; every
|
||||
text/background pair meets WCAG 2.1 AA (>= 4.5:1):
|
||||
ink on bg 15.8:1 · ink on surface 14.6:1 · ink-soft on surface 8.2:1
|
||||
dark bg ink on brand 6.4:1 · brand-ink on surface 11.9:1.
|
||||
Semantic families (accent/ok/err) inherit the built-in theme. */
|
||||
:root {
|
||||
--bg: #0a0e1a;
|
||||
--surface: #111726;
|
||||
--ink: #e6e9f0;
|
||||
--ink-soft: #a8b0c8;
|
||||
--line: #232c44;
|
||||
--brand: #818cf8;
|
||||
--brand-soft: #1a1f38;
|
||||
--brand-ink: #c7d2fe;
|
||||
}
|
||||
```
|
||||
- Before committing, re-verify the five ratios above (e.g. a 10-line python contrast calc against the built-in pairs' house comment style) — if any pair misses 4.5:1, adjust the value, not the bar (AGENTS.md rule 5).
|
||||
- Spot-check that no component CSS hard-codes a color the theme should own: `grep -n "f43f5e\|#f0e6e6\|#0f0a0a" frontend/assets/styles.css` should match ONLY the `:root` palette block + comments (if a literal lives elsewhere, note it in the task report — do NOT refactor styles.css in this phase).
|
||||
2. `frontend/assets/themes/README.md` (NEW) — the authoring guide, house-comment style:
|
||||
- How themes load: `BOR_THEME=<file>` → `/api/config` → `brand.js` inserts `<link rel="stylesheet" href="/assets/themes/<file>">` AFTER `styles.css` (later wins the cascade — that is the whole mechanism).
|
||||
- The variable table: the 8 identity variables (bg/surface/ink/ink-soft/line/brand/brand-soft/brand-ink) with the built-in values as reference, and the note that accent/ok/err are semantic and should stay.
|
||||
- Rules: filename `^[a-z0-9_-]+\.css$` (lowercase, bare filename — the server validator rejects anything else at startup); one `:root` block; every text/background pair ≥4.5:1 (AGENTS.md rule 5); never white-on-brand (the built-in's documented 3.7:1 trap) — dark bg ink on brand, as the built-in does.
|
||||
- Deployment: works in dev immediately (served from the static dir); in the container, rebuild the image (the `cp -r` picks up whatever is in `frontend/assets/themes/` at build time — no Containerfile edit for new files, A7).
|
||||
3. `Containerfile` — stage-1 `RUN` chain: after the `styles.css` minify line, add
|
||||
```
|
||||
&& cp -r ./assets/themes /out/assets/themes \
|
||||
```
|
||||
(A7: whole directory, no per-file esbuild — future themes need no Containerfile change). No other stage changes; the runtime stage copies the whole `/out` static tree, so the dir flows through untouched.
|
||||
4. Dev-server sanity (record in task report): with `BOR_THEME=indigo.css`, `http://localhost:8000/assets/themes/indigo.css` returns 200 from the static dir; with the default env the built-in palette renders (the file existing but `BOR_THEME` empty changes NOTHING — loading is opt-in via the env var, not directory scanning).
|
||||
|
||||
## Testing & Quality
|
||||
- No Python logic; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
- Add the theme-file unit pins to `tests/unit/test_frontend_brand.py` (or `test_stale_ui_copy.py`'s sibling if the executor prefers a new `tests/unit/test_themes.py`): the file exists, starts with a `:root` block, contains exactly the 8 `--` overrides (no other declarations), and the Containerfile contains the `cp -r ./assets/themes` line.
|
||||
- Browser proof (computed `--brand`, the link tag, visual sanity) lands in task 05's E2E.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `frontend/assets/themes/indigo.css` + `README.md` exist; all five contrast ratios verified ≥4.5:1 (calc output in the task report).
|
||||
- [ ] `Containerfile` ships `assets/themes/` (the line is in stage 1, after the styles.css minify).
|
||||
- [ ] The `grep` spot-check for hard-coded identity colors is recorded (expect: `:root` block + comments only).
|
||||
- [ ] Unit pins green; full suite green; lint + types clean.
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 04 — Document the customization surface (.env.example + README)
|
||||
|
||||
**Phase:** `62_ui_customization` · **Source:** `TODO.md:3` — "Allow UI customization. This is brain of reese, but I want anyone to be able to deploy it with their name…"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
A deployer who has never seen the codebase can find, understand, and apply all four customization vars (name, placeholder, footer, theme) from the docs alone — the house standard set by the phase-39 `BOR_APP_NAME` documentation.
|
||||
|
||||
## Work
|
||||
1. `.env.example` — in the `# --- App ---` block, right after the `BOR_APP_NAME` comment line (~L9), add (same comment style, phase-cited):
|
||||
```bash
|
||||
# BOR_INPUT_PLACEHOLDER=Ask me anything… # composer placeholder, chat page (phase 62)
|
||||
# BOR_FOOTER_TEXT=Powered by self-hosted models # footer line on every page (phase 62)
|
||||
# BOR_THEME= # filename under frontend/assets/themes/ (e.g. indigo.css) — overrides the built-in palette; empty = built-in (phase 62)
|
||||
```
|
||||
2. `README.md` — two edits:
|
||||
- The env-var table (the `BOR_APP_NAME` row sits ~L734): add three rows immediately after it, matching that row's column style — what it affects, the default, the no-op-when-unset contract:
|
||||
- `BOR_INPUT_PLACEHOLDER` | `Ask me anything…` | the chat composer placeholder; applied by `assets/brand.js` from `GET /api/config`; unset ⇒ the template default
|
||||
- `BOR_FOOTER_TEXT` | `Powered by self-hosted models` | the footer line on all 9 pages (the `.footer-text` spans); same mechanism
|
||||
- `BOR_THEME` | *(empty)* | a filename under `frontend/assets/themes/` (e.g. `indigo.css`) — a `:root` palette override injected after `styles.css`; the server refuses a malformed name at startup (bare `^[a-z0-9_-]+\.css$` filename); a missing file degrades to the built-in theme; unset ⇒ built-in dark-tech palette
|
||||
- A short **"Customizing the look"** subsection (3–6 sentences) placed directly above or below the env table (mirror the placement of the nearest feature section — the executor picks the spot that reads best and notes it in the task report): the app ships as "Brain of Reese" but every identity string is an env var (`BOR_APP_NAME` / `BOR_INPUT_PLACEHOLDER` / `BOR_FOOTER_TEXT`), themes are plain CSS variable overrides (`frontend/assets/themes/README.md` is the authoring guide, `indigo.css` the example), and everything unset renders the defaults byte-identically.
|
||||
3. No code changes in this task.
|
||||
|
||||
## Testing & Quality
|
||||
- Docs-only; `uv run pytest` green unchanged; `uv run ruff check . && uv run pyright` clean.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `.env.example` carries the three new commented vars in the App block with the exact default values (so copy-paste of the example reproduces the defaults).
|
||||
- [ ] README env table has the three rows + the "Customizing the look" subsection; the README builds/reads cleanly (no broken table rows, links resolve — `frontend/assets/themes/README.md` exists from task 03).
|
||||
- [ ] Full suite green, lint + types clean.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Task 05 — Dedicated E2E + regression pass + atomic commit
|
||||
|
||||
**Phase:** `62_ui_customization` · **Source:** `TODO.md:3` — "Allow UI customization. This is brain of reese, but I want anyone to be able to deploy it with their name…"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
One dedicated Playwright suite proves the whole contract end-to-end — an instance booted with all three customization vars shows the custom look, the default shared server is BYTE-IDENTICAL to today, and a malformed `BOR_THEME` refuses to boot — then one atomic commit and the phase is done.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_ui_customization.py` (NEW — one story, one file, run in isolation, DB up). Boots a second app instance exactly the way `test_configurable_brand.py`'s `testy_server` does (same env block, same DB, mock-LLM base URL, admin auth, static dir, mock-calibrated threshold) with THREE changes: port `APP_PORT + 2` (the brand suite owns `APP_PORT + 1` — do not collide), and the env overrides
|
||||
`BOR_INPUT_PLACEHOLDER="Ask the archive…"`, `BOR_FOOTER_TEXT="Custom footer line"`, `BOR_THEME="indigo.css"`.
|
||||
Test → contract mapping:
|
||||
1. `test_config_serves_the_overrides` — the instance's `GET /api/config` returns the three overridden values alongside the six-key set (the name stays the DEFAULT — this suite does not re-test `BOR_APP_NAME`; that is the phase-39 suite's job).
|
||||
2. `test_chat_page_shows_custom_placeholder_footer_theme` (instance) — on the chat page: `#message-input` placeholder === `Ask the archive…`; the first `.footer-text` textContent === `Custom footer line`; a `link#theme-override[rel="stylesheet"][href="/assets/themes/indigo.css"]` exists in the `<head>`; `getComputedStyle(document.documentElement).getPropertyValue("--brand").trim()` === `#818cf8` (the indigo value — custom properties return the specified token, so the string compare is stable; the built-in is `#f43f5e`).
|
||||
3. `test_footer_text_applies_on_other_pages` (instance) — on a second page (e.g. `/login.html`): the `.footer-text` textContent === `Custom footer line` (multi-page proof, no composer on that page — the placeholder application no-ops without error).
|
||||
4. `test_default_server_is_byte_identical` (shared conftest server) — placeholder === `Ask me anything…`; footer === `Powered by self-hosted models`; NO `#theme-override` link; computed `--brand` === `#f43f5e` (the phase-39/61 no-op contract).
|
||||
5. `test_malformed_theme_refuses_startup` — boot a short-lived uvicorn subprocess with `BOR_THEME="../evil.css"` (any port): the process exits non-zero within the timeout (or never starts listening) and its stderr names the value — the phase-56 fail-loud style, proven end-to-end, not just via the validator unit test.
|
||||
- Determinism: settled-state assertions throughout; Playwright `expect` retries ride out the brand.js fetch (the same pattern the brand suite documents in its module docstring).
|
||||
2. Regression E2E pass (each in isolation, DB up): `test_configurable_brand.py` (its key-set assertions were updated in task 01), `test_stale_ui_copy.py` (phase 61's copy contract is this phase's baseline), `test_smoke.py`, `test_dark_tech_theme.py` (the built-in palette is untouched when unset).
|
||||
3. One atomic commit (`.agent/` untracked — owner instruction, commit 281f355):
|
||||
```bash
|
||||
git add app/ frontend/ tests/ .env.example README.md Containerfile && git commit --no-gpg-sign -m "feat(web): customizable placeholder, footer text, and color theme via BOR_* env vars"
|
||||
```
|
||||
Commit body: one line — the three env vars + the indigo example theme + the docs/theme-guide pointers.
|
||||
|
||||
## Testing & Quality
|
||||
- `uv run pytest` green (unit + integration); coverage TOTAL **>90%** (validate.sh gate).
|
||||
- `uv run pytest tests/e2e/test_ui_customization.py -v --no-cov` green in isolation.
|
||||
- `uv run ruff check . && uv run pyright` clean.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `tests/e2e/test_ui_customization.py` exists (5 mapped tests) and is green in isolation (DB up).
|
||||
- [ ] The four regression E2E suites are green in isolation.
|
||||
- [ ] Full suite green, coverage >90%, lint + types clean.
|
||||
- [ ] Exactly one new commit on `main`, staged paths `app/ frontend/ tests/ .env.example README.md Containerfile`, `--no-gpg-sign`.
|
||||
- [ ] Phase dir moved to `.agent/phases/complete/62_ui_customization/` (`.agent/` stays untracked).
|
||||
@@ -0,0 +1,54 @@
|
||||
# Phase 63 — Unambiguous Document Listing for Agent Tools
|
||||
|
||||
**Source:** Live troubleshooting report (2026-09-01) — the `brain-of-reese` agent at `https://brain.experimental.reeseapps.com/` cannot read documents because the `list_documents` tool output uses an ambiguous `source/path — title` format that the LLM cannot reliably parse when paths contain `/` characters.
|
||||
|
||||
**Story:** `.agent/user_stories/agent-document-tools.md` (phase 37, extended)
|
||||
|
||||
**Context:** The agent's grounded-turn loop uses two tools: `list_documents` (returns the catalog) and `read_document` (reads one document). The catalog output is formatted as `source/path — title` (one line per document), but when a file lives inside a subdirectory of its source, the output looks like `brain-of-reese-main/homelab/aws-route53.md — aws-route53`. The LLM cannot reliably determine where `source` ends and `path` begins because both contain `/` separators. The thinking trace shows the LLM cycling through multiple failed attempts, each time guessing a different split.
|
||||
|
||||
The fix changes the catalog format to unambiguous `source: X | path: Y | title: Z` lines and updates the tool descriptions to tell the LLM how to parse them.
|
||||
|
||||
## Objective
|
||||
|
||||
The agent can always extract the correct `(source, path)` pair from the `list_documents` output, regardless of how many `/` characters the path contains.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `62_ui_customization` (todo, preceding — functional: no code changes in this phase touch UI or config)
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_agent_list_format.md` — change `list_catalog` output from `source/path — title` to `source: X | path: Y | title: Z`; update the `read_document` tool descriptions to reference the new format.
|
||||
2. `02_mock_and_unit_tests.md` — update `tests/e2e/mock_llm.py::_catalog_docs` to parse the new format; update `tests/unit/test_agent.py` (the "No document at" refusal line is unchanged — it names the lookup failure, not the list format).
|
||||
3. `03_e2e_document_tools.md` — validate that `tests/e2e/test_agent_document_tools.py` and `tests/e2e/test_agent_unlimited_tools.py` still pass (they assert on SSE frames and UI rendering, not on the catalog text format — the tool call `argument` field `source/path` is unchanged).
|
||||
4. `04_docs.md` — update any inline documentation that references the old format.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: `tests/unit/test_agent.py` — the "No document at" refusal message is unchanged (it reports the lookup failure, not the list format). The `_catalog_docs` mock parsing must produce identical `(source, path)` tuples for all existing test catalogs.
|
||||
- Integration: `tests/integration/test_agent_tools.py` — `list_catalog` ordering assertions are unchanged (the function returns `[(source, path, title), ...]` tuples; only the string formatting in `_execute_tool` changes).
|
||||
- E2E (mandatory, house rule): `tests/e2e/test_agent_document_tools.py` and `tests/e2e/test_agent_unlimited_tools.py` run in isolation — the SSE `argument` field (`source/path`) is unchanged; the mock's `_catalog_docs` parsing produces identical results; the tool lines and source chips render identically.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `list_catalog` output uses `source: X | path: Y | title: Z` format — verifiable by a unit test that patches `list_catalog` and checks the formatted string.
|
||||
- [ ] The `read_document` tool descriptions tell the LLM to extract `source` and `path` from the `source:` / `path:` labels.
|
||||
- [ ] `tests/e2e/mock_llm.py::_catalog_docs` parses the new format and produces identical `(source, path)` tuples for all existing test catalogs.
|
||||
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_agent_document_tools.py tests/e2e/test_agent_unlimited_tools.py -v --no-cov` green in isolation (DB up).
|
||||
- [ ] Regression E2E suites green in isolation: `test_chat_api.py`, `test_sse_events.py`, `test_whole_document_context.py`.
|
||||
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/` (`.agent/` stays untracked).
|
||||
|
||||
## Locked decisions
|
||||
|
||||
- **Owner-locked (2026-09-01, troubleshooting report, A1):** the new format uses `source: X | path: Y | title: Z` — one line per document, pipe-delimited fields, no ambiguity even with deeply nested paths.
|
||||
- **Owner-locked (2026-09-01, troubleshooting report, A2):** the `read_document` tool descriptions are updated to reference the new format (tell the LLM how to parse `source:` and `path:` labels).
|
||||
- **Owner-locked (2026-09-01, troubleshooting report, A3):** the SSE `argument` field (`source/path`) is unchanged — it is the tool call argument, not the list output. The frontend tool-line rendering is unchanged.
|
||||
- **Owner-locked (2026-09-01, troubleshooting report, A4):** the "No document at …" refusal message is unchanged — it reports the lookup failure, not the list format.
|
||||
|
||||
## Commit
|
||||
|
||||
```bash
|
||||
git add app/ tests/ && git commit --no-gpg-sign -m "fix(agent): unambiguous document listing format for LLM parsing"
|
||||
```
|
||||
@@ -0,0 +1,75 @@
|
||||
# Task 01 — Change `list_catalog` output format + update tool descriptions
|
||||
|
||||
**Goal:** Make the `list_documents` catalog output unambiguous for LLM parsing, and update the `read_document` tool descriptions to tell the LLM how to extract `source` and `path`.
|
||||
|
||||
## Changes
|
||||
|
||||
### `app/rag/agent.py`
|
||||
|
||||
#### 1. Change `list_catalog` output format
|
||||
|
||||
In `_execute_tool`, the `list_documents` handler formats the catalog as:
|
||||
|
||||
```python
|
||||
# BEFORE (ambiguous when path contains /)
|
||||
listing = f"{len(rows)} documents:\n" + "\n".join(
|
||||
f"{source}/{path} — {title}" for source, path, title in rows
|
||||
)
|
||||
```
|
||||
|
||||
Change to:
|
||||
|
||||
```python
|
||||
# AFTER (unambiguous — each field is labeled)
|
||||
listing = f"{len(rows)} documents:\n" + "\n".join(
|
||||
f"source: {source} | path: {path} | title: {title}"
|
||||
for source, path, title in rows
|
||||
)
|
||||
```
|
||||
|
||||
#### 2. Update `read_document` tool descriptions
|
||||
|
||||
In `AGENT_TOOLS`, update the `source` and `path` parameter descriptions to reference the new format:
|
||||
|
||||
```python
|
||||
# BEFORE
|
||||
"source": {
|
||||
"description": (
|
||||
"The document's source (a directory basename, e.g. 'Homelab')."
|
||||
),
|
||||
},
|
||||
"path": {
|
||||
"description": (
|
||||
"The document's path relative to its source directory."
|
||||
),
|
||||
},
|
||||
|
||||
# AFTER
|
||||
"source": {
|
||||
"description": (
|
||||
"The document's source, as shown after 'source: ' in the "
|
||||
"list_documents output (e.g. 'Homelab' from "
|
||||
"'source: Homelab | path: homelab/aws-route53.md')."
|
||||
),
|
||||
},
|
||||
"path": {
|
||||
"description": (
|
||||
"The document's path, as shown after 'path: ' in the "
|
||||
"list_documents output (e.g. 'homelab/aws-route53.md' from "
|
||||
"'source: Homelab | path: homelab/aws-route53.md')."
|
||||
),
|
||||
},
|
||||
```
|
||||
|
||||
### `app/rag/prompts.py`
|
||||
|
||||
No changes needed — the prompt text references the tool names but not the output format.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `app/rag/agent.py` — `list_catalog` formatting + `AGENT_TOOLS` descriptions
|
||||
|
||||
## Verification
|
||||
|
||||
- `uv run pytest tests/unit/test_agent.py -v --no-cov` — the tool shape tests (`test_agent_tools_names_and_parameters`) check the description strings, so they must be updated to match.
|
||||
- `uv run ruff check app/rag/agent.py && uv run pyright app/rag/agent.py` — lint + types clean.
|
||||
@@ -0,0 +1,85 @@
|
||||
# Task 02 — Update mock parsing + unit tests for the new format
|
||||
|
||||
**Goal:** Update `tests/e2e/mock_llm.py::_catalog_docs` to parse the new `source: X | path: Y | title: Z` format, and update `tests/unit/test_agent.py` to match the new tool descriptions.
|
||||
|
||||
## Changes
|
||||
|
||||
### `tests/e2e/mock_llm.py` — `_catalog_docs` function
|
||||
|
||||
The current parsing uses `head.rpartition("/")` on the `source/path` head:
|
||||
|
||||
```python
|
||||
# BEFORE
|
||||
def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
|
||||
docs: list[tuple[str, str]] = []
|
||||
for m in _messages(body):
|
||||
if m.get("role") != "tool":
|
||||
continue
|
||||
content = str(m.get("content") or "")
|
||||
if content.startswith(_READ_RESULT_PREFIX):
|
||||
continue
|
||||
for line in content.splitlines():
|
||||
head = line.split(" — ", 1)[0].strip()
|
||||
if "/" in head:
|
||||
source, _, path = head.rpartition("/")
|
||||
if source and path:
|
||||
docs.append((source, path))
|
||||
return docs
|
||||
```
|
||||
|
||||
Change to parse the new labeled format:
|
||||
|
||||
```python
|
||||
# AFTER
|
||||
def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
|
||||
docs: list[tuple[str, str]] = []
|
||||
for m in _messages(body):
|
||||
if m.get("role") != "tool":
|
||||
continue
|
||||
content = str(m.get("content") or "")
|
||||
if content.startswith(_READ_RESULT_PREFIX):
|
||||
continue
|
||||
for line in content.splitlines():
|
||||
# New format: "source: X | path: Y | title: Z"
|
||||
if not line.startswith("source: "):
|
||||
continue
|
||||
parts = line.split(" | ", 2)
|
||||
if len(parts) < 3:
|
||||
continue
|
||||
source = parts[0].removeprefix("source: ").strip()
|
||||
path_part = parts[1]
|
||||
if not path_part.startswith("path: "):
|
||||
continue
|
||||
path = path_part.removeprefix("path: ").strip()
|
||||
if source and path:
|
||||
docs.append((source, path))
|
||||
return docs
|
||||
```
|
||||
|
||||
### `tests/unit/test_agent.py` — tool description assertions
|
||||
|
||||
The test `test_agent_tools_names_and_parameters` asserts the exact description strings:
|
||||
|
||||
```python
|
||||
# BEFORE (line ~109)
|
||||
assert by_name["read_document"]["function"]["description"] == (
|
||||
"Add the full content of one more indexed document to your context"
|
||||
)
|
||||
```
|
||||
|
||||
This assertion is about the function description (not parameter descriptions), so it stays the same. But the test file may have inline assertions about the `source` and `path` parameter descriptions — check and update if any exist.
|
||||
|
||||
### `tests/unit/test_agent.py` — refusal message
|
||||
|
||||
The refusal message `"No document at S/ghost.md — check the list_documents output."` is unchanged — it reports the lookup failure, not the list format. No changes needed here.
|
||||
|
||||
## Files changed
|
||||
|
||||
- `tests/e2e/mock_llm.py` — `_catalog_docs` parsing
|
||||
- `tests/unit/test_agent.py` — only if inline assertions reference the old format (verify)
|
||||
|
||||
## Verification
|
||||
|
||||
- `uv run pytest tests/unit/test_agent.py -v --no-cov` — all agent unit tests pass.
|
||||
- `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov` — the mock's `_catalog_docs` must produce identical `(source, path)` tuples for the test catalogs.
|
||||
- `uv run ruff check tests/ && uv run pyright tests/` — lint + types clean.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Task 03 — E2E document tools validation
|
||||
|
||||
**Goal:** Run the E2E suites for agent document tools and verify they still pass with the new catalog format.
|
||||
|
||||
## What to verify
|
||||
|
||||
The E2E tests assert on:
|
||||
1. **SSE frames** — the `argument` field of `tool` events is `source/path` (e.g., `"Deployments/example-record-file.json"`). This is the tool call argument, NOT the list output format. **Unchanged.**
|
||||
2. **UI rendering** — tool lines show "Listing documents" and "Reading <source/path>". **Unchanged.**
|
||||
3. **Source chips** — display `${s.source}/${s.path}`. **Unchanged.**
|
||||
4. **Answer content** — the mock's deterministic answer quoting the read document. **Unchanged** (the mock reads the same `(source, path)` tuples from the catalog).
|
||||
5. **Query log** — sources field stores `"source/path, source/path"`. **Unchanged.**
|
||||
|
||||
The only thing that changes is the **text of the catalog** that the mock parses. The mock's `_catalog_docs` function must produce identical `(source, path)` tuples for the test catalogs, which means the tool flow classification (`_tool_flow`) and the read step will work identically.
|
||||
|
||||
## Run commands
|
||||
|
||||
```bash
|
||||
# DB must be up: podman compose up -d db
|
||||
uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov
|
||||
uv run pytest tests/e2e/test_agent_unlimited_tools.py -v --no-cov
|
||||
```
|
||||
|
||||
## Regression suites to run
|
||||
|
||||
```bash
|
||||
uv run pytest tests/e2e/test_chat_api.py -v --no-cov
|
||||
uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov
|
||||
uv run pytest tests/e2e/test_sse_events.py -v --no-cov
|
||||
```
|
||||
|
||||
## Expected outcome
|
||||
|
||||
All tests pass without modification (aside from the mock parsing change in task 02). The catalog text format is an internal detail of the agent loop — the tool call arguments, SSE frames, UI rendering, and answer content are all unchanged.
|
||||
|
||||
## If tests fail
|
||||
|
||||
1. Check that `_catalog_docs` produces the same `(source, path)` tuples as before for the test catalogs.
|
||||
2. Verify the mock's `_tool_flow` classification still works (it calls `_catalog_docs`).
|
||||
3. Check that the SSE `argument` field is still `source/path` (it should be — `_execute_tool` builds it from `call.arguments["source"] + "/" + call.arguments["path"]`, which is unchanged).
|
||||
|
||||
## Files changed
|
||||
|
||||
- None (verification only — all changes were in tasks 01 and 02)
|
||||
@@ -0,0 +1,44 @@
|
||||
# Task 04 — Update documentation
|
||||
|
||||
**Goal:** Update all documentation that references the old `source/path — title` catalog format to the new `source: X | path: Y | title: Z` format.
|
||||
|
||||
## Files to update
|
||||
|
||||
### `README.md` — Agent document tools section (~L168)
|
||||
|
||||
```markdown
|
||||
# BEFORE
|
||||
* **`list_documents`** — lists every indexed document, one
|
||||
`source/path — title` line each (the same order as the Sources page);
|
||||
|
||||
# AFTER
|
||||
* **`list_documents`** — lists every indexed document, one
|
||||
`source: X | path: Y | title: Z` line each (the same order as the
|
||||
Sources page);
|
||||
```
|
||||
|
||||
### `.agent/user_stories/agent-document-tools.md` — Acceptance criteria (~L38)
|
||||
|
||||
```markdown
|
||||
# BEFORE
|
||||
3. `app/rag/agent.py`: the loop — budgets from
|
||||
`BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` (default 1/1); the
|
||||
`list_documents` tool returns the DB catalog
|
||||
(`source/path — title` lines, `/api/docs` order);
|
||||
|
||||
# AFTER
|
||||
3. `app/rag/agent.py`: the loop — budgets from
|
||||
`BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` (default 1/1); the
|
||||
`list_documents` tool returns the DB catalog
|
||||
(`source: X | path: Y | title: Z` lines, `/api/docs` order);
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
- `grep -n "source/path — title\|source/path - title" README.md .agent/user_stories/*.md` should return no results.
|
||||
- `uv run ruff check README.md .agent/` — lint clean (markdown files are not linted by ruff, but the command should not error).
|
||||
|
||||
## Files changed
|
||||
|
||||
- `README.md` — Agent document tools section
|
||||
- `.agent/user_stories/agent-document-tools.md` — Acceptance criteria item 3
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task 01 — The importer progress hook
|
||||
|
||||
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "Both should show current file being processed in real time, not just 'syncing' or 'uploading'."
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
`import_sources` gains an optional per-file progress callback so both long-running jobs (admin sync, upload scan — tasks 02/03) can report the file being processed right now. Existing callers pass nothing and see zero behavior or performance change.
|
||||
|
||||
## Work
|
||||
1. `app/rag/importer.py`:
|
||||
- Add an optional keyword argument to `import_sources`: `progress: Callable[[str, str, int, int], None] | None = None` — signature `(source, rel_posix_path, done, total)`.
|
||||
- **Semantics:** called once per importable file, immediately before `await _index_file(...)`, with `done` = the 1-based index of the current file **across all sources** and `total` = the total number of importable files across all `sources` roots. `rel` is the same POSIX path the doc rows use (`path.relative_to(root).as_posix()`).
|
||||
- `total` is computed **only when `progress` is provided**: pre-walk every root with the existing `iter_importable_files` (same extension/exclusion rules — directory stats only, no file reads). When `progress is None`, no pre-walk happens: existing callers (`scripts/import_docs.py`, `app/api/sync.py` until task 02, `app/api/git_sources.py` until task 03) are byte-identical in behavior and cost.
|
||||
- The `limit` debug path is unchanged: the callback still fires per processed file; `done` never exceeds the limit, `total` stays the full pre-walk count (an incomplete walk must not misreport the denominator).
|
||||
- No special handling for a raising callback — the hooks in this repo (tasks 02/03) only assign dataclass fields. Keep the loop clean; no try/except around the call.
|
||||
- Module docstring: one line noting the optional progress hook (phase 64).
|
||||
2. `tests/unit/test_importer.py` — unit tests:
|
||||
- Multi-root, multi-file: the callback receives the exact `(source, rel, done, total)` sequence (both roots interleaved in `sources` order, `total` = combined count).
|
||||
- `progress=None`: no callback, and a pre-walk sentinel (e.g., monkeypatch `iter_importable_files` with a call counter) proves the walk happens exactly as many times as before this change (no extra pass).
|
||||
- Skipped/unchanged/error files still count in the sequence (the callback fires before `_index_file`, so an `EmbeddingError` file was already reported as current).
|
||||
- `limit=`: callback fires only for processed files; `total` is still the full count.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: as above; full suite green.
|
||||
- Coverage: **>90%** on `app/` (the hook is small; the `None` path and the pre-walk path both get dedicated tests).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `import_sources(..., progress=fn)` reports every importable file in order with correct `done`/`total`; `progress=None` callers are unchanged (no extra walk).
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] No behavior change in completed work (sync/upload still pass no hook yet).
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 02 — Sync status reports the current file
|
||||
|
||||
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "Both should show current file being processed in real time, not just 'syncing' or 'uploading'."
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
`GET /api/sync/status` carries the file the admin sync is processing right now (`current_file` + `files_done`/`files_total`), so the RAG-page button (task 04) can render a live label instead of a bare "Syncing…".
|
||||
|
||||
## Work
|
||||
1. `app/api/sync.py`:
|
||||
- `SyncStatus` dataclass: add `current_file: str | None = None`, `files_done: int = 0`, `files_total: int = 0` (the four-state machine is untouched).
|
||||
- `GET /api/sync/status` response: add the three keys — `"current_file": str | null`, `"files_done": int`, `"files_total": int` (idle: `null`/`0`/`0`). The existing keys (`state`, `started_at`, `finished_at`, `detail`, `error`) are unchanged, so the current UI and every existing consumer keep working.
|
||||
- `_run_sync`: on start, explicitly reset the three fields (alongside the existing resets). Pass the task-01 hook to `import_sources(sources, llm, prune=True, progress=_hook)` where `_hook(source, rel, done, total)` assigns `_status.current_file = f"{source}/{rel}"`, `_status.files_done = done`, `_status.files_total = total`. The module already has its single `_status` instance — the closure captures it exactly like the existing state assignments.
|
||||
- Terminal states: on `success` and on `failed` set `_status.current_file = None` (keep the final `files_done`/`files_total` — the run's last position is useful context and costs nothing). The clone/pull phase before the import reports no file yet (`current_file` stays `None`) — per ASSUMPTION A4 the label then shows just "Syncing…".
|
||||
- Module docstring: the status paragraph gains one line on the progress fields (phase 64).
|
||||
2. `tests/unit/test_sync_button.py` — unit tests (extend the existing sync-API unit coverage, house fixture for the in-memory run):
|
||||
- Idle status shape: the three new keys present with `null`/`0`/`0`.
|
||||
- Mid-run: drive `_run_sync` with the existing mocked-import seam (the suite already stubs the pipeline) + a `progress`-shaped call injected through the real hook closure — `GET /api/sync/status` reports the assigned `current_file`/`files_done`/`files_total` while `state == "running"`.
|
||||
- Terminal: after success and after failure, `current_file` is `null` (counts retain the final values).
|
||||
- Backward shape: every pre-existing key in the response is unchanged (pin the full response dict on the idle state).
|
||||
|
||||
- ASSUMPTION (owner-locked 2026-09-01): A4 — per-file granularity; `source/relative/path` form; no file shown during the clone/pull phase (bare "Syncing…").
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: as above; full suite green.
|
||||
- Coverage: **>90%** on `app/` (the `_run_sync` terminal branches already have coverage — extend, don't duplicate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `GET /api/sync/status` returns `current_file`/`files_done`/`files_total` (null/0 idle), updated per file while a sync runs, `current_file` null in terminal states.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] No behavior change in completed work (the UI still shows "Syncing…" until task 04).
|
||||
@@ -0,0 +1,48 @@
|
||||
# Task 03 — The upload becomes background (202 + status endpoint)
|
||||
|
||||
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "Also the user can navigate away from the upload way before 'uploading' finishes." (+ "once the zipfile/tarball is on the server there should be a little 'successfully uploaded' notification toast")
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
`POST /api/git-sources/upload` returns **202** the moment the archive is fully received (safe on disk), and unpack → swap → row upsert → model check → import → overview run in a background task behind a new `GET /api/git-sources/upload/status` (the phase-32 `SyncStatus` pattern, including task-02-style `current_file`/counts). Navigating away mid-scan no longer aborts anything.
|
||||
|
||||
## Work
|
||||
1. `app/schemas.py` — add `UploadAccepted(BaseModel)`: `detail: str = "upload received"`, `name: str` (the safe source name). `UploadOut` is **kept** — it becomes the shape of the status `detail` on success.
|
||||
2. `app/api/git_sources.py`:
|
||||
- **`UploadStatus` dataclass** (module level, mirrors `app.api.sync.SyncStatus`): `state: Literal["idle", "running", "success", "failed"]`, `started_at`/`finished_at`, `current_file: str | None`, `files_done: int`, `files_total: int`, `detail: dict`, `error: str | None`; one instance `_upload_status`. Keep the module-level `_upload_in_progress` **bool flag** (not a task-done check) with its existing rationale — the flag is checked and set with **no await in between, BEFORE streaming**, because the handler now awaits (the 1 MiB-chunk stream) before the background task exists: a task-done check alone would let a concurrent POST slip through during the receive and start a second run. The flag is cleared in `_run_upload`'s `finally` (at the end of the background run) and on the exception path where the task was never created (wrap stream + `create_task` in try/except, clear + re-raise).
|
||||
- **`POST /upload` → `status_code=202`, `response_model=UploadAccepted`**; drop the `db: Session = Depends(get_db)` dependency from the signature (the handler no longer touches the DB — the upsert moves to the background). Inline (request) work, unchanged semantics: (1) name/format gate (422s as today, including `archive_source_name`'s messages), (2) 409 `an upload is already in progress` while the flag is held, (3) stream the upload in `_STREAM_CHUNK` chunks into the dotfile temp with the `upload_max_mb` cap (413 naming the cap, temp deleted). Then `asyncio.create_task(_run_upload(...))` and return `UploadAccepted(name=<safe name>)` — **the file is on disk; 202 is the "successfully uploaded" moment the UI toasts on (ASSUMPTION A2).**
|
||||
- **`_run_upload(name, filename, total_bytes, upload_root, temp_upload, temp_unpack)`** (module function, the phase-32 `_run_sync` shape):
|
||||
1. `_upload_status` → `running` (started_at, finished_at `None`, `current_file` `None`, counts 0, detail `{}`, error `None`).
|
||||
2. `unpack_archive(temp_upload, temp_unpack, max_bytes)` — `ArchiveUploadError` → `failed` (sanitized via the imported `_sanitize_error`), temps deleted; the compressed temp is unlinked after unpack (phase-49 locked decision: only unpacked content is kept).
|
||||
3. Zero entries → `failed` "the archive contains no files".
|
||||
4. `swap_in(temp_unpack, final_dir)` — `ArchiveUploadError` → `failed` sanitized (a failure here leaves the previous folder/row/KB untouched).
|
||||
5. Row upsert **in a short-lived `SessionLocal()`** (open/close around it, the `effective_sources`/`bump_sources_version` pattern from `app/api/sync.py` — never the request session, whose lock discipline is what the old inline `db.close()` comment guarded): by `path` (expanded `final_dir`), `kind="local"`, `url` = same path; an existing row is left as-is (`added_at` preserved); a concurrent-insert `IntegrityError` → `failed` "a local source with this path already exists: <path>" (the folder stays — the row exists, the next sync sees it).
|
||||
6. `check_models(LLMClient())` — `ModelUnavailableError` → `failed` with the sanitized message (the phase-49 503 becomes a status state; the folder/row are committed, so the next sync/re-upload retries idempotently — ASSUMPTION A5).
|
||||
7. `import_sources([final_dir], llm, prune=True, progress=_hook)` — `_hook(source, rel, done, total)` assigns `_upload_status.current_file = f"{source}/{rel}"` + the counts (task-01 hook).
|
||||
8. Change-gated `regenerate_overview(llm)` (added + updated > 0 — unchanged).
|
||||
9. The per-upload INFO log line (PLAN §9 / AGENTS.md rule 10) **moves here**, same fields as today (`upload: name=… file=… bytes_in=… files=… added=… updated=… unchanged=… pruned=… errors=… overview=… total_ms=…` — `total_ms` now the background run's duration).
|
||||
10. `success`: finished_at, `detail` = the `UploadOut` fields as a dict (`source=name`, `files`, `added`, `updated`, `unchanged`, `pruned`, `errors`, `chunks`, `overview`).
|
||||
- `CancelledError` is deliberately **not** caught (app shutdown cancels the task — the `_run_sync` rule). The `finally` cleans both temps (defensive, as today) and clears `_upload_in_progress`.
|
||||
- **`GET /upload/status`** (router dependency already admin-only): response `{"state", "started_at", "finished_at", "current_file", "files_done", "files_total", "detail", "error"}` — identical key set to `GET /api/sync/status` (ISO-8601 or null, same as there).
|
||||
- Docstrings: module docstring's upload paragraph → phase-64 contract (202 + background + status endpoint; the inline gate list stays accurate — steps 1–3 are inline, 4–10 are background); route docstring rewritten to match.
|
||||
3. `tests/unit/test_archive_upload.py` — adapt the suite to the 202 contract (keep every scenario, change the observation point from the HTTP response to the status endpoint, polling until terminal):
|
||||
- 202 + `UploadAccepted` body (`detail` + safe `name`); the temp upload file exists on disk at that point.
|
||||
- Inline gates unchanged: non-archive extension 422 (accepted set named), unsafe name 422, over-cap 413 (cap named, temp deleted), 409 while a run is in flight (flag still held).
|
||||
- Success: status `success`, `detail` carries the `UploadOut` fields with correct counts, the `kind="local"` row exists (and a re-upload under the same name preserves it — the existing in-place-replace scenario, now observed via the second run's status), the folder is in place, no dotfile temps left in `upload_root`.
|
||||
- Corrupt/traversal archive → status `failed`, sanitized error, KB + rows untouched, temps deleted.
|
||||
- Zero-entry archive → status `failed` "the archive contains no files".
|
||||
- Model down (stub `check_models` raising `ModelUnavailableError`) → status `failed` sanitized; folder + row exist (idempotent-retry precondition, ASSUMPTION A5).
|
||||
- Mid-run: `current_file`/`files_done`/`files_total` reported while `running` (same hook-injection seam as task 02), `current_file` null in terminal states.
|
||||
- Re-upload while a run is in flight → 409 (the flag, not task-done, is the gate — pin a second POST during the receive window in the existing flag test, or extend it).
|
||||
|
||||
- ASSUMPTION (owner-locked 2026-09-01): A1 — 202 = "archive safely on disk"; the scan runs server-side afterwards; one upload at a time. A5 — post-202 failures are status states (`failed` + sanitized error), never HTTP errors; pre-swap failures leave KB/folders/rows untouched, post-swap failures keep folder + row for an idempotent retry.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: as above (the adapted suite is this task's gate); full suite green.
|
||||
- Coverage: **>90%** on `app/` — every background branch (unpack fail, zero-entry, swap fail, IntegrityError, model down, success, cancel-cleanup) gets a dedicated test; the `_run_upload` failure branches are the new code.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `POST /api/git-sources/upload` answers 202 after the receive; `GET /api/git-sources/upload/status` mirrors the sync status shape (incl. `current_file`) and is admin-only (403 anonymous).
|
||||
- [ ] A scan keeps running (and completes) after the client disconnects — unit-level: the background task is created before the response and its outcome lands in `_upload_status`.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] The `test_archive_upload_sources.py` E2E suite will be adapted in task 06 (it is expected to fail until then — noted, not fixed here).
|
||||
@@ -0,0 +1,38 @@
|
||||
# Task 04 — RAG-page sync button: live file, upload-scan awareness
|
||||
|
||||
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "Both should show current file being processed in real time" + "Need to trigger the syncing button animation if the user clicks on sources after clicking upload"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
The sync button on `/sources.html` shows the current file while a **sync** runs ("Syncing… <file>") and, while an **upload scan** is in flight, enters the same running animation showing the upload's current file ("Importing <file>"); when the upload scan finishes the button settles and the catalog refreshes.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/sources.js`:
|
||||
- **Label builder** `fmtSyncLabel(kind, currentFile, done, total)` — `kind` is `"sync" | "upload"`: `"sync"` → `Syncing…` + (file ? ` ${file}` : "") ; `"upload"` → `Importing` + (file ? ` ${file}` : "") ; both append ` (done/total)` only when `total > 0` (A4 — no file yet during clone/pull or unpack → bare prefix). The full untruncated path is what the status endpoints report; the label shows it, truncated by CSS.
|
||||
- **`enterSyncRunningState(kind, currentFile, done, total)`** (the existing no-arg version gains parameters): same mechanics as today (disabled, `aria-busy`, icon `is-spinning`, no `is-error`) plus: `syncBtn.title = currentFile` (removed when null — full path on hover) and `syncResult.textContent = fmtSyncLabel(...)` **without** the truncation, so the existing `role="status"` `#sync-result` announcer reads the full live path to screen readers.
|
||||
- **Unified polling** — `startSyncPolling`'s tick now fetches **both** `GET /api/sync/status` and `GET /api/git-sources/upload/status` (both admin-only; the 403 branch on the sync fetch already hides the button — a 403 on the upload fetch is treated as "no upload", a network blip retries next tick). Decision tree, in order, per tick:
|
||||
1. sync `running` → `enterSyncRunningState("sync", sync.current_file, sync.files_done, sync.files_total)`; reschedule.
|
||||
2. else upload `running` → `enterSyncRunningState("upload", upload.current_file, upload.files_done, upload.files_total)`; reschedule. ← *the "clicking on sources after clicking upload" contract (A3).*
|
||||
3. else sync `success` → `applySyncSuccess` (unchanged); stop.
|
||||
4. else sync `failed` → `applySyncFailure` (unchanged); stop.
|
||||
5. else upload `success` → `settleSyncButton("Sync sources")`; `syncResult.textContent = ""` (A3 — the sync-result line never renders upload counts; they live on the Sources page); `hideSyncError()`; `emitSyncStatus({ state: "idle" })`; `loadDocs()` (the KB changed — the new documents must appear); stop.
|
||||
6. else upload `failed` → `settleSyncButton("Sync sources")`; `syncResult.textContent = ""`; `hideSyncError()`; `emitSyncStatus({ state: "idle" })`; stop (the failure UI is the Sources page's error banner — A3).
|
||||
7. else both idle → `applySyncIdle`; stop.
|
||||
- **`startSync` click handler**: the 202/409 branch calls `enterSyncRunningState("sync", null, 0, 0)` (the existing `emitSyncStatus({ state: "running" })` dedup via `lastSyncState` stays).
|
||||
- **`initSyncButton` load-time re-attach**: unchanged for sync states; when the sync state is `idle`, additionally fetch the upload status — if it is `running`, `enterSyncRunningState("upload", …)` + `emitSyncStatus({ state: "running" })` + `startSyncPolling()` (a terminal upload is a no-op: the boot-time `loadDocs()` already shows the current catalog).
|
||||
- Header comment: the sync-button block documents the phase-64 contract (live file label, the two-job decision tree, the A3 settle behavior).
|
||||
2. `frontend/assets/styles.css` — `.sync-label`: truncate long paths — `display: inline-block; max-width: min(16rem, 40vw); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; vertical-align: bottom;` (the full path stays in the button `title` + `#sync-result`).
|
||||
3. `frontend/sources.html` — no structural change; the sync-button comment block updated to the phase-64 contract.
|
||||
4. Frontend source pins — new `tests/unit/test_frontend_sync_upload.py` (the existing `tests/unit/test_frontend_feedback.py` pin style), covering: the `fmtSyncLabel` contract (both kinds, file present/absent, counts only when `total > 0`); `enterSyncRunningState` writing the full path into `title` + `#sync-result`; the tick decision tree (running-upload → "Importing" label; upload success → settle + `loadDocs` + no sync-result line; upload failed → settle, no error banner; both idle → `applySyncIdle`); the re-attach branch (sync idle + upload running → running state + polling starts).
|
||||
|
||||
- ASSUMPTION (owner-locked 2026-09-01): A3 — during an upload scan the sync button animates with "Importing <file>"; on upload completion it settles to "Sync sources", clears the result line, and refreshes the catalog; the sync-result line never renders upload counts, and an upload failure is not surfaced on the RAG page. A4 — per-file granularity, `source/relative/path`, ellipsis truncation, bare prefix before the first file.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: source pins as above; full suite green.
|
||||
- Coverage: **>90%** on `app/` (unchanged — frontend-only task).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] During a sync run the button label shows `Syncing… <file> (n/m)`; during an in-flight upload scan it shows `Importing <file> (n/m)` with the spinning icon.
|
||||
- [ ] On upload completion the button settles, the catalog refreshes, and no upload counts appear in `#sync-result`; on page load with a running upload the button re-attaches to the running state.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] No behavior change in completed work (plain sync run renders as before plus the file in the label).
|
||||
@@ -0,0 +1,39 @@
|
||||
# Task 05 — Upload UI: success toast on receive, live "Processing…" label
|
||||
|
||||
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "once the zipfile/tarball is on the server there should be a little 'successfully uploaded' notification toast so the user knows they can navigate away"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
On `/git-sources.html` the upload flow follows the new 202 contract: the moment the archive is received the page shows a **"Successfully uploaded — <file>" toast** (safe to navigate away), the button switches to a live **"Processing… <file> (n/m)"** label driven by `GET /api/git-sources/upload/status` polling, and a page reload mid-scan re-attaches instead of dead-ending.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/git-sources.js`:
|
||||
- **`showUploadToast(message)`** — page-local, the phase-55 share-toast pattern from `frontend/assets/app.js` (lazy-created single node, class `toast`, `.is-visible` toggles the entry transition, `role="status" aria-live="polite"`, ~5 s auto-dismiss, a new toast replaces a pending one — clear the prior timer, never stack). Uses the existing `.toast` CSS as-is.
|
||||
- **Submit handler** (the existing one, reworked):
|
||||
- The no-file guard, the error/result clearing, and the "Uploading…" transfer label stay (the transfer is now short — 202 arrives when the receive finishes).
|
||||
- **202** (parse the `UploadAccepted` body for `name`; body parse failure degrades to the picked file's name): `showUploadToast(\`Successfully uploaded — ${file.name}\`)`; `uploadFileInput.value = ""`; enter the **processing state** (`uploadBtn.disabled = true`, `uploadBtn.textContent = "Processing…"`, `uploadBtn.title = ""`) and `startUploadPolling()`.
|
||||
- **409** (`an upload is already in progress`): **no error banner** — enter the processing state + `startUploadPolling()` (re-attach; never stale). The old "server detail inline" branch does NOT apply to 409 anymore.
|
||||
- **other non-ok** (422 name/format, 413 cap, 5xx): the existing `apiDetail` error banner, file selection KEPT (the existing re-pick convention), button restored in `finally`.
|
||||
- **network failure** (`catch`): the existing "Could not reach the server" banner, button restored.
|
||||
- `finally` restores the button **only when no polling is active** (while `startUploadPolling` owns the button, it stays disabled/Processing — the §7.4 never-stale rule).
|
||||
- **`startUploadPolling()`** — 2 s cadence (the `SYNC_POLL_MS` house value, local `const UPLOAD_POLL_MS = 2000`), single timer, one at a time (guard against double-start): each tick fetches `GET /api/git-sources/upload/status`:
|
||||
- `running` → `uploadBtn.textContent = "Processing…" + (current_file ? \` ${current_file}\` : "") + (files_total > 0 ? \` (${files_done}/${files_total})\` : "")`; `uploadBtn.title = current_file || ""` (full path on hover); reschedule. (A4 — unpack phase shows bare "Processing…".)
|
||||
- `success` → stop; `uploadResult` line = `fmtUploadResult(detail)` (the existing helper reads exactly these keys); `announce(\`Archive uploaded: ${detail.source}.\`)`; `uploadFileInput.value = ""`; restore the button (enabled, "Upload & scan", title removed); `loadSources()` (the row lands / refreshes). **No toast here** — it already fired at 202.
|
||||
- `failed` → stop; `uploadError` banner = `status.error` (sanitized server-side); restore the button; `loadSources()` (post-swap failures keep the row — the list state may have changed; the file selection is kept for a one-click re-upload).
|
||||
- `idle` → stop; restore the button (defensive — a started run never returns to idle).
|
||||
- **Boot re-attach** — in the admin branch where `loadSources()` runs at boot, fetch the upload status once: `running` → processing state + `startUploadPolling()`; `success` → render the last result line only (no announce, no toast); `failed` → the error banner; `idle` → nothing.
|
||||
- Header comment: the upload block rewritten to the phase-64 contract (202 + toast + polling + re-attach; the phase-49 synchronous paragraph marked superseded).
|
||||
2. `frontend/git-sources.html` — no structural change (the toast node is JS-created, phase-55 pattern); the phase-49 form comment updated to the phase-64 contract.
|
||||
3. Frontend source pins — extend `tests/unit/test_frontend_sync_upload.py`: the toast contract (fires on 202 with `Successfully uploaded — <name>`, `role="status"`, auto-dismiss timer, single-node reuse); the processing-label builder (file present/absent, counts only when `total > 0`, full path in `title`); 409 → processing state + polling (NOT the error banner); the polling decision tree (success → result line + `loadSources`, no toast; failed → error banner + `loadSources`; idle → restore); the `finally` never-restoring-while-polling guard; the boot re-attach branches.
|
||||
|
||||
- ASSUMPTION (owner-locked 2026-09-01): A2 — the toast fires at 202 on this page, auto-dismisses, and is success-only (failures use the existing `#archive-upload-error` banner). A4 — bare "Processing…" during the unpack phase; `source/relative/path` granularity afterwards.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: source pins as above; full suite green.
|
||||
- Coverage: **>90%** on `app/` (unchanged — frontend-only task).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] Uploading an archive shows the "Successfully uploaded — <file>" toast as soon as the 202 arrives — before the scan finishes — and the button then tracks the scan with a live file label.
|
||||
- [ ] Reloading `/git-sources.html` mid-scan resumes the Processing state (no error, no second upload); a finished/failed run re-renders its result/banner.
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] No behavior change in completed work (the 422/413 error paths, the re-upload flow, and the source list are untouched apart from the new async flow).
|
||||
Reference in New Issue
Block a user