feat(sources): real-time file progress for sync and upload — background upload with success toast

This commit is contained in:
2026-09-01 23:51:43 -04:00
parent cddc84c7db
commit 4677d86f49
103 changed files with 5914 additions and 456 deletions
@@ -1,62 +0,0 @@
# Phase 61 — Stale UI Copy (retire the homelab-era text)
**Source:** `TODO.md` L4 — "Clean up the UI, there's text that references old features or talks about the way BOR used to work. For example the page-sub text in knowledge base references !/Homelab and ~/Deployments but those aren't the only sources and sometimes aren't included."
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-01)
**Context:** Verified copy inventory (audited 2026-09-01, all line numbers as of this writing):
- `frontend/index.html` L6 — meta description "Ask Brain of Reese anything about the homelab and deployments."; L199–201 — empty-state sub "I've read through the homelab and deployment notes — ask me anything, and I'll point you at the exact doc. You've got this."; L231 — composer `placeholder="Ask me about the homelab…"`.
- `app/config.py` L265–270 — the DEFAULT suggestion chips: 3 of 4 are owner-corpus-specific ("How is my Kubernetes cluster set up?", "What's my backup strategy?", "How do I deploy a new service?", "What's currently running in the homelab?"). Rendered as empty-state chips by `app.js` from `GET /api/suggestions`.
- `frontend/sources.html` L110–114 — the KB page-sub cited in the TODO: "Every `*.md` file indexed from `~/Homelab` and `~/Deployments`. Press **Sync sources** to clone the repos and re-import." Stale on three counts: sources are now git repos + local directories + uploaded archives (phases 28/38/49); import extensions are configurable, not only `*.md` (phase 56); Sync pulls the configured git repos and imports local dirs — it does not clone "the repos" in general.
- `frontend/git-sources.html` L175 — form example `placeholder="https://github.com/you/homelab.git"`.
- ALL 9 pages — footer `<span>Powered by Reese's self-hosted models</span>` (`index.html` L245, `sources.html` L186, `tuning.html` L148, `document.html` L150, `git-sources.html` L243, `history.html` L173, `login.html` L131, `shared.html` L147, `doc-edit.html` L118).
- OUT OF SCOPE (owner-locked A3): code/HTML *comments* mentioning "homelab" (`frontend/assets/app.js` L180/L1398/L1419, `frontend/assets/history.js` L39/L244, `frontend/index.html` L176) — not user-visible; left untouched to keep the diff minimal.
- The phase-39 brand layer (`frontend/assets/brand.js`) already rewrites every occurrence of the literal "Brain of Reese" (titles, `.brand-text`, text nodes, aria/placeholder/meta attributes) — so strings that embed the BRAND name (e.g. the visually-hidden label "Ask Brain of Reese a question", `index.html` ~L227) stay as-is: the layer keeps them in sync when `BOR_APP_NAME` is set.
- No existing test pins the exact default chip text (`tests/integration/test_api.py` L75–81 asserts only shape: a list, `len >= 3`, non-empty strings).
- `.agent/` is UNTRACKED (owner instruction, commit 281f355) — the phase commit stages `frontend/ tests/ app/` only.
- The E2E shared conftest server boots with the default env — this phase's E2E asserts the NEW neutral copy against that shared server.
## Objective
Every user-visible string in the app describes how BOR works TODAY (any configured sources — git, local directories, uploads — with configurable extensions) instead of the owner's original `~/Homelab` + `~/Deployments` deployment: accurate, neutral defaults on all pages, with the exact replacement copy locked below.
## Dependencies
- `60_sticky_navbar` (complete, preceding — no functional dependency; ordering by number)
## Tasks
1. `01_chat_page_copy.md` — neutralize the chat page (`index.html` meta description, empty-state sub, composer placeholder) + the default suggestion chips in `app/config.py`.
2. `02_sources_pages_copy.md` — rewrite the KB page-sub + review the remaining page-subs and the git-sources example URL against current behavior.
3. `03_footer_defaults.md` — all 9 footers → neutral default text, tagged `class="footer-text"` (the hook phase 62's env var drives).
4. `04_tests_and_commit.md` — unit text pins, dedicated E2E `tests/e2e/test_stale_ui_copy.py`, regression suites, one atomic commit.
## Testing & Quality
- Unit pins (house style, `tests/unit/test_frontend_brand.py` pattern — read the HTML/`config.py` as text): the old literals are GONE, the locked replacements are present, all 9 pages carry `.footer-text`.
- E2E (mandatory, house rule): `tests/e2e/test_stale_ui_copy.py`, run in isolation against the shared conftest server — no visible page text contains "homelab"/"deployment" (case-insensitive), placeholder/meta/footer/chips match the locked copy.
- Coverage: **>90%** on `app/` (validate.sh gate — this phase changes one Settings default, so the gate is meaningful, not ceremonial).
## Completion Criteria
- [ ] No user-visible "homelab"/"deployment(s)" text on any of the 9 pages (case-insensitive; code comments excepted per A3) — proven by the E2E body-text scan on chat + sources pages and the DOM assertions.
- [ ] KB page-sub describes the CURRENT source model (git repos + local directories + uploaded archives; Sync pulls/imports).
- [ ] All 9 footers read the locked neutral default inside `<span class="footer-text">`.
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `uv run pytest tests/e2e/test_stale_ui_copy.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_smoke.py`, `test_configurable_brand.py`, `test_header_consistency.py`, `test_dark_tech_theme.py`.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/` (`.agent/` stays untracked).
## Locked decisions
- **Owner-locked (2026-09-01, roadmap confirmation, A1):** exact replacement copy —
- meta description (`index.html` L6): `Ask anything about your indexed documents — every answer cites the exact doc.`
- empty-state sub (`index.html` L199–201): `I've read through your documents — ask me anything, and I'll point you at the exact doc. You've got this.`
- composer placeholder (`index.html` L231): `Ask me anything…`
- KB page-sub (`sources.html` L110–114): `Every file indexed from your configured sources — git repositories, local directories, and uploaded archives. Press <strong>Sync sources</strong> to pull the latest and re-import.`
- footer (all 9 pages): `Powered by self-hosted models`
- **Owner-locked (2026-09-01, roadmap confirmation, A2):** the four default suggestion chips become —
1. `What documents are in the knowledge base?`
2. `Which source does each answer come from?`
3. `How do I add a new source?`
4. `Summarize the most recent document.`
(operators keep `BOR_SUGGESTIONS` for their own corpus).
- **Owner-locked (2026-09-01, roadmap confirmation, A3):** the git-sources example becomes `https://github.com/you/your-repo.git`; code/HTML comments mentioning "homelab" are left untouched (not user-visible).
## Commit
```bash
git add frontend/ tests/ app/ && git commit --no-gpg-sign -m "fix(web): retire the stale homelab-era copy — neutral, accurate defaults on every page"
```
@@ -1,39 +0,0 @@
# Task 01 — Neutralize the chat page copy + default suggestion chips
**Phase:** `61_stale_ui_copy` · **Source:** `TODO.md:4` — "Clean up the UI, there's text that references old features or talks about the way BOR used to work…"
**Story:** n/a (TODO-derived)
## Objective
The chat page (the app's front door) and the default onboarding chips describe the app generically — "your documents / your knowledge base" — instead of the owner's homelab, so a first-time deployer sees copy that fits THEIR corpus from the first render.
## Work
1. `frontend/index.html`:
- L6 — the `<meta name="description">` content becomes exactly (A1):
`Ask anything about your indexed documents — every answer cites the exact doc.`
- L199–201 — the `.empty-state-sub` paragraph text becomes exactly (A1):
`I've read through your documents — ask me anything, and I'll point you at the exact doc. You've got this.`
(keep the surrounding markup; the "You've got this." closer survives by design).
- L231 — the `#message-input` `placeholder` becomes exactly (A1): `Ask me anything…`
- Leave UNTOUCHED: the visually-hidden `<label>` "Ask Brain of Reese a question" (~L227) and any string embedding the literal "Brain of Reese" — the phase-39 brand layer (`brand.js` attribute pass) keeps those in sync with `BOR_APP_NAME`.
2. `app/config.py` L265–270 — the `suggestions` default list becomes exactly (A2):
```python
suggestions: list[str] = [
"What documents are in the knowledge base?",
"Which source does each answer come from?",
"How do I add a new source?",
"Summarize the most recent document.",
]
```
(the `BOR_SUGGESTIONS` JSON env override is unchanged — operators still override per corpus).
3. No other file changes in this task (sources pages and footers are tasks 02/03; the unit pins land in task 04).
## Testing & Quality
- No existing test pins the old default chip text (`tests/integration/test_api.py` L75–81 asserts shape only: list, `len >= 3`, non-empty strings — the new 4-chip default satisfies it).
- `uv run pytest` green (full unit + integration); `uv run ruff check . && uv run pyright` clean.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] `grep -n "homelab" frontend/index.html` matches ONLY the L176 code comment (A3) — no user-visible homelab text remains in the chat template.
- [ ] `grep -n "Ask me anything" frontend/index.html` shows the new placeholder (exactly one user-visible occurrence).
- [ ] `app/config.py` default chips == the four locked (A2) strings; `uv run pytest tests/integration/test_api.py -v` green.
- [ ] Full suite green, lint + types clean, coverage >90%.
@@ -1,37 +0,0 @@
# Task 02 — Rewrite the Knowledge-base copy to the current source model
**Phase:** `61_stale_ui_copy` · **Source:** `TODO.md:4` — "…the page-sub text in knowledge base references !/Homelab and ~/Deployments but those aren't the only sources and sometimes aren't included."
**Story:** n/a (TODO-derived)
## Objective
The Sources/KB page — the page the TODO cites by name — and the remaining page-subs describe the source model as BOR implements it TODAY: configured git repositories, local directories, and uploaded archives (phases 28/38/49), with import extensions set by `BOR_IMPORT_EXTENSIONS` (phase 56) — not "every `*.md` in `~/Homelab` and `~/Deployments`".
## Work
1. `frontend/sources.html` L110–114 — the `<p class="page-sub">` becomes exactly (A1), keeping the `<strong>` around Sync sources:
```html
<p class="page-sub">
Every file indexed from your configured sources — git repositories,
local directories, and uploaded archives. Press <strong>Sync sources</strong>
to pull the latest and re-import.
</p>
```
(line-wrapped as in the original; the `<code>` tags for `~/Homelab` / `~/Deployments` are deleted, not reworded).
2. `frontend/git-sources.html` L175 — the repo-URL form example becomes (A3):
`placeholder="https://github.com/you/your-repo.git"`
3. Verification read-through (fix only what is factually stale; do not reword accurate copy):
- `frontend/sources.html` — the sync result/error copy (`#sync-result` announcer strings in `frontend/assets/sources.js`, the `#sync-error-banner` text): confirm it matches what Sync actually does (pulls configured git repos, imports local directories, re-imports; "no sources configured" loud failure). The phase-35+ error strings already name the real failure modes — expect no change; record the outcome in the commit message body only if something was fixed.
- The remaining page-subs — `tuning.html` L104–108 (Global Tuning), `history.html` L106–110 (Saved chats), `doc-edit.html` L38–42 (Edit doc), `git-sources.html` L135–139 (Git sources): each describes current behavior (verified at audit time) — expect no change; if any is stale against current behavior, fix it with neutral, accurate wording and note it in the commit message.
- The anonymous sources gate copy (`frontend/sources.html` `#sources-gate`) — confirms what stays public vs. locked; accurate — expect no change.
4. Do NOT touch code/HTML comments mentioning "homelab" anywhere (A3).
## Testing & Quality
- No logic changes in this task — the full suite stays green unchanged; `uv run ruff check . && uv run pyright` clean.
- Coverage: **>90%** on `app/` (validate.sh gate).
- The text pins for the new copy land in task 04 (unit + E2E).
## Completion Criteria
- [ ] `grep -n "Homelab\|Deployments" frontend/sources.html` is empty.
- [ ] `grep -n "homelab" frontend/git-sources.html` is empty.
- [ ] The KB page-sub reads the locked (A1) text, including `<strong>Sync sources</strong>`.
- [ ] The verification read-through is done and its outcome (changed / no change per page) is recorded in the task completion report.
- [ ] Full suite green, lint + types clean.
@@ -1,32 +0,0 @@
# Task 03 — Neutral footer default on all 9 pages (+ the `.footer-text` hook)
**Phase:** `61_stale_ui_copy` · **Source:** `TODO.md:4` — "Clean up the UI, there's text that references old features or talks about the way BOR used to work…"
**Story:** n/a (TODO-derived)
## Objective
The footer line — duplicated verbatim on all 9 pages — stops naming the owner ("Powered by **Reese's** self-hosted models") and reads the locked neutral default; each footer span is tagged `class="footer-text"` so phase 62's `BOR_FOOTER_TEXT` env var has one stable selector to drive on every page.
## Work
1. On ALL 9 pages, replace the identical footer span (audit-verified locations):
```html
<span>Powered by Reese's self-hosted models</span>
```
with exactly (A1):
```html
<span class="footer-text">Powered by self-hosted models</span>
```
Locations: `frontend/index.html` L245 · `frontend/sources.html` L186 · `frontend/tuning.html` L148 · `frontend/document.html` L150 · `frontend/git-sources.html` L243 · `frontend/history.html` L173 · `frontend/login.html` L131 · `frontend/shared.html` L147 · `frontend/doc-edit.html` L118.
- The `<span class="footer-version" id="app-version"></span>` siblings (where present) are untouched.
- No CSS change: `.footer-inner > span` is unstyled inline text today; adding the class is selector-only.
2. Sanity: `grep -rn "Reese's" frontend/` must be empty after this task (the possessive form existed ONLY in these footers — the brand layer's literal is "Brain of Reese", which is a different string and stays).
## Testing & Quality
- No logic changes — full suite green unchanged; `uv run ruff check . && uv run pyright` clean.
- Coverage: **>90%** on `app/` (validate.sh gate).
- The 9-page `.footer-text` pin lands in task 04's unit test.
## Completion Criteria
- [ ] All 9 pages carry `<span class="footer-text">Powered by self-hosted models</span>` (exactly 9 occurrences: `grep -rc 'class="footer-text"' frontend/*.html`).
- [ ] `grep -rn "Reese's" frontend/` is empty.
- [ ] The phase-39 brand layer is unaffected: `uv run pytest tests/unit/test_frontend_brand.py tests/e2e/test_configurable_brand.py -v --no-cov` — the unit file green; (the E2E needs DB up — run it in task 04's regression pass instead if the DB is down).
- [ ] Full suite green, lint + types clean.
@@ -1,41 +0,0 @@
# Task 04 — Unit text pins + dedicated E2E + regression pass + atomic commit
**Phase:** `61_stale_ui_copy` · **Source:** `TODO.md:4` — "Clean up the UI, there's text that references old features or talks about the way BOR used to work…"
**Story:** n/a (TODO-derived)
## Objective
The new neutral copy is pinned at two layers so it can never silently regress: unit text pins (no browser, house style) and one dedicated Playwright suite asserting what a visitor actually SEES on the shared default server — then one atomic commit.
## Work
1. `tests/unit/test_stale_ui_copy.py` (NEW — house pattern `tests/unit/test_frontend_brand.py`: read files as text, assert substrings):
- Negative pins (GONE): `"the homelab and deployments"` (meta), `"the homelab and deployment notes"` (empty-state sub), `"Ask me about the homelab"` (placeholder), `"~/Homelab"` and `"~/Deployments"` (KB page-sub), `"Reese's self-hosted models"` (footers), the four old default chips incl. `"What's currently running in the homelab?"` — checked against `frontend/index.html`, `frontend/sources.html`, `frontend/git-sources.html`, and `app/config.py` where they lived.
- Positive pins (PRESENT, exactly once where asserted): the five locked (A1) strings in their files; the four locked (A2) chips in `app/config.py`; `class="footer-text"` in ALL 9 HTML pages (iterate the same `HTML_PAGES` tuple as `test_frontend_brand.py`).
- The old chip strings are asserted absent from `app/config.py` (they may still appear NOWHERE else — they are retired, not relocated).
2. `tests/e2e/test_stale_ui_copy.py` (NEW — one story, one file, run in isolation, DB up; shared conftest server, default env — no second instance needed):
- Chat page (`/` or `/index.html`):
- `#message-input` placeholder === `Ask me anything…`;
- `document.querySelector('meta[name="description"]').content` === the locked (A1) meta string;
- the first `.footer-text` textContent === `Powered by self-hosted models`;
- `document.body.innerText.toLowerCase()` contains neither `homelab` nor `deployment` (the rendered empty-state sub AND the four rendered suggestion chips are covered by this scan — chips come from `GET /api/suggestions`, default list);
- the four rendered `.suggestions` chips' texts === the locked (A2) list, in order.
- Sources page (`/sources.html`): the `.page-sub` innerText contains neither `Homelab` nor `Deployments` (case-insensitive) and DOES contain `configured sources`; (the page is anonymously viewable — the catalog gate hides the table, not the page-head).
- Git sources page (`/git-sources.html`): the repo-URL input's `placeholder` attribute === `https://github.com/you/your-repo.git` (read from the DOM — the element exists even while `#git-sources-content` is `hidden` for anonymous; no sign-in needed).
- Determinism: all assertions are settled-state (static HTML + one fetch of /api/suggestions); Playwright `expect` retries ride out chip rendering.
3. Regression E2E pass (each in isolation, DB up): `test_smoke.py`, `test_configurable_brand.py`, `test_header_consistency.py`, `test_dark_tech_theme.py`.
4. One atomic commit (`.agent/` untracked — owner instruction, commit 281f355):
```bash
git add frontend/ tests/ app/ && git commit --no-gpg-sign -m "fix(web): retire the stale homelab-era copy — neutral, accurate defaults on every page"
```
Commit body: the verification read-through outcome from task 02 (which pages needed no change / what was fixed) — one line each.
## Testing & Quality
- `uv run pytest` green (unit + integration); coverage TOTAL **>90%** (validate.sh gate).
- `uv run pytest tests/e2e/test_stale_ui_copy.py -v --no-cov` green in isolation.
- `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] `tests/unit/test_stale_ui_copy.py` + `tests/e2e/test_stale_ui_copy.py` exist and are green (E2E in isolation, DB up).
- [ ] The four regression E2E suites are green in isolation.
- [ ] Full suite green, coverage >90%, lint + types clean.
- [ ] Exactly one new commit on `main`, staged paths `frontend/ tests/ app/` only, `--no-gpg-sign`.
- [ ] Phase dir moved to `.agent/phases/complete/61_stale_ui_copy/` (`.agent/` stays untracked).
@@ -1,54 +0,0 @@
# Phase 62 — UI Customization (deploy with your own look)
**Source:** `TODO.md` L3 — "Allow UI customization. This is brain of reese, but I want anyone to be able to deploy it with their name. The project should support an environment variable to call it 'Brain of <name>' along with custom message-input placeholder, custom footer-inner text, custom color themes, and anything else that an end-user might want to customize."
**Story:** n/a (TODO-derived — owner roadmap confirmation 2026-09-01; builds on the COMPLETED phase-39 capability, whose dir was never filed under `complete/` — the code + `tests/e2e/test_configurable_brand.py` exist, commit `fe55be0`)
**Context:** The app-name half of this TODO item is DONE (phase 39): `BOR_APP_NAME` → `Settings.app_name` (`app/config.py` L43) → `GET /api/config` (`app/api/config.py`, public, returns `{app_name, version, docs_repo_configured}`) → the classic `frontend/assets/brand.js` layer (parse-time `window.BOR_BRAND`, one `cache: "no-store"` fetch whose never-rejecting promise is `window.BOR_CONFIG_PROMISE`, DOM passes for title / `.brand-text` / text nodes / aria-label+placeholder+meta attributes; fetch failure ⇒ default stands + `console.warn`, the page never breaks). This phase extends that EXACT mechanism to three more settings — no new network surface, no new script, no templating engine (the frontend stays `StaticFiles(html=True)`, `app/main.py` L101):
- Composer placeholder: `#message-input` exists only on `frontend/index.html` (the chat page).
- Footer text: after phase 61, ALL 9 pages carry `<span class="footer-text">…</span>` inside `.footer-inner`.
- Theme: the entire palette is `:root` custom properties in `frontend/assets/styles.css` L7–25 (`--bg, --surface, --ink, --ink-soft, --line, --brand, --brand-soft, --brand-ink, --accent-bg, --accent-ink, --accent-line, --ok-bg, --ok-ink, --err-bg, --err-ink, --err-line`) — a theme file overriding those variables is a drop-in; no component CSS references literal colors except via the variables (spot-verify while authoring the example theme).
- No-CDN rule (AGENTS.md rule 6): theme CSS is served from the FastAPI static dir (`frontend/` in dev via `BOR_STATIC_DIR`; `/app/static` in the container).
- `Containerfile` stage 1 esbuild/minify chain + per-page HTML copy does NOT ship a new `frontend/assets/themes/` dir — one line is added (task 03, A7).
- Contract tests that must be updated when `/api/config` grows: `tests/integration/test_api.py` L21–46 (exact key set, two tests) and `tests/e2e/test_configurable_brand.py` L150 + L158 (exact key set); `tests/unit/test_frontend_brand.py` pins the brand layer.
- The phase-39 E2E boots its own second instance (`testy_server`, `APP_PORT + 1`, `BOR_APP_NAME` override) — phase 62's E2E reuses that pattern on a distinct port (task 05).
- `.agent/` is UNTRACKED (owner instruction, commit 281f355) — the phase commit stages `app/ frontend/ tests/ .env.example README.md Containerfile`.
## Objective
Anyone can deploy BOR under their own identity with env vars alone: the app name (already shipped) plus a custom composer placeholder (`BOR_INPUT_PLACEHOLDER`), custom footer line (`BOR_FOOTER_TEXT`), and a custom color theme (`BOR_THEME=indigo.css` → `frontend/assets/themes/indigo.css`). Unset ⇒ byte-identical to the phase-61 neutral defaults.
## Dependencies
- `61_stale_ui_copy` (todo, preceding — FUNCTIONAL: phase 61's neutral strings become these Settings' DEFAULTS, and its `.footer-text` class is the footer selector this phase drives; the "unset ⇒ byte-identical" contract is measured against phase-61 copy)
## Tasks
1. `01_config_settings.md` — three new `Settings` fields (+ fail-loud `theme` filename validator) + three new `GET /api/config` keys; update the exact-key-set contract tests (integration + the phase-39 E2E).
2. `02_brand_layer_customization.md` — `brand.js` applies placeholder / footer text / theme link from the same boot fetch; unit pins.
3. `03_theme_assets.md` — `frontend/assets/themes/indigo.css` example theme (WCAG-AA palette) + `themes/README.md` authoring guide + the one `Containerfile` line.
4. `04_docs.md` — `.env.example` + README env table + "Customizing the look" section.
5. `05_e2e_customization.md` — `tests/e2e/test_ui_customization.py` (own instance, all three vars set; default instance byte-identical; malformed-theme fail-loud), regression suites, one atomic commit.
## Testing & Quality
- Unit: `theme` validator (empty ok; bare filename ok; path/`..`/non-`.css` rejected naming the value); `GET /api/config` returns the six keys with defaults AND with overrides; brand-layer JS pins (house style).
- Integration: the config contract tests updated to the six-key set (task 01).
- E2E (mandatory, house rule): `tests/e2e/test_ui_customization.py`, run in isolation — overridden instance shows custom placeholder/footer/computed theme color on the pages; the shared default server is byte-identical (default placeholder/footer, NO theme link, built-in `--brand`).
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] `GET /api/config` returns exactly `{app_name, version, docs_repo_configured, input_placeholder, footer_text, theme}` — proven by integration + E2E.
- [ ] With `BOR_INPUT_PLACEHOLDER` / `BOR_FOOTER_TEXT` / `BOR_THEME=indigo.css` set: the chat composer placeholder, the footer line on multiple pages, and the computed `:root --brand` all reflect the overrides; a `<link href="/assets/themes/indigo.css">` is present.
- [ ] With nothing set: placeholder `Ask me anything…`, footer `Powered by self-hosted models`, no theme link, `--brand` = built-in `#f43f5e` — the phase-39/61 no-op contract holds.
- [ ] `BOR_THEME=../evil.css` (or any malformed value) refuses startup, naming the value (phase-56 house style).
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `uv run pytest tests/e2e/test_ui_customization.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_configurable_brand.py`, `test_stale_ui_copy.py`, `test_smoke.py`, `test_dark_tech_theme.py`.
- [ ] `frontend/assets/themes/` ships in the container image (Containerfile line) and serves in dev (static dir).
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/` (`.agent/` stays untracked).
## Locked decisions
- **Owner-locked (2026-09-01, roadmap confirmation, A4):** env var names — `BOR_INPUT_PLACEHOLDER`, `BOR_FOOTER_TEXT`, `BOR_THEME` (theme value = a FILENAME, e.g. `indigo.css`).
- **Owner-locked (2026-09-01, roadmap confirmation, A5):** the theme mechanism — `BOR_THEME` names a file under `frontend/assets/themes/` served statically by FastAPI (no-CDN intact); the file overrides the `:root` palette variables; the example theme is `indigo.css` (dark indigo/slate, every text/background pair ≥4.5:1); a MALFORMED `BOR_THEME` (anything not matching `^[a-z0-9_-]+\.css$`) fails startup loudly, naming the value (phase-56 house style); a theme file MISSING at runtime degrades to the built-in theme (`console.warn`, the page never breaks — the loadHealth/brand.js house style).
- **Owner-locked (2026-09-01, roadmap confirmation, A6):** the TODO's open-ended "anything else that an end-user might want to customize" is scoped to these three vars + the existing `BOR_APP_NAME` / `BOR_SUGGESTIONS`; the settings → `/api/config` → `brand.js` path is the extension point for future items (no further surface invented now).
- **Owner-locked (2026-09-01, roadmap confirmation, A7):** the Containerfile ships themes via `cp -r ./assets/themes /out/assets/themes` in stage 1 (no per-file esbuild minify — a future theme file needs no Containerfile edit).
## Commit
```bash
git add app/ frontend/ tests/ .env.example README.md Containerfile && git commit --no-gpg-sign -m "feat(web): customizable placeholder, footer text, and color theme via BOR_* env vars"
```
@@ -1,48 +0,0 @@
# Task 01 — Three new Settings + `/api/config` keys (+ validator)
**Phase:** `62_ui_customization` · **Source:** `TODO.md:3` — "…an environment variable to call it 'Brain of <name>' along with custom message-input placeholder, custom footer-inner text, custom color themes…"
**Story:** n/a (TODO-derived)
## Objective
The backend half of the customization layer: three new `BOR_`-prefixed settings with phase-61's neutral strings as DEFAULTS, surfaced through the existing public `GET /api/config` boot fetch — with the existing exact-key-set contract tests moved to the new six-key set so the suite stays green from this task on.
## Work
1. `app/config.py` — in the `# --- App ---` block (after `static_dir`, ~L47), add:
```python
# --- UI customization (phase 62, TODO L3) ---
# Defaults are the phase-61 neutral copy — UNSET => byte-identical UI.
input_placeholder: str = "Ask me anything…"
footer_text: str = "Powered by self-hosted models"
#: Theme file NAME under frontend/assets/themes/ (e.g. "indigo.css");
#: empty = the built-in dark-tech palette. Validated: bare filename
#: only — no paths, no ".." (no-CDN: served from the static dir).
theme: str = ""
```
plus a `@field_validator("theme")` (pydantic v2, `mode="after"`): empty string passes; the value must match `^[a-z0-9_-]+\.css$` (ASCII, lowercase, bare filename) or the validator raises `ValueError` naming the offending value and the allowed shape (phase-56 fail-loud house style — a typo in `.env` must kill startup, not silently 404).
- `input_placeholder` / `footer_text`: no validation beyond being strings (empty ⇒ the template default stands, handled by brand.js treating empty as "skip").
2. `app/api/config.py` — the `GET /api/config` response grows to exactly six keys:
```python
return {
"app_name": settings.app_name,
"version": settings.app_version,
"docs_repo_configured": settings.docs_configured,
"input_placeholder": settings.input_placeholder,
"footer_text": settings.footer_text,
"theme": settings.theme,
}
```
(values passed through verbatim — the frontend brand layer decides "empty = keep the template default"; NO new secrets surface: these are display strings, same public posture as `app_name`). Update the module + endpoint docstrings to cite phase 62.
3. Contract-test updates (MUST land with this task or the suite goes red):
- `tests/integration/test_api.py` L21–46: both exact-key-set assertions become `{"app_name", "version", "docs_repo_configured", "input_placeholder", "footer_text", "theme"}`; assert the DEFAULT values (`"Ask me anything…"`, `"Powered by self-hosted models"`, `""`); extend `test_config_follows_overridden_app_name`'s pattern with a `test_config_serves_ui_customization_overrides` test (fresh app with `Settings(input_placeholder=…, footer_text=…, theme="indigo.css")` → the three keys reflect the overrides).
- `tests/e2e/test_configurable_brand.py` L150 + L158: the two exact-key-set assertions grow to the six-key set (values untouched — that suite's instance has no phase-62 overrides, so the new keys are their defaults).
4. Unit tests — `tests/unit/test_config.py` (existing file, add cases): `theme` validator — `""` ok; `"indigo.css"` ok; `"Indigo.css"` rejected; `"../evil.css"` rejected; `"a/b.css"` rejected; `"indigo"` (no extension) rejected — each rejection names the value. Plus one test that `GET /api/config` (integration) default response carries the three new keys with the locked defaults.
## Testing & Quality
- `uv run pytest` green (unit + integration); `uv run ruff check . && uv run pyright` clean.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] `Settings` exposes `input_placeholder` / `footer_text` / `theme` with the locked defaults; `BOR_THEME=../evil.css` in the env fails app startup, naming the value (verified once via a unit/validator test — the E2E boots-check lands in task 05).
- [ ] `GET /api/config` returns exactly the six keys (defaults + overrides) — integration tests green.
- [ ] `tests/e2e/test_configurable_brand.py`'s key-set assertions updated (suite itself re-run in task 05's regression pass, DB up).
- [ ] Full suite green, lint + types clean, coverage >90%.
@@ -1,35 +0,0 @@
# Task 02 — `brand.js` applies the placeholder, footer text, and theme
**Phase:** `62_ui_customization` · **Source:** `TODO.md:3` — "…custom message-input placeholder, custom footer-inner text, custom color themes…"
**Story:** n/a (TODO-derived)
## Objective
The frontend half: the EXISTING `brand.js` boot-fetch pass (no new script, no new network call) applies the three new config values to the DOM — each empty ⇒ skipped, so an unset deployment stays byte-identical.
## Work
1. `frontend/assets/brand.js` — inside the SAME settled-config `.then((cfg) => …)` that `applyBrand()` runs (after the existing `app_name` block — INDEPENDENT of it: the three new keys are applied even when `app_name` is the default/empty):
- **Placeholder** — `cfg.input_placeholder` non-empty → `document.querySelector("#message-input")?.setAttribute("placeholder", cfg.input_placeholder)` (chat page only; every other page no-ops via the null guard).
- **Footer text** — `cfg.footer_text` non-empty → `document.querySelectorAll(".footer-text").forEach((el) => { el.textContent = cfg.footer_text; })` (all 9 pages, phase-61 hook; `textContent` — no escaping needed, an operator string can't inject markup).
- **Theme** — `cfg.theme` non-empty → insert a stylesheet link IMMEDIATELY AFTER the existing `<link rel="stylesheet" href="/assets/styles.css">` (find it by `rel="stylesheet"` + `href` ending in `styles.css`; append after it so the theme overrides win by cascade order):
```js
const link = document.createElement("link");
link.rel = "stylesheet";
link.href = "/assets/themes/" + cfg.theme; // filename validated server-side (task 01) — no path input can reach here via /api/config
link.id = "theme-override";
link.onerror = () => console.warn("brand: theme " + cfg.theme + " did not load — the built-in theme stands.");
stylesLink.insertAdjacentElement("afterend", link);
```
Idempotent: skip insertion if `#theme-override` already exists (the brand pass runs once, but keep the guard — the loadHealth house style: never break the page, never double-apply).
- Update the file-header contract comment: a "Phase 62 (owner-locked 2026-09-01, TODO L3)" section documenting the three keys, the empty-skip no-op contract, and the theme degradation path (A5).
- Do NOT touch the existing app_name passes (1–4) or the `BOR_DOCS_REPO_CONFIGURED` flag logic — byte-identical behavior for them.
2. `tests/unit/test_frontend_brand.py` — add pins (house style, read `brand.js` as text): markers for the three applications are present — `#message-input` placeholder application, the `.footer-text` selector, the `assets/themes/` link insertion with the `theme-override` id + `onerror` warn; the `app_name` literal default pin still holds.
## Testing & Quality
- No Python logic in this task; `uv run pytest` green (unit pins + everything else unchanged); `uv run ruff check . && uv run pyright` clean.
- Coverage: **>90%** on `app/` (validate.sh gate).
- Browser-level proof lands in task 05's E2E.
## Completion Criteria
- [ ] `brand.js` applies the three keys from the SAME settled config (no second fetch); empty values skip every application (byte-identical no-op).
- [ ] Theme link is inserted after `styles.css`, guarded by `#theme-override`, and degrades with a `console.warn` on 404 (A5).
- [ ] Unit pins green; full suite green; lint + types clean.
@@ -1,53 +0,0 @@
# Task 03 — Example theme (`indigo.css`) + authoring guide + Containerfile line
**Phase:** `62_ui_customization` · **Source:** `TODO.md:3` — "…custom color themes…"
**Story:** n/a (TODO-derived)
## Objective
A working example theme a deployer can point `BOR_THEME` at out of the box — plus the documentation that makes "write my own theme" a 15-minute job, and the one Containerfile line that ships the directory in the container image (A7).
## Work
1. `frontend/assets/themes/indigo.css` (NEW) — overrides ONLY the 8 identity variables in a single `:root` block; the semantic families (accent/ok/err) are deliberately UNTOUCHED (they encode states — deflection amber, success green, error red — and are already AA in the built-in theme; a theme that keeps them stays honest):
```css
/* Phase 62 example theme — dark indigo/slate.
Overrides the :root identity palette from styles.css; every
text/background pair meets WCAG 2.1 AA (>= 4.5:1):
ink on bg 15.8:1 · ink on surface 14.6:1 · ink-soft on surface 8.2:1
dark bg ink on brand 6.4:1 · brand-ink on surface 11.9:1.
Semantic families (accent/ok/err) inherit the built-in theme. */
:root {
--bg: #0a0e1a;
--surface: #111726;
--ink: #e6e9f0;
--ink-soft: #a8b0c8;
--line: #232c44;
--brand: #818cf8;
--brand-soft: #1a1f38;
--brand-ink: #c7d2fe;
}
```
- Before committing, re-verify the five ratios above (e.g. a 10-line python contrast calc against the built-in pairs' house comment style) — if any pair misses 4.5:1, adjust the value, not the bar (AGENTS.md rule 5).
- Spot-check that no component CSS hard-codes a color the theme should own: `grep -n "f43f5e\|#f0e6e6\|#0f0a0a" frontend/assets/styles.css` should match ONLY the `:root` palette block + comments (if a literal lives elsewhere, note it in the task report — do NOT refactor styles.css in this phase).
2. `frontend/assets/themes/README.md` (NEW) — the authoring guide, house-comment style:
- How themes load: `BOR_THEME=<file>` → `/api/config` → `brand.js` inserts `<link rel="stylesheet" href="/assets/themes/<file>">` AFTER `styles.css` (later wins the cascade — that is the whole mechanism).
- The variable table: the 8 identity variables (bg/surface/ink/ink-soft/line/brand/brand-soft/brand-ink) with the built-in values as reference, and the note that accent/ok/err are semantic and should stay.
- Rules: filename `^[a-z0-9_-]+\.css$` (lowercase, bare filename — the server validator rejects anything else at startup); one `:root` block; every text/background pair ≥4.5:1 (AGENTS.md rule 5); never white-on-brand (the built-in's documented 3.7:1 trap) — dark bg ink on brand, as the built-in does.
- Deployment: works in dev immediately (served from the static dir); in the container, rebuild the image (the `cp -r` picks up whatever is in `frontend/assets/themes/` at build time — no Containerfile edit for new files, A7).
3. `Containerfile` — stage-1 `RUN` chain: after the `styles.css` minify line, add
```
&& cp -r ./assets/themes /out/assets/themes \
```
(A7: whole directory, no per-file esbuild — future themes need no Containerfile change). No other stage changes; the runtime stage copies the whole `/out` static tree, so the dir flows through untouched.
4. Dev-server sanity (record in task report): with `BOR_THEME=indigo.css`, `http://localhost:8000/assets/themes/indigo.css` returns 200 from the static dir; with the default env the built-in palette renders (the file existing but `BOR_THEME` empty changes NOTHING — loading is opt-in via the env var, not directory scanning).
## Testing & Quality
- No Python logic; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- Coverage: **>90%** on `app/` (validate.sh gate).
- Add the theme-file unit pins to `tests/unit/test_frontend_brand.py` (or `test_stale_ui_copy.py`'s sibling if the executor prefers a new `tests/unit/test_themes.py`): the file exists, starts with a `:root` block, contains exactly the 8 `--` overrides (no other declarations), and the Containerfile contains the `cp -r ./assets/themes` line.
- Browser proof (computed `--brand`, the link tag, visual sanity) lands in task 05's E2E.
## Completion Criteria
- [ ] `frontend/assets/themes/indigo.css` + `README.md` exist; all five contrast ratios verified ≥4.5:1 (calc output in the task report).
- [ ] `Containerfile` ships `assets/themes/` (the line is in stage 1, after the styles.css minify).
- [ ] The `grep` spot-check for hard-coded identity colors is recorded (expect: `:root` block + comments only).
- [ ] Unit pins green; full suite green; lint + types clean.
@@ -1,31 +0,0 @@
# Task 04 — Document the customization surface (.env.example + README)
**Phase:** `62_ui_customization` · **Source:** `TODO.md:3` — "Allow UI customization. This is brain of reese, but I want anyone to be able to deploy it with their name…"
**Story:** n/a (TODO-derived)
## Objective
A deployer who has never seen the codebase can find, understand, and apply all four customization vars (name, placeholder, footer, theme) from the docs alone — the house standard set by the phase-39 `BOR_APP_NAME` documentation.
## Work
1. `.env.example` — in the `# --- App ---` block, right after the `BOR_APP_NAME` comment line (~L9), add (same comment style, phase-cited):
```bash
# BOR_INPUT_PLACEHOLDER=Ask me anything… # composer placeholder, chat page (phase 62)
# BOR_FOOTER_TEXT=Powered by self-hosted models # footer line on every page (phase 62)
# BOR_THEME= # filename under frontend/assets/themes/ (e.g. indigo.css) — overrides the built-in palette; empty = built-in (phase 62)
```
2. `README.md` — two edits:
- The env-var table (the `BOR_APP_NAME` row sits ~L734): add three rows immediately after it, matching that row's column style — what it affects, the default, the no-op-when-unset contract:
- `BOR_INPUT_PLACEHOLDER` | `Ask me anything…` | the chat composer placeholder; applied by `assets/brand.js` from `GET /api/config`; unset ⇒ the template default
- `BOR_FOOTER_TEXT` | `Powered by self-hosted models` | the footer line on all 9 pages (the `.footer-text` spans); same mechanism
- `BOR_THEME` | *(empty)* | a filename under `frontend/assets/themes/` (e.g. `indigo.css`) — a `:root` palette override injected after `styles.css`; the server refuses a malformed name at startup (bare `^[a-z0-9_-]+\.css$` filename); a missing file degrades to the built-in theme; unset ⇒ built-in dark-tech palette
- A short **"Customizing the look"** subsection (3–6 sentences) placed directly above or below the env table (mirror the placement of the nearest feature section — the executor picks the spot that reads best and notes it in the task report): the app ships as "Brain of Reese" but every identity string is an env var (`BOR_APP_NAME` / `BOR_INPUT_PLACEHOLDER` / `BOR_FOOTER_TEXT`), themes are plain CSS variable overrides (`frontend/assets/themes/README.md` is the authoring guide, `indigo.css` the example), and everything unset renders the defaults byte-identically.
3. No code changes in this task.
## Testing & Quality
- Docs-only; `uv run pytest` green unchanged; `uv run ruff check . && uv run pyright` clean.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] `.env.example` carries the three new commented vars in the App block with the exact default values (so copy-paste of the example reproduces the defaults).
- [ ] README env table has the three rows + the "Customizing the look" subsection; the README builds/reads cleanly (no broken table rows, links resolve — `frontend/assets/themes/README.md` exists from task 03).
- [ ] Full suite green, lint + types clean.
@@ -1,36 +0,0 @@
# Task 05 — Dedicated E2E + regression pass + atomic commit
**Phase:** `62_ui_customization` · **Source:** `TODO.md:3` — "Allow UI customization. This is brain of reese, but I want anyone to be able to deploy it with their name…"
**Story:** n/a (TODO-derived)
## Objective
One dedicated Playwright suite proves the whole contract end-to-end — an instance booted with all three customization vars shows the custom look, the default shared server is BYTE-IDENTICAL to today, and a malformed `BOR_THEME` refuses to boot — then one atomic commit and the phase is done.
## Work
1. `tests/e2e/test_ui_customization.py` (NEW — one story, one file, run in isolation, DB up). Boots a second app instance exactly the way `test_configurable_brand.py`'s `testy_server` does (same env block, same DB, mock-LLM base URL, admin auth, static dir, mock-calibrated threshold) with THREE changes: port `APP_PORT + 2` (the brand suite owns `APP_PORT + 1` — do not collide), and the env overrides
`BOR_INPUT_PLACEHOLDER="Ask the archive…"`, `BOR_FOOTER_TEXT="Custom footer line"`, `BOR_THEME="indigo.css"`.
Test → contract mapping:
1. `test_config_serves_the_overrides` — the instance's `GET /api/config` returns the three overridden values alongside the six-key set (the name stays the DEFAULT — this suite does not re-test `BOR_APP_NAME`; that is the phase-39 suite's job).
2. `test_chat_page_shows_custom_placeholder_footer_theme` (instance) — on the chat page: `#message-input` placeholder === `Ask the archive…`; the first `.footer-text` textContent === `Custom footer line`; a `link#theme-override[rel="stylesheet"][href="/assets/themes/indigo.css"]` exists in the `<head>`; `getComputedStyle(document.documentElement).getPropertyValue("--brand").trim()` === `#818cf8` (the indigo value — custom properties return the specified token, so the string compare is stable; the built-in is `#f43f5e`).
3. `test_footer_text_applies_on_other_pages` (instance) — on a second page (e.g. `/login.html`): the `.footer-text` textContent === `Custom footer line` (multi-page proof, no composer on that page — the placeholder application no-ops without error).
4. `test_default_server_is_byte_identical` (shared conftest server) — placeholder === `Ask me anything…`; footer === `Powered by self-hosted models`; NO `#theme-override` link; computed `--brand` === `#f43f5e` (the phase-39/61 no-op contract).
5. `test_malformed_theme_refuses_startup` — boot a short-lived uvicorn subprocess with `BOR_THEME="../evil.css"` (any port): the process exits non-zero within the timeout (or never starts listening) and its stderr names the value — the phase-56 fail-loud style, proven end-to-end, not just via the validator unit test.
- Determinism: settled-state assertions throughout; Playwright `expect` retries ride out the brand.js fetch (the same pattern the brand suite documents in its module docstring).
2. Regression E2E pass (each in isolation, DB up): `test_configurable_brand.py` (its key-set assertions were updated in task 01), `test_stale_ui_copy.py` (phase 61's copy contract is this phase's baseline), `test_smoke.py`, `test_dark_tech_theme.py` (the built-in palette is untouched when unset).
3. One atomic commit (`.agent/` untracked — owner instruction, commit 281f355):
```bash
git add app/ frontend/ tests/ .env.example README.md Containerfile && git commit --no-gpg-sign -m "feat(web): customizable placeholder, footer text, and color theme via BOR_* env vars"
```
Commit body: one line — the three env vars + the indigo example theme + the docs/theme-guide pointers.
## Testing & Quality
- `uv run pytest` green (unit + integration); coverage TOTAL **>90%** (validate.sh gate).
- `uv run pytest tests/e2e/test_ui_customization.py -v --no-cov` green in isolation.
- `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] `tests/e2e/test_ui_customization.py` exists (5 mapped tests) and is green in isolation (DB up).
- [ ] The four regression E2E suites are green in isolation.
- [ ] Full suite green, coverage >90%, lint + types clean.
- [ ] Exactly one new commit on `main`, staged paths `app/ frontend/ tests/ .env.example README.md Containerfile`, `--no-gpg-sign`.
- [ ] Phase dir moved to `.agent/phases/complete/62_ui_customization/` (`.agent/` stays untracked).
@@ -1,30 +0,0 @@
# Task 01 — The importer progress hook
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "Both should show current file being processed in real time, not just 'syncing' or 'uploading'."
**Story:** n/a (TODO-derived)
## Objective
`import_sources` gains an optional per-file progress callback so both long-running jobs (admin sync, upload scan — tasks 02/03) can report the file being processed right now. Existing callers pass nothing and see zero behavior or performance change.
## Work
1. `app/rag/importer.py`:
- Add an optional keyword argument to `import_sources`: `progress: Callable[[str, str, int, int], None] | None = None` — signature `(source, rel_posix_path, done, total)`.
- **Semantics:** called once per importable file, immediately before `await _index_file(...)`, with `done` = the 1-based index of the current file **across all sources** and `total` = the total number of importable files across all `sources` roots. `rel` is the same POSIX path the doc rows use (`path.relative_to(root).as_posix()`).
- `total` is computed **only when `progress` is provided**: pre-walk every root with the existing `iter_importable_files` (same extension/exclusion rules — directory stats only, no file reads). When `progress is None`, no pre-walk happens: existing callers (`scripts/import_docs.py`, `app/api/sync.py` until task 02, `app/api/git_sources.py` until task 03) are byte-identical in behavior and cost.
- The `limit` debug path is unchanged: the callback still fires per processed file; `done` never exceeds the limit, `total` stays the full pre-walk count (an incomplete walk must not misreport the denominator).
- No special handling for a raising callback — the hooks in this repo (tasks 02/03) only assign dataclass fields. Keep the loop clean; no try/except around the call.
- Module docstring: one line noting the optional progress hook (phase 64).
2. `tests/unit/test_importer.py` — unit tests:
- Multi-root, multi-file: the callback receives the exact `(source, rel, done, total)` sequence (both roots interleaved in `sources` order, `total` = combined count).
- `progress=None`: no callback, and a pre-walk sentinel (e.g., monkeypatch `iter_importable_files` with a call counter) proves the walk happens exactly as many times as before this change (no extra pass).
- Skipped/unchanged/error files still count in the sequence (the callback fires before `_index_file`, so an `EmbeddingError` file was already reported as current).
- `limit=`: callback fires only for processed files; `total` is still the full count.
## Testing & Quality
- Unit: as above; full suite green.
- Coverage: **>90%** on `app/` (the hook is small; the `None` path and the pre-walk path both get dedicated tests).
## Completion Criteria
- [ ] `import_sources(..., progress=fn)` reports every importable file in order with correct `done`/`total`; `progress=None` callers are unchanged (no extra walk).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] No behavior change in completed work (sync/upload still pass no hook yet).
@@ -1,31 +0,0 @@
# Task 02 — Sync status reports the current file
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "Both should show current file being processed in real time, not just 'syncing' or 'uploading'."
**Story:** n/a (TODO-derived)
## Objective
`GET /api/sync/status` carries the file the admin sync is processing right now (`current_file` + `files_done`/`files_total`), so the RAG-page button (task 04) can render a live label instead of a bare "Syncing…".
## Work
1. `app/api/sync.py`:
- `SyncStatus` dataclass: add `current_file: str | None = None`, `files_done: int = 0`, `files_total: int = 0` (the four-state machine is untouched).
- `GET /api/sync/status` response: add the three keys — `"current_file": str | null`, `"files_done": int`, `"files_total": int` (idle: `null`/`0`/`0`). The existing keys (`state`, `started_at`, `finished_at`, `detail`, `error`) are unchanged, so the current UI and every existing consumer keep working.
- `_run_sync`: on start, explicitly reset the three fields (alongside the existing resets). Pass the task-01 hook to `import_sources(sources, llm, prune=True, progress=_hook)` where `_hook(source, rel, done, total)` assigns `_status.current_file = f"{source}/{rel}"`, `_status.files_done = done`, `_status.files_total = total`. The module already has its single `_status` instance — the closure captures it exactly like the existing state assignments.
- Terminal states: on `success` and on `failed` set `_status.current_file = None` (keep the final `files_done`/`files_total` — the run's last position is useful context and costs nothing). The clone/pull phase before the import reports no file yet (`current_file` stays `None`) — per ASSUMPTION A4 the label then shows just "Syncing…".
- Module docstring: the status paragraph gains one line on the progress fields (phase 64).
2. `tests/unit/test_sync_button.py` — unit tests (extend the existing sync-API unit coverage, house fixture for the in-memory run):
- Idle status shape: the three new keys present with `null`/`0`/`0`.
- Mid-run: drive `_run_sync` with the existing mocked-import seam (the suite already stubs the pipeline) + a `progress`-shaped call injected through the real hook closure — `GET /api/sync/status` reports the assigned `current_file`/`files_done`/`files_total` while `state == "running"`.
- Terminal: after success and after failure, `current_file` is `null` (counts retain the final values).
- Backward shape: every pre-existing key in the response is unchanged (pin the full response dict on the idle state).
- ASSUMPTION (owner-locked 2026-09-01): A4 — per-file granularity; `source/relative/path` form; no file shown during the clone/pull phase (bare "Syncing…").
## Testing & Quality
- Unit: as above; full suite green.
- Coverage: **>90%** on `app/` (the `_run_sync` terminal branches already have coverage — extend, don't duplicate).
## Completion Criteria
- [ ] `GET /api/sync/status` returns `current_file`/`files_done`/`files_total` (null/0 idle), updated per file while a sync runs, `current_file` null in terminal states.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] No behavior change in completed work (the UI still shows "Syncing…" until task 04).
@@ -1,48 +0,0 @@
# Task 03 — The upload becomes background (202 + status endpoint)
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "Also the user can navigate away from the upload way before 'uploading' finishes." (+ "once the zipfile/tarball is on the server there should be a little 'successfully uploaded' notification toast")
**Story:** n/a (TODO-derived)
## Objective
`POST /api/git-sources/upload` returns **202** the moment the archive is fully received (safe on disk), and unpack → swap → row upsert → model check → import → overview run in a background task behind a new `GET /api/git-sources/upload/status` (the phase-32 `SyncStatus` pattern, including task-02-style `current_file`/counts). Navigating away mid-scan no longer aborts anything.
## Work
1. `app/schemas.py` — add `UploadAccepted(BaseModel)`: `detail: str = "upload received"`, `name: str` (the safe source name). `UploadOut` is **kept** — it becomes the shape of the status `detail` on success.
2. `app/api/git_sources.py`:
- **`UploadStatus` dataclass** (module level, mirrors `app.api.sync.SyncStatus`): `state: Literal["idle", "running", "success", "failed"]`, `started_at`/`finished_at`, `current_file: str | None`, `files_done: int`, `files_total: int`, `detail: dict`, `error: str | None`; one instance `_upload_status`. Keep the module-level `_upload_in_progress` **bool flag** (not a task-done check) with its existing rationale — the flag is checked and set with **no await in between, BEFORE streaming**, because the handler now awaits (the 1 MiB-chunk stream) before the background task exists: a task-done check alone would let a concurrent POST slip through during the receive and start a second run. The flag is cleared in `_run_upload`'s `finally` (at the end of the background run) and on the exception path where the task was never created (wrap stream + `create_task` in try/except, clear + re-raise).
- **`POST /upload` → `status_code=202`, `response_model=UploadAccepted`**; drop the `db: Session = Depends(get_db)` dependency from the signature (the handler no longer touches the DB — the upsert moves to the background). Inline (request) work, unchanged semantics: (1) name/format gate (422s as today, including `archive_source_name`'s messages), (2) 409 `an upload is already in progress` while the flag is held, (3) stream the upload in `_STREAM_CHUNK` chunks into the dotfile temp with the `upload_max_mb` cap (413 naming the cap, temp deleted). Then `asyncio.create_task(_run_upload(...))` and return `UploadAccepted(name=<safe name>)` — **the file is on disk; 202 is the "successfully uploaded" moment the UI toasts on (ASSUMPTION A2).**
- **`_run_upload(name, filename, total_bytes, upload_root, temp_upload, temp_unpack)`** (module function, the phase-32 `_run_sync` shape):
1. `_upload_status` → `running` (started_at, finished_at `None`, `current_file` `None`, counts 0, detail `{}`, error `None`).
2. `unpack_archive(temp_upload, temp_unpack, max_bytes)` — `ArchiveUploadError` → `failed` (sanitized via the imported `_sanitize_error`), temps deleted; the compressed temp is unlinked after unpack (phase-49 locked decision: only unpacked content is kept).
3. Zero entries → `failed` "the archive contains no files".
4. `swap_in(temp_unpack, final_dir)` — `ArchiveUploadError` → `failed` sanitized (a failure here leaves the previous folder/row/KB untouched).
5. Row upsert **in a short-lived `SessionLocal()`** (open/close around it, the `effective_sources`/`bump_sources_version` pattern from `app/api/sync.py` — never the request session, whose lock discipline is what the old inline `db.close()` comment guarded): by `path` (expanded `final_dir`), `kind="local"`, `url` = same path; an existing row is left as-is (`added_at` preserved); a concurrent-insert `IntegrityError` → `failed` "a local source with this path already exists: <path>" (the folder stays — the row exists, the next sync sees it).
6. `check_models(LLMClient())` — `ModelUnavailableError` → `failed` with the sanitized message (the phase-49 503 becomes a status state; the folder/row are committed, so the next sync/re-upload retries idempotently — ASSUMPTION A5).
7. `import_sources([final_dir], llm, prune=True, progress=_hook)` — `_hook(source, rel, done, total)` assigns `_upload_status.current_file = f"{source}/{rel}"` + the counts (task-01 hook).
8. Change-gated `regenerate_overview(llm)` (added + updated > 0 — unchanged).
9. The per-upload INFO log line (PLAN §9 / AGENTS.md rule 10) **moves here**, same fields as today (`upload: name=… file=… bytes_in=… files=… added=… updated=… unchanged=… pruned=… errors=… overview=… total_ms=…` — `total_ms` now the background run's duration).
10. `success`: finished_at, `detail` = the `UploadOut` fields as a dict (`source=name`, `files`, `added`, `updated`, `unchanged`, `pruned`, `errors`, `chunks`, `overview`).
- `CancelledError` is deliberately **not** caught (app shutdown cancels the task — the `_run_sync` rule). The `finally` cleans both temps (defensive, as today) and clears `_upload_in_progress`.
- **`GET /upload/status`** (router dependency already admin-only): response `{"state", "started_at", "finished_at", "current_file", "files_done", "files_total", "detail", "error"}` — identical key set to `GET /api/sync/status` (ISO-8601 or null, same as there).
- Docstrings: module docstring's upload paragraph → phase-64 contract (202 + background + status endpoint; the inline gate list stays accurate — steps 1–3 are inline, 4–10 are background); route docstring rewritten to match.
3. `tests/unit/test_archive_upload.py` — adapt the suite to the 202 contract (keep every scenario, change the observation point from the HTTP response to the status endpoint, polling until terminal):
- 202 + `UploadAccepted` body (`detail` + safe `name`); the temp upload file exists on disk at that point.
- Inline gates unchanged: non-archive extension 422 (accepted set named), unsafe name 422, over-cap 413 (cap named, temp deleted), 409 while a run is in flight (flag still held).
- Success: status `success`, `detail` carries the `UploadOut` fields with correct counts, the `kind="local"` row exists (and a re-upload under the same name preserves it — the existing in-place-replace scenario, now observed via the second run's status), the folder is in place, no dotfile temps left in `upload_root`.
- Corrupt/traversal archive → status `failed`, sanitized error, KB + rows untouched, temps deleted.
- Zero-entry archive → status `failed` "the archive contains no files".
- Model down (stub `check_models` raising `ModelUnavailableError`) → status `failed` sanitized; folder + row exist (idempotent-retry precondition, ASSUMPTION A5).
- Mid-run: `current_file`/`files_done`/`files_total` reported while `running` (same hook-injection seam as task 02), `current_file` null in terminal states.
- Re-upload while a run is in flight → 409 (the flag, not task-done, is the gate — pin a second POST during the receive window in the existing flag test, or extend it).
- ASSUMPTION (owner-locked 2026-09-01): A1 — 202 = "archive safely on disk"; the scan runs server-side afterwards; one upload at a time. A5 — post-202 failures are status states (`failed` + sanitized error), never HTTP errors; pre-swap failures leave KB/folders/rows untouched, post-swap failures keep folder + row for an idempotent retry.
## Testing & Quality
- Unit: as above (the adapted suite is this task's gate); full suite green.
- Coverage: **>90%** on `app/` — every background branch (unpack fail, zero-entry, swap fail, IntegrityError, model down, success, cancel-cleanup) gets a dedicated test; the `_run_upload` failure branches are the new code.
## Completion Criteria
- [ ] `POST /api/git-sources/upload` answers 202 after the receive; `GET /api/git-sources/upload/status` mirrors the sync status shape (incl. `current_file`) and is admin-only (403 anonymous).
- [ ] A scan keeps running (and completes) after the client disconnects — unit-level: the background task is created before the response and its outcome lands in `_upload_status`.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] The `test_archive_upload_sources.py` E2E suite will be adapted in task 06 (it is expected to fail until then — noted, not fixed here).
@@ -1,38 +0,0 @@
# Task 04 — RAG-page sync button: live file, upload-scan awareness
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "Both should show current file being processed in real time" + "Need to trigger the syncing button animation if the user clicks on sources after clicking upload"
**Story:** n/a (TODO-derived)
## Objective
The sync button on `/sources.html` shows the current file while a **sync** runs ("Syncing… <file>") and, while an **upload scan** is in flight, enters the same running animation showing the upload's current file ("Importing <file>"); when the upload scan finishes the button settles and the catalog refreshes.
## Work
1. `frontend/assets/sources.js`:
- **Label builder** `fmtSyncLabel(kind, currentFile, done, total)` — `kind` is `"sync" | "upload"`: `"sync"` → `Syncing…` + (file ? ` ${file}` : "") ; `"upload"` → `Importing` + (file ? ` ${file}` : "") ; both append ` (done/total)` only when `total > 0` (A4 — no file yet during clone/pull or unpack → bare prefix). The full untruncated path is what the status endpoints report; the label shows it, truncated by CSS.
- **`enterSyncRunningState(kind, currentFile, done, total)`** (the existing no-arg version gains parameters): same mechanics as today (disabled, `aria-busy`, icon `is-spinning`, no `is-error`) plus: `syncBtn.title = currentFile` (removed when null — full path on hover) and `syncResult.textContent = fmtSyncLabel(...)` **without** the truncation, so the existing `role="status"` `#sync-result` announcer reads the full live path to screen readers.
- **Unified polling** — `startSyncPolling`'s tick now fetches **both** `GET /api/sync/status` and `GET /api/git-sources/upload/status` (both admin-only; the 403 branch on the sync fetch already hides the button — a 403 on the upload fetch is treated as "no upload", a network blip retries next tick). Decision tree, in order, per tick:
1. sync `running` → `enterSyncRunningState("sync", sync.current_file, sync.files_done, sync.files_total)`; reschedule.
2. else upload `running` → `enterSyncRunningState("upload", upload.current_file, upload.files_done, upload.files_total)`; reschedule. ← *the "clicking on sources after clicking upload" contract (A3).*
3. else sync `success` → `applySyncSuccess` (unchanged); stop.
4. else sync `failed` → `applySyncFailure` (unchanged); stop.
5. else upload `success` → `settleSyncButton("Sync sources")`; `syncResult.textContent = ""` (A3 — the sync-result line never renders upload counts; they live on the Sources page); `hideSyncError()`; `emitSyncStatus({ state: "idle" })`; `loadDocs()` (the KB changed — the new documents must appear); stop.
6. else upload `failed` → `settleSyncButton("Sync sources")`; `syncResult.textContent = ""`; `hideSyncError()`; `emitSyncStatus({ state: "idle" })`; stop (the failure UI is the Sources page's error banner — A3).
7. else both idle → `applySyncIdle`; stop.
- **`startSync` click handler**: the 202/409 branch calls `enterSyncRunningState("sync", null, 0, 0)` (the existing `emitSyncStatus({ state: "running" })` dedup via `lastSyncState` stays).
- **`initSyncButton` load-time re-attach**: unchanged for sync states; when the sync state is `idle`, additionally fetch the upload status — if it is `running`, `enterSyncRunningState("upload", …)` + `emitSyncStatus({ state: "running" })` + `startSyncPolling()` (a terminal upload is a no-op: the boot-time `loadDocs()` already shows the current catalog).
- Header comment: the sync-button block documents the phase-64 contract (live file label, the two-job decision tree, the A3 settle behavior).
2. `frontend/assets/styles.css` — `.sync-label`: truncate long paths — `display: inline-block; max-width: min(16rem, 40vw); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; vertical-align: bottom;` (the full path stays in the button `title` + `#sync-result`).
3. `frontend/sources.html` — no structural change; the sync-button comment block updated to the phase-64 contract.
4. Frontend source pins — new `tests/unit/test_frontend_sync_upload.py` (the existing `tests/unit/test_frontend_feedback.py` pin style), covering: the `fmtSyncLabel` contract (both kinds, file present/absent, counts only when `total > 0`); `enterSyncRunningState` writing the full path into `title` + `#sync-result`; the tick decision tree (running-upload → "Importing" label; upload success → settle + `loadDocs` + no sync-result line; upload failed → settle, no error banner; both idle → `applySyncIdle`); the re-attach branch (sync idle + upload running → running state + polling starts).
- ASSUMPTION (owner-locked 2026-09-01): A3 — during an upload scan the sync button animates with "Importing <file>"; on upload completion it settles to "Sync sources", clears the result line, and refreshes the catalog; the sync-result line never renders upload counts, and an upload failure is not surfaced on the RAG page. A4 — per-file granularity, `source/relative/path`, ellipsis truncation, bare prefix before the first file.
## Testing & Quality
- Unit: source pins as above; full suite green.
- Coverage: **>90%** on `app/` (unchanged — frontend-only task).
## Completion Criteria
- [ ] During a sync run the button label shows `Syncing… <file> (n/m)`; during an in-flight upload scan it shows `Importing <file> (n/m)` with the spinning icon.
- [ ] On upload completion the button settles, the catalog refreshes, and no upload counts appear in `#sync-result`; on page load with a running upload the button re-attaches to the running state.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] No behavior change in completed work (plain sync run renders as before plus the file in the label).
@@ -1,39 +0,0 @@
# Task 05 — Upload UI: success toast on receive, live "Processing…" label
**Phase:** `64_sync_upload_progress` · **Source:** `TODO.md:3` — "once the zipfile/tarball is on the server there should be a little 'successfully uploaded' notification toast so the user knows they can navigate away"
**Story:** n/a (TODO-derived)
## Objective
On `/git-sources.html` the upload flow follows the new 202 contract: the moment the archive is received the page shows a **"Successfully uploaded — <file>" toast** (safe to navigate away), the button switches to a live **"Processing… <file> (n/m)"** label driven by `GET /api/git-sources/upload/status` polling, and a page reload mid-scan re-attaches instead of dead-ending.
## Work
1. `frontend/assets/git-sources.js`:
- **`showUploadToast(message)`** — page-local, the phase-55 share-toast pattern from `frontend/assets/app.js` (lazy-created single node, class `toast`, `.is-visible` toggles the entry transition, `role="status" aria-live="polite"`, ~5 s auto-dismiss, a new toast replaces a pending one — clear the prior timer, never stack). Uses the existing `.toast` CSS as-is.
- **Submit handler** (the existing one, reworked):
- The no-file guard, the error/result clearing, and the "Uploading…" transfer label stay (the transfer is now short — 202 arrives when the receive finishes).
- **202** (parse the `UploadAccepted` body for `name`; body parse failure degrades to the picked file's name): `showUploadToast(\`Successfully uploaded — ${file.name}\`)`; `uploadFileInput.value = ""`; enter the **processing state** (`uploadBtn.disabled = true`, `uploadBtn.textContent = "Processing…"`, `uploadBtn.title = ""`) and `startUploadPolling()`.
- **409** (`an upload is already in progress`): **no error banner** — enter the processing state + `startUploadPolling()` (re-attach; never stale). The old "server detail inline" branch does NOT apply to 409 anymore.
- **other non-ok** (422 name/format, 413 cap, 5xx): the existing `apiDetail` error banner, file selection KEPT (the existing re-pick convention), button restored in `finally`.
- **network failure** (`catch`): the existing "Could not reach the server" banner, button restored.
- `finally` restores the button **only when no polling is active** (while `startUploadPolling` owns the button, it stays disabled/Processing — the §7.4 never-stale rule).
- **`startUploadPolling()`** — 2 s cadence (the `SYNC_POLL_MS` house value, local `const UPLOAD_POLL_MS = 2000`), single timer, one at a time (guard against double-start): each tick fetches `GET /api/git-sources/upload/status`:
- `running` → `uploadBtn.textContent = "Processing…" + (current_file ? \` ${current_file}\` : "") + (files_total > 0 ? \` (${files_done}/${files_total})\` : "")`; `uploadBtn.title = current_file || ""` (full path on hover); reschedule. (A4 — unpack phase shows bare "Processing…".)
- `success` → stop; `uploadResult` line = `fmtUploadResult(detail)` (the existing helper reads exactly these keys); `announce(\`Archive uploaded: ${detail.source}.\`)`; `uploadFileInput.value = ""`; restore the button (enabled, "Upload & scan", title removed); `loadSources()` (the row lands / refreshes). **No toast here** — it already fired at 202.
- `failed` → stop; `uploadError` banner = `status.error` (sanitized server-side); restore the button; `loadSources()` (post-swap failures keep the row — the list state may have changed; the file selection is kept for a one-click re-upload).
- `idle` → stop; restore the button (defensive — a started run never returns to idle).
- **Boot re-attach** — in the admin branch where `loadSources()` runs at boot, fetch the upload status once: `running` → processing state + `startUploadPolling()`; `success` → render the last result line only (no announce, no toast); `failed` → the error banner; `idle` → nothing.
- Header comment: the upload block rewritten to the phase-64 contract (202 + toast + polling + re-attach; the phase-49 synchronous paragraph marked superseded).
2. `frontend/git-sources.html` — no structural change (the toast node is JS-created, phase-55 pattern); the phase-49 form comment updated to the phase-64 contract.
3. Frontend source pins — extend `tests/unit/test_frontend_sync_upload.py`: the toast contract (fires on 202 with `Successfully uploaded — <name>`, `role="status"`, auto-dismiss timer, single-node reuse); the processing-label builder (file present/absent, counts only when `total > 0`, full path in `title`); 409 → processing state + polling (NOT the error banner); the polling decision tree (success → result line + `loadSources`, no toast; failed → error banner + `loadSources`; idle → restore); the `finally` never-restoring-while-polling guard; the boot re-attach branches.
- ASSUMPTION (owner-locked 2026-09-01): A2 — the toast fires at 202 on this page, auto-dismisses, and is success-only (failures use the existing `#archive-upload-error` banner). A4 — bare "Processing…" during the unpack phase; `source/relative/path` granularity afterwards.
## Testing & Quality
- Unit: source pins as above; full suite green.
- Coverage: **>90%** on `app/` (unchanged — frontend-only task).
## Completion Criteria
- [ ] Uploading an archive shows the "Successfully uploaded — <file>" toast as soon as the 202 arrives — before the scan finishes — and the button then tracks the scan with a live file label.
- [ ] Reloading `/git-sources.html` mid-scan resumes the Processing state (no error, no second upload); a finished/failed run re-renders its result/banner.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] No behavior change in completed work (the 422/413 error paths, the re-upload flow, and the source list are untouched apart from the new async flow).