feat(sources): real-time file progress for sync and upload — background upload with success toast

This commit is contained in:
2026-09-01 23:51:43 -04:00
parent cddc84c7db
commit 4677d86f49
103 changed files with 5914 additions and 456 deletions
@@ -0,0 +1,54 @@
# Phase 63 — Unambiguous Document Listing for Agent Tools
**Source:** Live troubleshooting report (2026-09-01) — the `brain-of-reese` agent at `https://brain.experimental.reeseapps.com/` cannot read documents because the `list_documents` tool output uses an ambiguous `source/path — title` format that the LLM cannot reliably parse when paths contain `/` characters.
**Story:** `.agent/user_stories/agent-document-tools.md` (phase 37, extended)
**Context:** The agent's grounded-turn loop uses two tools: `list_documents` (returns the catalog) and `read_document` (reads one document). The catalog output is formatted as `source/path — title` (one line per document), but when a file lives inside a subdirectory of its source, the output looks like `brain-of-reese-main/homelab/aws-route53.md — aws-route53`. The LLM cannot reliably determine where `source` ends and `path` begins because both contain `/` separators. The thinking trace shows the LLM cycling through multiple failed attempts, each time guessing a different split.
The fix changes the catalog format to unambiguous `source: X | path: Y | title: Z` lines and updates the tool descriptions to tell the LLM how to parse them.
## Objective
The agent can always extract the correct `(source, path)` pair from the `list_documents` output, regardless of how many `/` characters the path contains.
## Dependencies
- `62_ui_customization` (todo, preceding — functional: no code changes in this phase touch UI or config)
## Tasks
1. `01_agent_list_format.md` — change `list_catalog` output from `source/path — title` to `source: X | path: Y | title: Z`; update the `read_document` tool descriptions to reference the new format.
2. `02_mock_and_unit_tests.md` — update `tests/e2e/mock_llm.py::_catalog_docs` to parse the new format; update `tests/unit/test_agent.py` (the "No document at" refusal line is unchanged — it names the lookup failure, not the list format).
3. `03_e2e_document_tools.md` — validate that `tests/e2e/test_agent_document_tools.py` and `tests/e2e/test_agent_unlimited_tools.py` still pass (they assert on SSE frames and UI rendering, not on the catalog text format — the tool call `argument` field `source/path` is unchanged).
4. `04_docs.md` — update any inline documentation that references the old format.
## Testing & Quality
- Unit: `tests/unit/test_agent.py` — the "No document at" refusal message is unchanged (it reports the lookup failure, not the list format). The `_catalog_docs` mock parsing must produce identical `(source, path)` tuples for all existing test catalogs.
- Integration: `tests/integration/test_agent_tools.py` — `list_catalog` ordering assertions are unchanged (the function returns `[(source, path, title), ...]` tuples; only the string formatting in `_execute_tool` changes).
- E2E (mandatory, house rule): `tests/e2e/test_agent_document_tools.py` and `tests/e2e/test_agent_unlimited_tools.py` run in isolation — the SSE `argument` field (`source/path`) is unchanged; the mock's `_catalog_docs` parsing produces identical results; the tool lines and source chips render identically.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] `list_catalog` output uses `source: X | path: Y | title: Z` format — verifiable by a unit test that patches `list_catalog` and checks the formatted string.
- [ ] The `read_document` tool descriptions tell the LLM to extract `source` and `path` from the `source:` / `path:` labels.
- [ ] `tests/e2e/mock_llm.py::_catalog_docs` parses the new format and produces identical `(source, path)` tuples for all existing test catalogs.
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `uv run pytest tests/e2e/test_agent_document_tools.py tests/e2e/test_agent_unlimited_tools.py -v --no-cov` green in isolation (DB up).
- [ ] Regression E2E suites green in isolation: `test_chat_api.py`, `test_sse_events.py`, `test_whole_document_context.py`.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/` (`.agent/` stays untracked).
## Locked decisions
- **Owner-locked (2026-09-01, troubleshooting report, A1):** the new format uses `source: X | path: Y | title: Z` — one line per document, pipe-delimited fields, no ambiguity even with deeply nested paths.
- **Owner-locked (2026-09-01, troubleshooting report, A2):** the `read_document` tool descriptions are updated to reference the new format (tell the LLM how to parse `source:` and `path:` labels).
- **Owner-locked (2026-09-01, troubleshooting report, A3):** the SSE `argument` field (`source/path`) is unchanged — it is the tool call argument, not the list output. The frontend tool-line rendering is unchanged.
- **Owner-locked (2026-09-01, troubleshooting report, A4):** the "No document at …" refusal message is unchanged — it reports the lookup failure, not the list format.
## Commit
```bash
git add app/ tests/ && git commit --no-gpg-sign -m "fix(agent): unambiguous document listing format for LLM parsing"
```
@@ -0,0 +1,75 @@
# Task 01 — Change `list_catalog` output format + update tool descriptions
**Goal:** Make the `list_documents` catalog output unambiguous for LLM parsing, and update the `read_document` tool descriptions to tell the LLM how to extract `source` and `path`.
## Changes
### `app/rag/agent.py`
#### 1. Change `list_catalog` output format
In `_execute_tool`, the `list_documents` handler formats the catalog as:
```python
# BEFORE (ambiguous when path contains /)
listing = f"{len(rows)} documents:\n" + "\n".join(
f"{source}/{path} — {title}" for source, path, title in rows
)
```
Change to:
```python
# AFTER (unambiguous — each field is labeled)
listing = f"{len(rows)} documents:\n" + "\n".join(
f"source: {source} | path: {path} | title: {title}"
for source, path, title in rows
)
```
#### 2. Update `read_document` tool descriptions
In `AGENT_TOOLS`, update the `source` and `path` parameter descriptions to reference the new format:
```python
# BEFORE
"source": {
"description": (
"The document's source (a directory basename, e.g. 'Homelab')."
),
},
"path": {
"description": (
"The document's path relative to its source directory."
),
},
# AFTER
"source": {
"description": (
"The document's source, as shown after 'source: ' in the "
"list_documents output (e.g. 'Homelab' from "
"'source: Homelab | path: homelab/aws-route53.md')."
),
},
"path": {
"description": (
"The document's path, as shown after 'path: ' in the "
"list_documents output (e.g. 'homelab/aws-route53.md' from "
"'source: Homelab | path: homelab/aws-route53.md')."
),
},
```
### `app/rag/prompts.py`
No changes needed — the prompt text references the tool names but not the output format.
## Files changed
- `app/rag/agent.py` — `list_catalog` formatting + `AGENT_TOOLS` descriptions
## Verification
- `uv run pytest tests/unit/test_agent.py -v --no-cov` — the tool shape tests (`test_agent_tools_names_and_parameters`) check the description strings, so they must be updated to match.
- `uv run ruff check app/rag/agent.py && uv run pyright app/rag/agent.py` — lint + types clean.
@@ -0,0 +1,85 @@
# Task 02 — Update mock parsing + unit tests for the new format
**Goal:** Update `tests/e2e/mock_llm.py::_catalog_docs` to parse the new `source: X | path: Y | title: Z` format, and update `tests/unit/test_agent.py` to match the new tool descriptions.
## Changes
### `tests/e2e/mock_llm.py` — `_catalog_docs` function
The current parsing uses `head.rpartition("/")` on the `source/path` head:
```python
# BEFORE
def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
docs: list[tuple[str, str]] = []
for m in _messages(body):
if m.get("role") != "tool":
continue
content = str(m.get("content") or "")
if content.startswith(_READ_RESULT_PREFIX):
continue
for line in content.splitlines():
head = line.split(" — ", 1)[0].strip()
if "/" in head:
source, _, path = head.rpartition("/")
if source and path:
docs.append((source, path))
return docs
```
Change to parse the new labeled format:
```python
# AFTER
def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
docs: list[tuple[str, str]] = []
for m in _messages(body):
if m.get("role") != "tool":
continue
content = str(m.get("content") or "")
if content.startswith(_READ_RESULT_PREFIX):
continue
for line in content.splitlines():
# New format: "source: X | path: Y | title: Z"
if not line.startswith("source: "):
continue
parts = line.split(" | ", 2)
if len(parts) < 3:
continue
source = parts[0].removeprefix("source: ").strip()
path_part = parts[1]
if not path_part.startswith("path: "):
continue
path = path_part.removeprefix("path: ").strip()
if source and path:
docs.append((source, path))
return docs
```
### `tests/unit/test_agent.py` — tool description assertions
The test `test_agent_tools_names_and_parameters` asserts the exact description strings:
```python
# BEFORE (line ~109)
assert by_name["read_document"]["function"]["description"] == (
"Add the full content of one more indexed document to your context"
)
```
This assertion is about the function description (not parameter descriptions), so it stays the same. But the test file may have inline assertions about the `source` and `path` parameter descriptions — check and update if any exist.
### `tests/unit/test_agent.py` — refusal message
The refusal message `"No document at S/ghost.md — check the list_documents output."` is unchanged — it reports the lookup failure, not the list format. No changes needed here.
## Files changed
- `tests/e2e/mock_llm.py` — `_catalog_docs` parsing
- `tests/unit/test_agent.py` — only if inline assertions reference the old format (verify)
## Verification
- `uv run pytest tests/unit/test_agent.py -v --no-cov` — all agent unit tests pass.
- `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov` — the mock's `_catalog_docs` must produce identical `(source, path)` tuples for the test catalogs.
- `uv run ruff check tests/ && uv run pyright tests/` — lint + types clean.
@@ -0,0 +1,44 @@
# Task 03 — E2E document tools validation
**Goal:** Run the E2E suites for agent document tools and verify they still pass with the new catalog format.
## What to verify
The E2E tests assert on:
1. **SSE frames** — the `argument` field of `tool` events is `source/path` (e.g., `"Deployments/example-record-file.json"`). This is the tool call argument, NOT the list output format. **Unchanged.**
2. **UI rendering** — tool lines show "Listing documents" and "Reading <source/path>". **Unchanged.**
3. **Source chips** — display `${s.source}/${s.path}`. **Unchanged.**
4. **Answer content** — the mock's deterministic answer quoting the read document. **Unchanged** (the mock reads the same `(source, path)` tuples from the catalog).
5. **Query log** — sources field stores `"source/path, source/path"`. **Unchanged.**
The only thing that changes is the **text of the catalog** that the mock parses. The mock's `_catalog_docs` function must produce identical `(source, path)` tuples for the test catalogs, which means the tool flow classification (`_tool_flow`) and the read step will work identically.
## Run commands
```bash
# DB must be up: podman compose up -d db
uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov
uv run pytest tests/e2e/test_agent_unlimited_tools.py -v --no-cov
```
## Regression suites to run
```bash
uv run pytest tests/e2e/test_chat_api.py -v --no-cov
uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov
uv run pytest tests/e2e/test_sse_events.py -v --no-cov
```
## Expected outcome
All tests pass without modification (aside from the mock parsing change in task 02). The catalog text format is an internal detail of the agent loop — the tool call arguments, SSE frames, UI rendering, and answer content are all unchanged.
## If tests fail
1. Check that `_catalog_docs` produces the same `(source, path)` tuples as before for the test catalogs.
2. Verify the mock's `_tool_flow` classification still works (it calls `_catalog_docs`).
3. Check that the SSE `argument` field is still `source/path` (it should be — `_execute_tool` builds it from `call.arguments["source"] + "/" + call.arguments["path"]`, which is unchanged).
## Files changed
- None (verification only — all changes were in tasks 01 and 02)
@@ -0,0 +1,44 @@
# Task 04 — Update documentation
**Goal:** Update all documentation that references the old `source/path — title` catalog format to the new `source: X | path: Y | title: Z` format.
## Files to update
### `README.md` — Agent document tools section (~L168)
```markdown
# BEFORE
* **`list_documents`** — lists every indexed document, one
`source/path — title` line each (the same order as the Sources page);
# AFTER
* **`list_documents`** — lists every indexed document, one
`source: X | path: Y | title: Z` line each (the same order as the
Sources page);
```
### `.agent/user_stories/agent-document-tools.md` — Acceptance criteria (~L38)
```markdown
# BEFORE
3. `app/rag/agent.py`: the loop — budgets from
`BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` (default 1/1); the
`list_documents` tool returns the DB catalog
(`source/path — title` lines, `/api/docs` order);
# AFTER
3. `app/rag/agent.py`: the loop — budgets from
`BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` (default 1/1); the
`list_documents` tool returns the DB catalog
(`source: X | path: Y | title: Z` lines, `/api/docs` order);
```
## Verification
- `grep -n "source/path — title\|source/path - title" README.md .agent/user_stories/*.md` should return no results.
- `uv run ruff check README.md .agent/` — lint clean (markdown files are not linted by ruff, but the command should not error).
## Files changed
- `README.md` — Agent document tools section
- `.agent/user_stories/agent-document-tools.md` — Acceptance criteria item 3
@@ -0,0 +1,19 @@
All verification passes green. Phase 61 is fully complete — final verification pass only, no code changes needed.
**Phase 61 (stale_ui_copy) — final verification report**
Verified (all 4 task files already in `complete/`; no work remaining, no defects found):
- Chat page: meta description, empty-state sub, composer placeholder, and the 4 default chips in `app/config.py` match the locked (A1/A2) copy exactly
- KB page-sub (`sources.html`) reads the locked current-source-model text (git repos + local dirs + uploaded archives, Sync pulls/re-imports); git-sources placeholder is `https://github.com/you/your-repo.git` (A3)
- All 9 pages carry `<span class="footer-text">Powered by self-hosted models</span>` (1 each); `grep -rn "Reese's" frontend/` empty; only remaining "homelab" is the A3-exempt L176 code comment in `index.html`
- Tests: `tests/unit/test_stale_ui_copy.py` + `tests/e2e/test_stale_ui_copy.py` exist with real pins (body-text scan, DOM assertions)
Results:
- `uv run pytest --cov=app --cov-report=term-missing`: **1201 passed**, coverage TOTAL **99%** (>90% ✓)
- `uv run ruff check . && uv run pyright`: clean (0 errors)
- `uv run pytest tests/e2e/test_stale_ui_copy.py -v --no-cov`: **5 passed** in isolation (DB up)
- Regression E2E in isolation: `test_smoke` 3 ✓, `test_configurable_brand` 5 ✓, `test_header_consistency` 3 ✓, `test_dark_tech_theme` 6 ✓
- Commit `baefcde` `--no-gpg-sign` exists, staged paths only `app/ frontend/ tests/` ✓
Completion criteria: all 7 met (E2E body scan, KB page-sub, 9 footers, suite/coverage/lint, phase E2E, regression suites, single commit).
Next pending phase: **62_ui_customization**.
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
........................................................................ [ 71%]
........................................................................ [ 77%]
........................................................................ [ 83%]
........................................................................ [ 89%]
........................................................................ [ 95%]
................................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 101 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2458 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,17 @@
**Task 01 complete — report**
Implemented:
- `frontend/index.html`: meta description, `.empty-state-sub`, and `#message-input` placeholder → locked neutral copy (A1); brand-embedding strings left untouched for the phase-39 brand layer.
- `app/config.py`: `suggestions` default → the four locked chips (A2); `BOR_SUGGESTIONS` override path unchanged.
- Defect fix (pre-existing — 3 `tests/integration/test_api.py` failures even on the pristine tree): the operator's local gitignored `.env` leaked into test apps via pydantic `env_file=".env"`. Made the test env hermetic — `tests/conftest.py` + `tests/e2e/conftest.py` now force `BOR_DOCS_REPO=""` and `BOR_SUGGESTIONS`=code default (derived from `Settings.model_fields` so it can't drift); `test_response_to_docs.py` "unconfigured" spawn forces empty docs repo (dotenv leak re-enabled the docs feature there too).
Results:
- `uv run pytest --cov=app` → 1193 passed; TOTAL 99% (>90%)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
- `bash .agent/validate.sh` → validation OK
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov` → 3 passed (validates the modified shared E2E server)
- Task grep criteria verified: `homelab` in `index.html` only at the L176 code comment (A3); no "deployment" text; one user-visible "Ask me anything"
Decisions: conftest changes are a required defect fix (suite was red before this task on this machine); no commit made (task 04 owns it).
Next pending: `02_sources_pages_copy.md` (KB page-sub + remaining page-subs + git-sources example URL).
@@ -0,0 +1,71 @@
........................................................................ [ 6%]
........................................................................ [ 12%]
........................................................................ [ 18%]
........................................................................ [ 24%]
........................................................................ [ 30%]
........................................................................ [ 36%]
........................................................................ [ 42%]
........................................................................ [ 48%]
........................................................................ [ 54%]
........................................................................ [ 60%]
........................................................................ [ 66%]
........................................................................ [ 72%]
........................................................................ [ 78%]
........................................................................ [ 84%]
........................................................................ [ 90%]
........................................................................ [ 96%]
......................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 101 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2458 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,19 @@
All gates green. Task 02 complete.
**Implemented**
- `frontend/sources.html` KB page-sub → locked (A1) copy: "Every file indexed from your configured sources — git repositories, local directories, and uploaded archives. Press **Sync sources** to pull the latest and re-import."
- `frontend/git-sources.html` example placeholder → `https://github.com/you/your-repo.git` (A3)
- `tests/unit/test_sync_button.py` — updated the stale copy-pin (it asserted the retired "clone the repos and re-import") to pin the new locked copy incl. the source-model wording
- Verification read-through, all **no change** (accurate today): `sources.js` sync result/error strings (server-driven, incl. "no sources configured (git or local)" from `app/api/sync.py:190`); page-subs in `tuning.html`, `history.html`, `doc-edit.html`, `git-sources.html`; `#sources-gate` copy (verified `GET /api/docs` admin-only, `GET /api/documents/content` public)
**Results**
- `uv run pytest --cov=app --cov-report=term` → 1193 passed, TOTAL **99%** (>90%)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov` → 3 passed; `tests/e2e/test_header_consistency.py -v --no-cov` → 3 passed
- `grep "Homelab\|Deployments" frontend/sources.html` and `grep "homelab" frontend/git-sources.html` both empty; only A3-exempt comments remain
**Decisions**
- One pre-existing test pinned the retired copy (defect in previously completed work) — updated to the locked replacement rather than the HTML reverted
- No commit made: the phase's single atomic commit belongs to task 04
**Next pending task:** `.agent/phases/todo/61_stale_ui_copy/03_footer_defaults.md`
@@ -0,0 +1,71 @@
........................................................................ [ 6%]
........................................................................ [ 12%]
........................................................................ [ 18%]
........................................................................ [ 24%]
........................................................................ [ 30%]
........................................................................ [ 36%]
........................................................................ [ 42%]
........................................................................ [ 48%]
........................................................................ [ 54%]
........................................................................ [ 60%]
........................................................................ [ 66%]
........................................................................ [ 72%]
........................................................................ [ 78%]
........................................................................ [ 84%]
........................................................................ [ 90%]
........................................................................ [ 96%]
......................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 101 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2458 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,14 @@
Task 03 is complete and green.
**Task 03 — Neutral footer default on all 9 pages (`.footer-text` hook)**
- Replaced `<span>Powered by Reese's self-hosted models</span>` → `<span class="footer-text">Powered by self-hosted models</span>` on all 9 pages (`index`, `sources`, `tuning`, `document`, `git-sources`, `history`, `login`, `shared`, `doc-edit`), exactly at the audited line numbers; `.footer-version` siblings untouched
- Verified `grep -c 'class="footer-text"' frontend/*.html` → 1 per file, 9 total; `grep -rn "Reese's" frontend/` → empty
- No CSS/JS changes — the class is a selector-only hook for phase 62's `BOR_FOOTER_TEXT`; no test pinned the old footer text
- Tests: `uv run pytest --cov=app` → 1193 passed (exit 0), coverage TOTAL **99%** (>90% gate); `uv run ruff check . && uv run pyright` → clean
- Brand layer unaffected: `uv run pytest tests/unit/test_frontend_brand.py tests/e2e/test_configurable_brand.py -v --no-cov` → 15 passed (DB up, so run now rather than deferred to task 04)
- The 9-page unit pin is deliberately deferred to task 04 per the task file; no new tests added in this task (no logic changes)
- Not committed — the single atomic phase commit is task 04's job; working tree carries tasks 01–03 changes
- Note: `.agent/PLAN.md` is absent from the repo; owner-locked copy was taken from the phase's `00_phase.md` locked decisions (A1)
**Next pending task:** `.agent/phases/todo/61_stale_ui_copy/04_tests_and_commit.md`
@@ -0,0 +1,71 @@
........................................................................ [ 6%]
........................................................................ [ 12%]
........................................................................ [ 18%]
........................................................................ [ 24%]
........................................................................ [ 30%]
........................................................................ [ 36%]
........................................................................ [ 42%]
........................................................................ [ 48%]
........................................................................ [ 54%]
........................................................................ [ 60%]
........................................................................ [ 66%]
........................................................................ [ 72%]
........................................................................ [ 78%]
........................................................................ [ 84%]
........................................................................ [ 90%]
........................................................................ [ 96%]
......................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 101 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2458 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,17 @@
Task 04 complete. Report:
**Implemented**
- `tests/unit/test_stale_ui_copy.py` (8 tests, house text-pin pattern): old homelab-era strings GONE from `index.html`/`sources.html`/`git-sources.html`; locked A1 strings present exactly once; `class="footer-text"` + neutral footer on all 9 pages; A2 chips in `app/config.py` in order; old chips absent from all of `app/` + `frontend/` (retired, not relocated).
- `tests/e2e/test_stale_ui_copy.py` (5 tests, shared conftest server, DB up): chat placeholder/meta/footer pins, full body scan for `homelab`/`deployment`, A2 chip list in order, sources `.page-sub` (no Homelab/Deployments, has "configured sources"), git-sources `#git-source-url` placeholder (A3) — all from the rendered DOM.
**Verification (all green)**
- `uv run pytest --cov=app` → 1201 passed, coverage TOTAL **99%** (>90% gate)
- `uv run pytest tests/e2e/test_stale_ui_copy.py -v --no-cov` → 5 passed (isolation, DB up)
- Regression E2E in isolation: `test_smoke.py` 3 ✓, `test_configurable_brand.py` 5 ✓, `test_header_consistency.py` 3 ✓, `test_dark_tech_theme.py` 6 ✓
- `uv run ruff check . && uv run pyright` → clean
**Decisions/deviations**
- Old git example URL also pinned (A3, "where they lived"); old-chip absence scan scoped to `app/`+`frontend/` (a repo-wide scan would hit the pin file itself).
- Atomic commit `baefcde` on `main` — staged `frontend/ tests/ app/` only, `--no-gpg-sign`, body carries the task-02 read-through outcome (all 6 checked locations: no change needed).
**Next pending task:** `62_ui_customization/01_config_settings.md`
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
........................................................................ [ 71%]
........................................................................ [ 77%]
........................................................................ [ 83%]
........................................................................ [ 89%]
........................................................................ [ 95%]
................................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 101 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2458 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,14 @@
All completion criteria verified. Report:
**Phase 62 — UI Customization: final verification pass (all tasks already complete, commit `c738105` present)**
- Verified implementation: 3 new `Settings` fields + fail-loud `theme` validator, six-key `/api/config`, `brand.js` placeholder/footer/theme-link passes, `themes/indigo.css` + README, Containerfile L29 `cp -r ./assets/themes`, `.env.example` + README docs.
- Live spot-checks: default `/api/config` = exactly the six keys with phase-61 defaults; `/assets/themes/indigo.css` served 200 from dev static dir; `../evil.css`, `a/b.css`, `INDIGO.CSS`, `indigo` all refused at startup, each naming the value.
- `uv run pytest --cov=app` — all green; **coverage TOTAL 99%** (>90% gate).
- `uv run ruff check . && uv run pyright` — clean (0 errors).
- `uv run pytest tests/e2e/test_ui_customization.py -v --no-cov` — **5 passed** in isolation (DB up).
- Regression E2E in isolation: `test_configurable_brand` 5✓, `test_stale_ui_copy` 5✓, `test_smoke` 3✓, `test_dark_tech_theme` 6✓.
- Commit criterion: single `--no-gpg-sign` commit `c738105 feat(web): customizable placeholder, footer text, and color theme via BOR_* env vars` (stages only `app/ frontend/ tests/ .env.example README.md Containerfile`); `.agent/` phase files already filed under `complete/` by the harness.
- All 9 completion criteria: **met**. No defects found; no fixes needed.
- Note: `.agent/PLAN.md` referenced by AGENTS.md does not exist in the repo — verified against the phase overview's locked decisions (A4–A7), all honored.
- Next pending phase: **`63_unambiguous_document_listing`**.
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
........................................................................ [ 70%]
........................................................................ [ 76%]
........................................................................ [ 82%]
........................................................................ [ 88%]
........................................................................ [ 94%]
.................................................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2469 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,19 @@
All green. Task complete — final verification summary:
**Implemented (task 01 — Settings + `/api/config` keys + validator):**
- `app/config.py`: `input_placeholder` / `footer_text` / `theme` (phase-61 neutral defaults) + `theme` field_validator — empty ok, bare `^[a-z0-9_-]+\.css$` only, malformed fails startup naming the value (phase-56 style)
- `app/api/config.py`: `GET /api/config` now returns exactly the six locked keys (values verbatim, same public posture)
- Contract tests moved to six keys: `tests/integration/test_api.py` (both key-set tests + new `test_config_serves_ui_customization_overrides` + default-value pins), `tests/e2e/test_configurable_brand.py` (both instances)
- Unit pins: `tests/unit/test_config.py` — validator accept/reject cases (each rejection names the value), env-override + `BOR_THEME=../evil.css` startup failure
**Test / lint / coverage results:**
- `uv run pytest` → **1211 passed**
- `uv run pytest --cov=app --cov-report=term` → **TOTAL 99%** (>90% gate)
- `uv run ruff check .` → clean · `uv run pyright` → 0 errors
- `bash .agent/validate.sh` → "validation OK" · e2e file collects (5 tests; run lands in task 05)
**Notable decisions:**
- Also updated `tests/unit/test_save_as_doc_button.py` L54 key-set pin (not in the task's inventory — suite would go red without it) and added the phase-61-style `.env`-leak guard for the three new vars in `tests/conftest.py`
- No commit made — the phase's single atomic commit belongs to task 05
**Next pending task:** `02_brand_layer_customization.md` (brand.js applies placeholder/footer/theme from the same boot fetch)
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
........................................................................ [ 71%]
........................................................................ [ 77%]
........................................................................ [ 83%]
........................................................................ [ 89%]
........................................................................ [ 95%]
........................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2469 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,23 @@
Task 02 complete. All gates green.
**Implemented**
- `frontend/assets/brand.js`: the three phase-62 keys apply from the SAME settled-config `.then` (no second fetch), placed after the app_name block and independent of it (the name passes are now wrapped in `if (name)` — behavior byte-identical to the old early-return):
- `input_placeholder` → `#message-input` placeholder (null-guard no-op off the chat page)
- `footer_text` → every `.footer-text` node via `textContent` (no markup injection)
- `theme` → `<link id="theme-override" href="/assets/themes/<name>">` inserted immediately after the `styles.css` link (cascade wins), idempotent via `#theme-override`, A5 degradation: `onerror` → `console.warn`, built-in theme stands
- Header contract comment: new "Phase 62 (owner-locked 2026-09-01, TODO L3)" items 5–7 + updated no-op property (all 4 env vars unset ⇒ byte-identical)
- `tests/unit/test_frontend_brand.py`: new pin `test_brand_js_applies_phase_62_customization_from_the_same_fetch` — single-fetch marker, all three application markers, empty-skip guards, phase-62-after-app_name ordering, app_name default literal still holds
- Existing app_name passes 1–4 and `BOR_DOCS_REPO_CONFIGURED` logic untouched (re-indent only)
**Results**
- `uv run pytest` → 1212 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL 99% (>90%)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- `bash .agent/validate.sh` → validation OK
- `node --check frontend/assets/brand.js` → syntax OK
**Decisions**
- Keyed the empty-skip guards on `typeof … === "string"` + truthiness, so a fetch failure (`cfg` null) is also a clean no-op (house style: page never breaks).
- No defects found in prior work — baseline was green before my changes.
**Next pending task:** `.agent/phases/todo/62_ui_customization/03_theme_assets.md`
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
........................................................................ [ 71%]
........................................................................ [ 77%]
........................................................................ [ 83%]
........................................................................ [ 89%]
........................................................................ [ 95%]
............................................................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2469 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,21 @@
All done and green. Final report:
**Implemented (task 03 — theme assets)**
- `frontend/assets/themes/indigo.css` — WCAG-AA dark indigo/slate theme, single `:root` block, exactly the 8 identity overrides (semantic accent/ok/err untouched)
- `frontend/assets/themes/README.md` — authoring guide (load path, variable table w/ built-in refs, filename regex, 4.5:1 bar, never-white-on-brand trap, A7 deployment)
- `Containerfile` — `&& cp -r ./assets/themes /out/assets/themes \` added in stage 1, after the styles.css minify (line 29)
- `tests/unit/test_themes.py` — 6 new pins: files exist, single leading `:root` block + nothing else, exactly the 8 locked overrides, 5 identity-pair ratios ≥4.5:1 computed from the file's own hex values, Containerfile cp line (position + no esbuild), guide contract markers
**Test / lint / coverage**
- `uv run pytest --cov=app --cov-report=term -q` → rc=0, 1218 tests, TOTAL **99%** (>90%)
- `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors
- `uv run pytest tests/unit/test_themes.py tests/integration/test_containerfile_assets.py -v --no-cov` → 12 passed
- Ratios (WCAG calc): ink/bg 15.84 · ink/surface 14.71 · ink-soft/surface 8.27 · bg-on-brand 6.46 · brand-ink/surface 11.99 — all ≥4.5:1
- Dev sanity: `BOR_THEME=indigo.css` → `GET /assets/themes/indigo.css` 200, `/api/config` theme="indigo.css", served HTML has 0 theme refs; default env → theme="", built-in `--brand: #f43f5e` stands (opt-in via env var, no dir scanning)
**Decisions / notes**
- Task's comment ratios were slightly off (14.6/8.2/6.4/11.9); wrote the computed values (14.7/8.3/6.5/12.0) — no palette values changed
- Required grep spot-check: `f43f5e|#f0e6e6|#0f0a0a` matches only the `:root` block (L9/11/14) + one comment (L1271) as expected; broader scan found decorative `rgb()` literals (bg grid, glow spots, `.send-btn` steps) — noted, not refactored per task
- Commit deferred to task 05's atomic phase commit (per phase plan)
**Next pending task:** `.agent/phases/todo/62_ui_customization/04_docs.md`
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
........................................................................ [ 70%]
........................................................................ [ 76%]
........................................................................ [ 82%]
........................................................................ [ 88%]
........................................................................ [ 94%]
.................................................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2469 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,12 @@
All gates green. Task complete.
**Report**
- `.env.example` — added the three commented vars (`BOR_INPUT_PLACEHOLDER`, `BOR_FOOTER_TEXT`, `BOR_THEME`) in the `# --- App ---` block directly after the `BOR_APP_NAME` line, task's exact text/defaults, phase-cited.
- `README.md` env table — three rows added immediately after the `BOR_APP_NAME` row (what it affects, default, unset ⇒ no-op contract; `BOR_THEME` row documents the startup validator regex + missing-file degradation).
- `README.md` — new `### Customizing the look` subsection (4 sentences) placed at the end of "Configuration reference", directly **below** the env table, before "## Troubleshooting" — mirrors the house pattern of `###` feature subsections under a parent `##`; links to `frontend/assets/themes/README.md` (verified exists, link resolves) and names `indigo.css` as the example.
- No code changes (task is docs-only, per spec).
- Tests/lint: `uv run pytest` → 1218 passed; `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% gate); `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors.
- No commit made — the phase's single atomic commit is defined in task 05 (harness/handoff per phase overview).
Next pending task: `.agent/phases/todo/62_ui_customization/05_e2e_customization.md` (E2E suite + regression suites + atomic commit).
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
........................................................................ [ 70%]
........................................................................ [ 76%]
........................................................................ [ 82%]
........................................................................ [ 88%]
........................................................................ [ 94%]
.................................................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2469 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,16 @@
All gates green and the atomic commit landed. Final report:
## Phase 62 / Task 05 — complete
**Implemented**
- `tests/e2e/test_ui_customization.py` — 5 mapped tests: `/api/config` six-key overrides + dev static serving of the theme; chat page custom placeholder/footer/`link#theme-override` + computed `--brand` `#818cf8`; multi-page footer (login, no composer); default server byte-identical (settled-fetch wait before the absence assert); malformed `BOR_THEME="../evil.css"` refuses boot via real uvicorn subprocess, stderr names the value. Own instance on `APP_PORT+2`, brand suite's env-block pattern.
- **Defect fix (task 02 work):** the phase-33/54 cache-bust rewrite serves `styles.css?v=<token>`, so brand.js's `el.href.endsWith("styles.css")` never matched and the theme link was never inserted — now matches the raw attribute path with query/fragment stripped; unit pin added for the exact regression.
- E2E conftest hardening: shared `app_server` pins the three new `BOR_` vars to code defaults (same leak-class fix as phase 61).
**Results**
- `uv run pytest tests/e2e/test_ui_customization.py -v --no-cov` → 5 passed (isolation, DB up)
- Regressions in isolation: `test_configurable_brand` 5✓, `test_stale_ui_copy` 5✓, `test_smoke` 3✓, `test_dark_tech_theme` 6✓
- `.agent/validate.sh` → unit+integration green, coverage TOTAL **99%** (>90%), ruff clean, pyright 0 errors
- Commit `c738105` on `main`, staged exactly `app/ frontend/ tests/ .env.example README.md Containerfile`, `--no-gpg-sign`
**Decisions:** `.agent/` left unstaged per task file (harness moves the phase dir). Next pending: `63_unambiguous_document_listing/01` (or the harness's queue order).
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
........................................................................ [ 70%]
........................................................................ [ 76%]
........................................................................ [ 82%]
........................................................................ [ 88%]
........................................................................ [ 94%]
.................................................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2469 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,17 @@
All verification complete. The one remaining `source/path —` match is an explanatory comment in a test describing the historical bug (introduced by this phase's own commit) — not stale.
**Phase 63 — Final verification pass: ALL GREEN** (all 4 tasks already in `complete/`; no code changes needed)
- Verified `app/rag/agent.py`: `_execute_tool` emits `source: X | path: Y | title: Z` lines; `AGENT_TOOLS` `read_document` param descriptions reference the `source:`/`path:` labels; "No document at …" refusal unchanged
- Verified `tests/e2e/mock_llm.py::_catalog_docs` parses the new format via labeled-field regex; README + user story updated to new format; no stale old-format references remain
- `uv run pytest --cov=app --cov-report=term-missing` → **1219 passed**, TOTAL coverage **99%** (>90% gate)
- `uv run ruff check . && uv run pyright` → **All checks passed / 0 errors**
- `uv run pytest tests/e2e/test_agent_document_tools.py tests/e2e/test_agent_unlimited_tools.py -v --no-cov` → **8 passed** (DB up)
- Regression in isolation: `test_whole_document_context.py` **3 passed**; `tests/integration/test_chat_api.py` **17 passed**; `tests/unit/test_sse_events.py` **12 passed**; `tests/integration/test_agent_tools.py` **4 passed**
- Commit `15a16a8 fix(agent): unambiguous document listing format for LLM parsing` already present (`--no-gpg-sign`); harness has moved task files to `complete/`
Completion criteria: (1) new format + unit test — ✓ (2) tool descriptions — ✓ (3) mock parsing — ✓ (4) pytest/coverage/lint — ✓ (5) E2E document tools — ✓ (6) regression suites — ✓ (7) commit + phase move — ✓ (commit exists; `00_phase.md` move handled by harness)
Deviation: `test_chat_api.py`/`test_sse_events.py` don't exist under `tests/e2e/`; ran their actual locations (integration/unit) in isolation — all green.
Next pending phase: none — `todo/` holds only phase 63 itself.
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 64%]
........................................................................ [ 70%]
........................................................................ [ 76%]
........................................................................ [ 82%]
........................................................................ [ 88%]
........................................................................ [ 94%]
................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2469 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,11 @@
**Task 01 complete — report**
- `app/rag/agent.py`: `list_documents` output now `source: X | path: Y | title: Z` (labeled, pipe-delimited — owner-locked A1); `read_document` `source`/`path` param descriptions now point the LLM at the `source:`/`path:` labels (A2, verbatim from task file); `list_documents` tool description + module docstring updated to match the new format (old wording would have misdescribed the output). "No document at …" refusal and SSE `argument` unchanged (A3/A4).
- `tests/unit/test_agent.py`: tool-shape test now asserts the new param descriptions; all three listing-format assertions updated.
- `tests/unit/test_mock_tool_flow.py`: `CATALOG_*` wire fixtures updated to the new format (they must stay byte-identical to `agent.py` output).
- Deviation: `tests/e2e/mock_llm.py::_catalog_docs` (nominally task 02) was pulled into this task — the old `rsplit("/", 1)` parse would have mis-read the new lines and broken the phase's mandatory E2E gate; now regex-parses the labeled fields, producing identical `(source, path)` tuples (verified round-trip incl. the `brain-of-reese-main/homelab/aws-route53.md` case from the bug report).
- `uv run pytest --cov=app` → 1218 passed, TOTAL coverage **99%** (>90%)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- `uv run pytest tests/e2e/test_agent_document_tools.py tests/e2e/test_agent_unlimited_tools.py -v --no-cov` → 8 passed (DB up); regression `test_whole_document_context.py` → 3 passed. (`test_chat_api.py`/`test_sse_events.py` listed in the phase don't exist in this repo.)
- No commit made — the phase (tasks 02–04 remaining) gets one atomic commit at phase completion per AGENTS.md rule 8.
- Next pending task: `63_unambiguous_document_listing/02_mock_and_unit_tests.md` (its mock/unit work is already done here — expect verification only).
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 65%]
........................................................................ [ 70%]
........................................................................ [ 76%]
........................................................................ [ 82%]
........................................................................ [ 88%]
........................................................................ [ 94%]
.................................................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2469 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,19 @@
All gates green. Task 02 complete.
**Task 02 report — mock + unit tests for the new catalog format**
- **State found:** task 01's session had already applied the task-02 code (noted as a deviation in its report); this session verified it end-to-end and closed the one test gap.
- `tests/e2e/mock_llm.py::_catalog_docs` — verified: regex-parses the labeled `source: X | path: Y | title: Z` lines; produces identical `(source, path)` tuples for all test catalogs.
- `tests/unit/test_agent.py` — verified: `test_agent_tools_names_and_parameters` asserts the new A2 `source:`/`path:` parameter descriptions; all three listing-format assertions use the new format; "No document at S/ghost.md" refusal unchanged (A4).
- **Added** `test_read_step_nested_path_stays_intact` in `tests/unit/test_mock_tool_flow.py` — pins the actual bug: the `brain-of-reese-main` / `homelab/aws-route53.md` nested path (multiple `/`) parses intact; the old `rpartition` parse misread it.
**Gates (exact commands → outcomes)**
- `uv run pytest tests/unit/test_agent.py -v --no-cov` → 22 passed
- `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov` → 4 passed (DB up)
- `uv run pytest tests/e2e/test_agent_document_tools.py tests/e2e/test_agent_unlimited_tools.py -v --no-cov` → 8 passed in isolation
- `uv run pytest --cov=app --cov-report=term-missing` → **1219 passed**, coverage TOTAL **99%** (>90%)
- `uv run ruff check tests/ && uv run pyright tests/` → clean; `uv run ruff check . && uv run pyright` → clean (0 errors)
**Decisions/deviations:** none — mock and unit work was already in the working tree from task 01's session (its report documents the pull-in); no commit made (one atomic commit lands at phase completion per AGENTS.md rule 8).
**Next pending task:** `63_unambiguous_document_listing/03_e2e_document_tools.md`
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 64%]
........................................................................ [ 70%]
........................................................................ [ 76%]
........................................................................ [ 82%]
........................................................................ [ 88%]
........................................................................ [ 94%]
................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2469 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,16 @@
All gates green. Task 03 is verification-only (no code changes needed, consistent with its "Files changed: None"). Following the phase convention (one atomic commit at phase completion, per AGENTS.md rule 8), no commit is made here.
**Task 03 complete — E2E document tools validation**
- **Verified (no code changes):** SSE `tool` frames carry `argument` = `source/path`; UI renders "Listing documents"/"Reading …"; source chips `(source, path)` pairs; mock's deterministic answer content; query log `sources` = `"source/path, source/path"` — all asserted by the passing E2E suites against the new `source: X | path: Y | title: Z` catalog format.
- **Gates (DB up, all in isolation):**
- `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov` → 4 passed
- `uv run pytest tests/e2e/test_agent_unlimited_tools.py -v --no-cov` → 4 passed
- `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov` → 3 passed
- `uv run pytest tests/e2e/test_chat_rag.py -v --no-cov` → 3 passed
- `uv run pytest --cov=app --cov-report=term-missing` → **1219 passed**, coverage TOTAL **99%** (>90%)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- **Deviations:** `test_chat_api.py` / `test_sse_events.py` named in the task never existed in this repo (verified via `git log --all`); ran `test_chat_rag.py` as the chat-API/SSE regression equivalent — same finding as task 01's report.
- **No defects found** in prior work; no commit (phase-level commit at completion).
**Next pending task:** `63_unambiguous_document_listing/04_docs.md`
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 64%]
........................................................................ [ 70%]
........................................................................ [ 76%]
........................................................................ [ 82%]
........................................................................ [ 88%]
........................................................................ [ 94%]
................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2469 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,18 @@
All green. Phase 63 is now fully complete.
**Task 04 — documentation update**
- `README.md` — "Agent document tools" section now documents the `list_documents` output as `source: X | path: Y | title: Z` lines
- `.agent/user_stories/agent-document-tools.md` — acceptance criterion 3 updated to the labeled format (reflowed, meaning unchanged)
- Verified: `grep "source/path — title\|source/path - title"` over `README.md` + `.agent/user_stories/*.md` → no matches; `uv run ruff check README.md .agent/` → no error. Other `source/path` hits (SSE `argument`, source chips, `app/config.py` KB-overview note) are locked unchanged per A3 / are a different format
- Committed phase work (tasks 01–04) as `15a16a8 fix(agent): unambiguous document listing format for LLM parsing` (`--no-gpg-sign`, staged `app/ tests/ README.md`; `.agent/` tree left for the harness, matching phases 61/62)
**Test / lint / coverage results**
- `uv run pytest --cov=app --cov-report=term-missing` → exit 0, 1231 results, 0 failures; coverage TOTAL **99%** (>90% gate)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- E2E isolated (DB up): `test_agent_document_tools.py` 4 passed · `test_agent_unlimited_tools.py` 4 passed · `test_whole_document_context.py` 3 passed
- Regression isolated: `tests/integration/test_chat_api.py` 17 passed · `tests/unit/test_sse_events.py` 12 passed (the phase's `tests/e2e/` paths for these two don't exist — actual locations used)
**Deviations:** none — phase commit command's `app/ tests/` extended with `README.md` (task 04's deliverable; phase-62 precedent).
**Next pending task:** none — task 04 was the last task of phase 63; `todo/` is now empty aside from this phase's harness-managed files.
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 53%]
........................................................................ [ 59%]
........................................................................ [ 64%]
........................................................................ [ 70%]
........................................................................ [ 76%]
........................................................................ [ 82%]
........................................................................ [ 88%]
........................................................................ [ 94%]
................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 173 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2469 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,15 @@
All green. Task complete.
**Report — phase 64, task 01 (importer progress hook)**
- `app/rag/importer.py`: `import_sources` gains optional kwarg `progress: Callable[[str, str, int, int], None] | None`; called once per importable file immediately before `_index_file` with `(source, rel_posix_path, done, total)` — `done` = 1-based index across all sources (`summary.files`), `total` = combined count from a pre-walk via the existing `iter_importable_files` (stats only, no file reads), done **only when `progress` is provided**; `limit` path unchanged (hook fires per processed file, `total` stays full); module docstring notes the hook (phase 64).
- `tests/unit/test_importer.py`: 4 new tests — exact multi-root sequence (`total` = combined), no-pre-walk sentinel (monkeypatched counting wrapper: 1 walk without hook vs 2 with), unchanged + `EmbeddingError` files still reported as current, and `limit=` (2 events, `total` = 3).
**Results**
- `uv run pytest --cov=app --cov-report=term`: **1223 passed**, `app/` coverage **99%** (>90% gate)
- `uv run ruff check . && uv run pyright`: clean (0 errors)
- Existing callers (`scripts/import_docs.py`, `app/api/sync.py`, `app/api/git_sources.py`) untouched and byte-identical in behavior — all pre-existing tests green.
**Notes:** `.agent/PLAN.md` no longer exists in the repo (removed in commit `281f355`); the phase files served as binding context — no deviations. No commit made (phase commit belongs to task 06).
**Next pending task:** `64_sync_upload_progress/02_sync_status_current_file.md`
@@ -0,0 +1,71 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 47%]
........................................................................ [ 52%]
........................................................................ [ 58%]
........................................................................ [ 64%]
........................................................................ [ 70%]
........................................................................ [ 76%]
........................................................................ [ 82%]
........................................................................ [ 88%]
........................................................................ [ 94%]
....................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 89 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 180 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2476 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,17 @@
Everything is green and the diff is clean. Task complete.
## Report
**Implemented (task 02 — sync status reports the current file):**
- `app/api/sync.py`: `SyncStatus` gains `current_file`/`files_done`/`files_total`; `GET /api/sync/status` returns them (null/0/0 idle, all pre-existing keys unchanged); `_run_sync` resets them at start, passes the task-01 hook to `import_sources(..., prune=True, progress=_hook)` assigning `f"{source}/{rel}"` + counts, and clears `current_file` in both terminal states while keeping final counts; module + endpoint + dataclass docstrings updated (clone/pull phase reports no file per A4).
- `tests/unit/test_sync_button.py`: 3 new unit tests — full idle dict pinned (new keys null/0/0), mid-run status via the runner's real hook closure (worker-thread + gated clone/import, also pins A4 no-file-during-clone and success-terminal behavior), failed-terminal clears `current_file` keeping counts with sanitized error.
- `tests/integration/test_sync_api.py` (defect fix for prior work): `FakeImportSources`/`failing_import` now accept + record the `progress` kwarg, idle-shape pin extended with the three keys, success test pins the hook wiring.
**Results:**
- `uv run pytest` → 1226 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 1226 passed, TOTAL 99% (`app/api/sync.py` 100%)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
**Decisions:** unit tests drive `_run_sync` directly (no DB/HTTP; seams monkeypatched on `app.api.sync` — the house mock-import pattern), since the task named `test_sync_button.py` though the API coverage lives in the integration suite; in-memory `GitSource` needs explicit `kind="git"` (Python default applies at INSERT flush). No UI changes — "Syncing…" stays until task 04. No commit (phase commit belongs to task 06).
**Next pending task:** `.agent/phases/todo/64_sync_upload_progress/03_upload_background_202.md`
@@ -0,0 +1,72 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 41%]
........................................................................ [ 46%]
........................................................................ [ 52%]
........................................................................ [ 58%]
........................................................................ [ 64%]
........................................................................ [ 70%]
........................................................................ [ 76%]
........................................................................ [ 82%]
........................................................................ [ 88%]
........................................................................ [ 93%]
........................................................................ [ 99%]
.. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 143 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 101 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 180 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 198 0 100%
-----------------------------------------------
TOTAL 2488 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,20 @@
All green. Final report:
## Task 03 complete — background upload (202 + status endpoint)
**Implemented**
- `app/schemas.py`: new `UploadAccepted` (`detail` + safe `name`); `UploadOut` kept — now the shape of the status `detail` on success.
- `app/api/git_sources.py`: `POST /api/git-sources/upload` → **202** the moment the archive is on disk (`db` dependency dropped; inline gates unchanged: name 422, flag 409 set *before* the streamed receive, cap 413 with temp deleted + flag released on the pre-task exception path); unpack → swap → row upsert (short-lived `SessionLocal`, `IntegrityError` → failed naming the path) → `check_models` → `import_sources(..., progress=_hook)` → gated overview → moved INFO log line → `success` with `UploadOut` fields, all in `_run_upload` (phase-32 `_run_sync` shape; failures land sanitized in `failed`; `finally` cleans temps + clears the flag; `CancelledError` not caught). New `UploadStatus` dataclass + `GET /api/git-sources/upload/status` (identical key set to `/api/sync/status`, admin-only).
- `tests/integration/test_git_sources_upload.py`: all 29 scenarios kept, observation point moved to the status endpoint (polled to terminal) — every background branch has a dedicated test (unpack/zero-entry/swap/IntegrityError/model-down/import-fail/success/cancel-cleanup), mid-run `current_file`/counts via the task-02 hook seam, flag-not-state 409 pins, temp-on-disk + 202-before-scan pins, TRUNCATE regression.
**Results**
- `uv run pytest --cov=app --cov-report=term` → **1233 passed**, app/ **99%** (`git_sources.py` 100%)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- `bash .agent/validate.sh` → validation OK; upload suite alone: 29 passed in ~1 s
**Notable decisions**
- The task's "adapt `tests/unit/test_archive_upload.py`" bullet misreferences the file: that suite covers the unchanged `archive_upload` *utility*; the 202-contract HTTP scenarios live in `tests/integration/test_git_sources_upload.py` (adapted there; unit file untouched, still green).
- Discovered: starlette's TestClient delivers the response only after the app's task-completion callback runs on the loop — a sync park inside `unpack_archive` stalls even the 202. So the temp-on-disk pin drives `_run_upload` directly (worker loop), and the HTTP "202 while in flight" pin uses the async import gate.
- `test_archive_upload_sources.py` E2E expected to fail until task 06 (noted, not fixed here).
**Next pending task:** `04_sync_button_live_file.md`
@@ -0,0 +1,72 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 29%]
........................................................................ [ 35%]
........................................................................ [ 40%]
........................................................................ [ 46%]
........................................................................ [ 52%]
........................................................................ [ 58%]
........................................................................ [ 64%]
........................................................................ [ 70%]
........................................................................ [ 75%]
........................................................................ [ 81%]
........................................................................ [ 87%]
........................................................................ [ 93%]
........................................................................ [ 99%]
......... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 181 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 101 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 180 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 201 0 100%
-----------------------------------------------
TOTAL 2529 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,14 @@
Task 04 complete. All changes are confined to `frontend/` + the two test files (the `app/`/integration diffs were pre-existing uncommitted work from tasks 01–03, untouched by me).
**Implemented**
- `frontend/assets/sources.js`: `fmtSyncLabel(kind, currentFile, done, total)` (sync → "Syncing…", upload → "Importing", file appended only when present, counts only when `total > 0`); `enterSyncRunningState(kind, file, done, total)` writes the full untruncated path to button `title` (removed when null) + `#sync-result` aria-live; unified tick fetches both `/api/sync/status` and `/api/git-sources/upload/status` (upload 403 = "no upload", blips retry) with the 7-branch decision tree (sync running → upload running → sync success → sync failed → upload success → upload failed → idle); upload success settles + `loadDocs()` with no counts in `#sync-result` (A3); 202/409 click branch → `enterSyncRunningState("sync", null, 0, 0)`; `initSyncButton` re-attaches a running upload when sync is idle; header comment documents the contract
- `frontend/assets/styles.css`: `.sync-label` ellipsis (inline-block, `min(16rem, 40vw)`, ellipsis, nowrap)
- `frontend/sources.html`: comment block updated (no structural change)
- New `tests/unit/test_frontend_sync_upload.py` (9 pins); adapted 2 now-stale pins in `tests/unit/test_sync_button.py`
**Results**
- `uv run pytest --cov=app --cov-report=term` → 1242 passed, `app/` 99% (>90% gate)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- E2E regressions in isolation: `test_sync_button.py` 3✓, `test_sync_model_down.py` 4✓, `test_git_sources_admin.py` 6✓, `test_local_directory_sources.py` 3✓; `node --check` + runtime `fmtSyncLabel` checks pass
**Notes**: no deviations from the task spec; no commit (task 06 owns it). Next pending task: `.agent/phases/todo/64_sync_upload_progress/05_upload_toast_progress.md`.
@@ -0,0 +1,72 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 28%]
........................................................................ [ 34%]
........................................................................ [ 40%]
........................................................................ [ 46%]
........................................................................ [ 52%]
........................................................................ [ 57%]
........................................................................ [ 63%]
........................................................................ [ 69%]
........................................................................ [ 75%]
........................................................................ [ 81%]
........................................................................ [ 86%]
........................................................................ [ 92%]
........................................................................ [ 98%]
.................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 181 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 101 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 180 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 201 0 100%
-----------------------------------------------
TOTAL 2529 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
@@ -0,0 +1,19 @@
Everything is verified. All deletions in `git-sources.js` are confined to the phase-49 upload doc/paragraph (marked superseded) and the synchronous submit handler; load/add/remove behavior is untouched.
## Report — Task 05 complete
**Implemented** (`frontend/assets/git-sources.js` + `frontend/git-sources.html`):
- `showUploadToast` — page-local phase-55 toast: single lazy node, `role="status" aria-live="polite"`, ~5 s auto-dismiss, new toast replaces pending one; `.toast` CSS reused as-is
- Submit rework: 202 → toast `Successfully uploaded — <name>` (UploadAccepted body, file-name fallback) → clear input → processing state → poll; 409 → re-attach, NO error banner; other non-2xx/network keep phase-49 banners + kept selection; `finally` restores only when no poll is active (§7.4)
- `startUploadPolling` — 2 s single-timer poll of `GET /api/git-sources/upload/status`: running → live `"Processing… <file> (n/m)"` + full-path `title`; success → result line + announce + `loadSources()` (no second toast); failed → sanitized error banner + `loadSources()`; idle → defensive restore
- `initUploadStatus` boot re-attach (admin branch): running → processing + poll; success → result line only; failed → banner; idle → nothing; header/form comments rewritten (phase-49 paragraph superseded)
- `tests/unit/test_frontend_sync_upload.py` extended with 10 source pins (all task-04 pins intact)
**Results:**
- `uv run pytest --cov=app --cov-report=term-missing` → 1251 passed, coverage **99%** (>90 ✓)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
- Ad-hoc Playwright sanity (temp files, deleted): toast-on-202 + auto-dismiss + single node, Processing→settle, reload re-render, 409 re-attach without banner, mid-scan reload (160 files) re-attaches to `Processing…` and settles — zero uncaught JS errors
**Notes:** `tests/e2e/test_archive_upload_sources.py` still expects the old 200 flow — expected-fail until task 06 (noted in task 03); no commit (phase commit lands with task 06).
**Next pending task:** `06_e2e_sync_upload_progress.md`
@@ -0,0 +1,72 @@
........................................................................ [ 5%]
........................................................................ [ 11%]
........................................................................ [ 17%]
........................................................................ [ 23%]
........................................................................ [ 28%]
........................................................................ [ 34%]
........................................................................ [ 40%]
........................................................................ [ 46%]
........................................................................ [ 51%]
........................................................................ [ 57%]
........................................................................ [ 63%]
........................................................................ [ 69%]
........................................................................ [ 74%]
........................................................................ [ 80%]
........................................................................ [ 86%]
........................................................................ [ 92%]
........................................................................ [ 97%]
........................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
-----------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 22 0 100%
app/api/chat.py 131 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 7 0 100%
app/api/doc_drafts.py 93 0 100%
app/api/docs.py 50 0 100%
app/api/git_sources.py 181 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 8 0 100%
app/api/sync.py 101 0 100%
app/config.py 112 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 20 0 100%
app/core/caching.py 108 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/logging.py 13 0 100%
app/db.py 21 0 100%
app/main.py 52 0 100%
app/models.py 86 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 78 0 100%
app/rag/archive_upload.py 128 0 100%
app/rag/chunker.py 206 4 98%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 180 3 98%
app/rag/llm.py 189 0 100%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 65 0 100%
app/rag/retriever.py 94 3 97%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 201 0 100%
-----------------------------------------------
TOTAL 2529 12 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
validation OK
+2 -1
View File
@@ -35,7 +35,8 @@ addition to "thinking".
3. `app/rag/agent.py`: the loop — budgets from 3. `app/rag/agent.py`: the loop — budgets from
`BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` (default 1/1); the `BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` (default 1/1); the
`list_documents` tool returns the DB catalog `list_documents` tool returns the DB catalog
(`source/path — title` lines, `/api/docs` order); `read_document` (`source: X | path: Y | title: Z` lines, `/api/docs` order);
`read_document`
appends the **full** document text (A7-revised: never truncated); after appends the **full** document text (A7-revised: never truncated); after
both budgets are spent the tools are dropped and the model must answer; both budgets are spent the tools are dropped and the model must answer;
safety cap on model rounds; the deflection (LOW) path is byte-identical. safety cap on model rounds; the deflection (LOW) path is byte-identical.
+290 -137
View File
@@ -17,13 +17,19 @@ Routes: ``GET`` (DB rows oldest-first, or the env list with
``path``, git rows — and env rows — report ``path: null``), ``POST`` ``path``, git rows — and env rows — report ``path: null``), ``POST``
(201, validated create; ``kind`` selects the validation: git → exactly (201, validated create; ``kind`` selects the validation: git → exactly
the phase-35 URL contract, local → an existing absolute directory, else the phase-35 URL contract, local → an existing absolute directory, else
422 naming the path), ``POST /upload`` (phase 49 — admin archive upload: 422 naming the path), ``POST /upload`` (phase 49, backgrounded in phase
``.tar``/``.tar.gz``/``.tgz``/``.zip`` streamed with a size cap, safely 64 task 03 — admin archive upload: the ``.tar``/``.tar.gz``/``.tgz``/
unpacked, atomically swapped in over an existing folder of the same ``.zip`` name/format gate + the 1 MiB-chunk receive with the
name, row upserted, then the synchronous single-source scan — see ``upload_max_mb`` cap run **inline** and answered 202 the moment the
:func:`upload_archive`), ``DELETE /{source_id}`` (204). The whole archive is safely on disk; unpack → swap → row upsert → model check →
router sits behind :func:`app.core.auth.require_admin` — anonymous single-source scan → change-gated overview then run in a **background
callers get 403 on every route. task** — see :func:`upload_archive` and :func:`_run_upload`),
``GET /upload/status`` (the phase-32 ``SyncStatus``-shaped in-memory
state of that run — incl. the phase-64 ``current_file`` /
``files_done`` / ``files_total`` progress fields; navigating away from
the page mid-scan no longer aborts anything), ``DELETE /{source_id}``
(204). The whole router sits behind :func:`app.core.auth.require_admin`
— anonymous callers get 403 on every route.
No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's
masking discipline), so every git 409/422 detail is a fixed generic masking discipline), so every git 409/422 detail is a fixed generic
@@ -33,21 +39,26 @@ owner sees exactly which directory failed.
Scope boundary (phase locked decisions): the CRUD routes do NOT Scope boundary (phase locked decisions): the CRUD routes do NOT
clone, import, or prune anything — the existing Sync button performs clone, import, or prune anything — the existing Sync button performs
that (a removal prunes on the next sync, ``prune=True``). The phase-49 that (a removal prunes on the next sync, ``prune=True``). The upload
upload route is the exception: it unpacks the archive and then scans route is the exception (phase 64, task 03): after the 202 receive
the single source synchronously in the request (``import_sources`` answer, its background task unpacks the archive, swaps it in, upserts
with ``prune=True`` + the change-gated overview refresh) and answers the row, probes the models, scans the single source
with the sync-style counts. (``import_sources`` with ``prune=True`` + the change-gated overview
refresh), and lands the sync-style counts (the ``UploadOut`` fields)
in the status ``detail``.
""" """
from __future__ import annotations from __future__ import annotations
import asyncio
import logging import logging
import re import re
import shutil import shutil
import time import time
import uuid import uuid
from dataclasses import dataclass, field
from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
from typing import Literal, cast from typing import Any, Literal, cast
from fastapi import APIRouter, Depends, File, HTTPException, Response, UploadFile from fastapi import APIRouter, Depends, File, HTTPException, Response, UploadFile
from sqlalchemy import select from sqlalchemy import select
@@ -57,7 +68,7 @@ from sqlalchemy.orm import Session
from app.api.sync import _sanitize_error from app.api.sync import _sanitize_error
from app.config import get_settings from app.config import get_settings
from app.core.auth import require_admin from app.core.auth import require_admin
from app.db import get_db from app.db import SessionLocal, get_db
from app.models import GitSource from app.models import GitSource
from app.rag.archive_upload import ( from app.rag.archive_upload import (
ARCHIVE_SUFFIXES, ARCHIVE_SUFFIXES,
@@ -67,9 +78,15 @@ from app.rag.archive_upload import (
unpack_archive, unpack_archive,
) )
from app.rag.importer import import_sources from app.rag.importer import import_sources
from app.rag.llm import LLMClient, ModelUnavailableError, check_models from app.rag.llm import LLMClient, check_models
from app.rag.overview import regenerate_overview from app.rag.overview import regenerate_overview
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut, GitSourceRow, UploadOut from app.schemas import (
GitSourceIn,
GitSourceList,
GitSourceOut,
GitSourceRow,
UploadAccepted,
)
logger = logging.getLogger("app.api.git_sources") logger = logging.getLogger("app.api.git_sources")
@@ -79,18 +96,56 @@ router = APIRouter(
dependencies=[Depends(require_admin)], # phase 16 pattern: admin-only surface dependencies=[Depends(require_admin)], # phase 16 pattern: admin-only surface
) )
#: One upload at a time (phase 49, task 02 — the phase-32 ``_task`` #: One upload at a time (phase 49, backgrounded in phase 64 task 03):
#: spirit): the flag is held from the name gate through the scan #: the flag is checked and set with **no await in between, BEFORE the
#: response. A plain bool, not an ``asyncio.Lock`` — it is checked and #: streaming receive** — the handler now awaits (the 1 MiB-chunk stream)
#: set with no await in between (a single app loop can never enter #: long before the background task exists, so a task-done check alone
#: would let a concurrent POST slip through the receive window and start
#: a second run. A plain bool, not an ``asyncio.Lock``: it is checked
#: and set with no await in between (a single app loop can never enter
#: twice), and it stays correct across requests that run on separate #: twice), and it stays correct across requests that run on separate
#: event loops (the TestClient convention). #: event loops (the TestClient convention). Held until
#: :func:`_run_upload`'s ``finally`` (end of the background run) — or
#: cleared on the inline exception path where the task was never
#: created.
_upload_in_progress = False _upload_in_progress = False
#: Streaming read size while counting compressed upload bytes (1 MiB #: Streaming read size while counting compressed upload bytes (1 MiB
#: chunks — the task-02 cap check granularity). #: chunks — the task-02 cap check granularity).
_STREAM_CHUNK = 1 << 20 _STREAM_CHUNK = 1 << 20
@dataclass
class UploadStatus:
"""In-memory state of the (at most one) in-flight upload run.
Mirrors :class:`app.api.sync.SyncStatus` (the phase-32 pattern,
phase 64 task 03): ``state`` is the same four-state machine
(``idle`` / ``running`` / ``success`` / ``failed``); terminal states
carry the run's ``detail`` (success — the ``UploadOut`` fields) or
``error`` (failure — sanitized) so the UI can render the last result
after a page reload (the re-attach behavior, task 05).
Phase 64 (task 03) progress fields: ``current_file`` is the
``source/relative/path`` the scan is processing right now (null
outside the import phase — unpack/swap/row/model-check first — and
in terminal states); ``files_done`` / ``files_total`` carry the
hook's done/total position and survive a terminal state (the run's
last position is useful context next to the error).
"""
state: Literal["idle", "running", "success", "failed"] = "idle"
started_at: datetime | None = None
finished_at: datetime | None = None
current_file: str | None = None
files_done: int = 0
files_total: int = 0
detail: dict[str, Any] = field(default_factory=dict)
error: str | None = None
_upload_status = UploadStatus()
#: Accepted git URL shapes — the trimmed URL must *start* with one of them. #: Accepted git URL shapes — the trimmed URL must *start* with one of them.
#: Covers the phase-28 real URLs (HTTPS + ``git@`` SSH); scp-style #: Covers the phase-28 real URLs (HTTPS + ``git@`` SSH); scp-style
#: ``host:repo`` is deliberately rejected (422). ASSUMPTION (task 02): the #: ``host:repo`` is deliberately rejected (422). ASSUMPTION (task 02): the
@@ -227,46 +282,55 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
) )
@router.post("/upload", response_model=UploadOut) @router.post("/upload", response_model=UploadAccepted, status_code=202)
async def upload_archive( async def upload_archive(
file: UploadFile = File(...), # noqa: B008 file: UploadFile = File(...), # noqa: B008
db: Session = Depends(get_db), # noqa: B008 ) -> UploadAccepted:
) -> UploadOut: """Receive a source archive; scan it in the background (phase 49,
"""Upload a source archive and scan it (phase 49, task 02). backgrounded in phase 64 task 03 — owner-locked A1/A2).
The scan is **synchronous in the request** (phase locked decisions, The **inline (request) work is exactly three gates** — steps 1–3 —
owner-confirmed) and mirrors the admin sync pipeline: everything else runs in a background task behind
``GET /upload/status`` (the phase-32 ``SyncStatus`` pattern), so
navigating away mid-scan no longer aborts anything:
1. name/format gate — only ``.tar``/``.tar.gz``/``.tgz``/``.zip`` 1. name/format gate — only ``.tar``/``.tar.gz``/``.tgz``/``.zip``
(422 naming the accepted set) and a safe source name (422 naming the accepted set) and a safe source name
(``archive_source_name`` — its message is the 422 detail); (``archive_source_name`` — its message is the 422 detail);
2. one at a time — 409 ``an upload is already in progress``; 2. one at a time — 409 ``an upload is already in progress`` while
the flag is held (checked and set with no await in between,
BEFORE the receive — see ``_upload_in_progress``);
3. stream the upload in 1 MiB chunks into a dotfile temp with the 3. stream the upload in 1 MiB chunks into a dotfile temp with the
``upload_max_mb`` cap — 413 naming the cap, temp deleted; ``upload_max_mb`` cap — 413 naming the cap, temp deleted.
Then the archive is **safely on disk** — 202 + ``UploadAccepted``
(the "successfully uploaded" moment the UI toasts on, A2) and
:func:`_run_upload` runs the rest on the app's event loop:
4. unpack to a temp sibling (traversal/symlink/device/corrupt/ 4. unpack to a temp sibling (traversal/symlink/device/corrupt/
over-cap all 422 with the task-01 user-safe message, temps over-cap → ``failed`` with the task-01 user-safe message, temps
deleted); a zero-entry archive is 422 ``the archive contains no deleted); a zero-entry archive is ``failed`` ``the archive
files`` — an archive with only non-A9 files is a VALID contains no files`` — an archive with only non-A9 files is a
replacement (the scan indexes nothing, prune removes the VALID replacement (the scan indexes nothing, prune removes the
source's docs); source's docs);
5. atomic swap-in — a same-name re-upload replaces the previous 5. atomic swap-in — a same-name re-upload replaces the previous
folder in place; a failure leaves the previous folder/row/KB folder in place; a failure leaves the previous folder/row/KB
untouched (422); untouched;
6. upsert the row by ``path`` (``kind='local'``; an existing row is 6. upsert the row by ``path`` (``kind='local'``; an existing row is
left as-is — ``added_at`` preserved — and the unique index is left as-is — ``added_at`` preserved — and the unique index is
the 409 backstop); the backstop: a concurrent insert lands ``failed`` with
7. fail-fast ``check_models`` — 503 with the sanitized ``a local source with this path already exists: <path>``);
model-unavailable message; the folder/row are already committed, 7. fail-fast ``check_models`` — ``ModelUnavailableError`` →
so the next sync/re-upload retries idempotently; ``failed`` with the sanitized message (the phase-49 503 becomes
8. ``import_sources([folder], llm, prune=True)`` + the change-gated a status state, A5); the folder/row are already committed, so
``regenerate_overview``; the next sync/re-upload retries idempotently;
9. one INFO log line (PLAN §9 / AGENTS.md rule 10); 8. ``import_sources([folder], llm, prune=True, progress=<hook>)``
10. 200 with the sync-detail count keys (``UploadOut``). + the change-gated ``regenerate_overview`` — the hook feeds the
status ``current_file`` / ``files_done`` / ``files_total``;
9. one INFO log line (PLAN §9 / AGENTS.md rule 10 — ``total_ms`` is
the background run's duration);
10. ``success`` — ``detail`` = the ``UploadOut`` fields.
""" """
started = time.monotonic()
settings = get_settings()
total = 0
# 1. Name/format gate — the accepted formats first (the 422 names # 1. Name/format gate — the accepted formats first (the 422 names
# them), then the task-01 safe-name derivation. A BARE suffix # them), then the task-01 safe-name derivation. A BARE suffix
# ("tar.gz") is an accepted format with no usable stem — it # ("tar.gz") is an accepted format with no usable stem — it
@@ -288,12 +352,15 @@ async def upload_archive(
raise HTTPException(status_code=422, detail=str(e)) from None raise HTTPException(status_code=422, detail=str(e)) from None
# 2. One at a time — the flag is checked and set with no await # 2. One at a time — the flag is checked and set with no await
# between, so the single app loop can never enter twice. # between, BEFORE the streaming receive: the background task
# does not exist yet, so the flag (not a task-done check) is the
# gate (see ``_upload_in_progress``).
global _upload_in_progress global _upload_in_progress
if _upload_in_progress: if _upload_in_progress:
raise HTTPException(status_code=409, detail="an upload is already in progress") raise HTTPException(status_code=409, detail="an upload is already in progress")
_upload_in_progress = True _upload_in_progress = True
settings = get_settings()
upload_root = Path(settings.upload_dir).expanduser() upload_root = Path(settings.upload_dir).expanduser()
upload_root.mkdir(parents=True, exist_ok=True) upload_root.mkdir(parents=True, exist_ok=True)
max_bytes = settings.upload_max_mb * 1024 * 1024 max_bytes = settings.upload_max_mb * 1024 * 1024
@@ -302,77 +369,192 @@ async def upload_archive(
try: try:
# 3. Stream with the compressed-size cap — dotfile temps are # 3. Stream with the compressed-size cap — dotfile temps are
# hidden from the upload dir's listing. # hidden from the upload dir's listing.
try: total = 0
with open(temp_upload, "wb") as out: with open(temp_upload, "wb") as out:
while chunk := await file.read(_STREAM_CHUNK): while chunk := await file.read(_STREAM_CHUNK):
total += len(chunk) total += len(chunk)
if total > max_bytes: if total > max_bytes:
raise HTTPException( raise HTTPException(
status_code=413, status_code=413,
detail=f"the upload exceeds the {settings.upload_max_mb} MiB limit", detail=f"the upload exceeds the {settings.upload_max_mb} MiB limit",
) )
out.write(chunk) out.write(chunk)
except HTTPException: # The archive is safely on disk — 202 is the "successfully
temp_upload.unlink(missing_ok=True) # uploaded" moment (A2). Steps 4–10 run in the background:
raise asyncio.create_task(
# 4. Unpack to a temp sibling; the compressed bytes are no _run_upload(name, filename, total, upload_root, temp_upload, temp_unpack)
# longer needed once unpacked (phase locked decision: only )
# the unpacked content is kept). except BaseException:
try: # The background task was never created (cap 413, a broken
unpack_archive(temp_upload, temp_unpack, max_bytes) # pipe, cancellation, or create_task itself): release the flag
except ArchiveUploadError as e: # so the next upload is not refused, and make sure no temp
temp_upload.unlink(missing_ok=True) # survives the failed receive.
shutil.rmtree(temp_unpack, ignore_errors=True) temp_upload.unlink(missing_ok=True)
raise HTTPException(status_code=422, detail=str(e)) from None _upload_in_progress = False
raise
return UploadAccepted(name=name)
@router.get("/upload/status")
def upload_status() -> dict[str, Any]:
"""Current upload state (the UI polls this — the phase-32
``GET /api/sync/status`` contract, identical key set).
``started_at`` / ``finished_at`` are ISO-8601 strings or null.
``current_file`` (phase 64) is the ``source/relative/path`` the
scan is processing right now — null during the unpack/swap/row/
model phases and in terminal states; ``files_done`` / ``files_total``
carry the hook's position (0/0 idle). The router dependency makes
it admin-only like every other route here.
"""
return {
"state": _upload_status.state,
"started_at": (
_upload_status.started_at.isoformat() if _upload_status.started_at else None
),
"finished_at": (
_upload_status.finished_at.isoformat() if _upload_status.finished_at else None
),
"detail": _upload_status.detail,
"error": _upload_status.error,
"current_file": _upload_status.current_file,
"files_done": _upload_status.files_done,
"files_total": _upload_status.files_total,
}
async def _run_upload(
name: str,
filename: str,
total_bytes: int,
upload_root: Path,
temp_upload: Path,
temp_unpack: Path,
) -> None:
"""The post-202 upload pipeline, one in-process background task
(the phase-32 ``_run_sync`` shape — A1).
Every failure mode (unpack, zero entries, swap, row, models,
import, anything else) lands in the ``failed`` state with a
sanitized ``error`` string — a background task must die in state,
never as an unobserved exception (A5: post-202 failures are status
states, never HTTP errors). ``CancelledError`` is deliberately *not*
caught: app shutdown cancels the task, and swallowing that would
mask a real stop. The ``finally`` cleans both temps (defensive —
each step already cleans its own) and clears ``_upload_in_progress``.
"""
global _upload_in_progress
started = time.monotonic()
_upload_status.state = "running"
_upload_status.started_at = datetime.now(UTC)
_upload_status.finished_at = None
_upload_status.current_file = None
_upload_status.files_done = 0
_upload_status.files_total = 0
_upload_status.detail = {}
_upload_status.error = None
try:
settings = get_settings()
max_bytes = settings.upload_max_mb * 1024 * 1024
# Step 4 — unpack to a temp sibling; the compressed bytes are
# no longer needed once unpacked (phase 49 locked decision:
# only the unpacked content is kept).
unpack_archive(temp_upload, temp_unpack, max_bytes)
temp_upload.unlink(missing_ok=True) temp_upload.unlink(missing_ok=True)
if not any(temp_unpack.iterdir()): if not any(temp_unpack.iterdir()):
# Zero entries = a user error. (Only non-A9 files is NOT an # Zero entries = a user error. (Only non-A9 files is NOT an
# error — it still has entries and is a valid replacement.) # error — it still has entries and is a valid replacement.)
shutil.rmtree(temp_unpack, ignore_errors=True) raise ArchiveUploadError("the archive contains no files")
raise HTTPException(status_code=422, detail="the archive contains no files") # Step 5 — swap in — a same-name re-upload replaces the
# 5. Swap in — a same-name re-upload replaces the previous # previous folder atomically; a failure leaves it, the row,
# folder atomically; a failure leaves it, the row, and the # and the KB untouched (the ``failed`` state carries the
# KB untouched. # user-safe message).
final_dir = upload_root / name final_dir = upload_root / name
try: swap_in(temp_unpack, final_dir)
swap_in(temp_unpack, final_dir) # Step 6 — upsert the row by path in a SHORT-LIVED session
except ArchiveUploadError as e: # (open/close around it — the ``effective_sources`` /
shutil.rmtree(temp_unpack, ignore_errors=True) # ``bump_sources_version`` pattern in ``app.api.sync``): the
raise HTTPException(status_code=422, detail=str(e)) from None # background task has no request session to leak locks from
# 6. Upsert the row by path — no duplicates: an existing row is # (the old inline ``db.close()`` discipline, now structural).
# left exactly as it is (``added_at`` preserved); the unique # No duplicates: an existing row is left exactly as it is
# index is the 409 backstop for a concurrent insert the # (``added_at`` preserved); the unique index is the backstop
# pre-check missed. # for a concurrent insert the pre-check missed.
path = str(final_dir) path = str(final_dir)
if db.scalar(select(GitSource).where(GitSource.path == path)) is None: db = SessionLocal()
_commit_new(
GitSource(url=path, kind="local", path=path),
f"a local source with this path already exists: {path}",
db,
)
# Release the request session NOW — the handler never touches
# ``db`` again (the scan below uses its own sessions). If the
# session stayed open, its uncommitted transaction (the
# ``_commit_new`` refresh SELECT) would hold ``git_sources``
# locks for the whole scan, and any concurrent TRUNCATE of the
# KB tables (the E2E isolation fixtures) would deadlock against
# the scan's own document locks — a cycle Postgres cannot see.
# ``get_db``'s teardown close() is idempotent.
db.close()
# 7. Fail-fast models (phase 41) — 503 with the sanitized
# message; nothing else is rolled back (the folder/row are
# committed and the next sync/re-upload retries idempotently).
llm = LLMClient()
try: try:
await check_models(llm) if db.scalar(select(GitSource).where(GitSource.path == path)) is None:
except ModelUnavailableError as e: db.add(GitSource(url=path, kind="local", path=path))
raise HTTPException(status_code=503, detail=_sanitize_error(str(e))) from None try:
# 8. Scan — single source, prune (dropped files leave the KB), db.commit()
# then the change-gated overview refresh (phases 31/32). except IntegrityError:
summary = await import_sources([final_dir], llm, prune=True) db.rollback()
raise ValueError(
f"a local source with this path already exists: {path}"
) from None
finally:
db.close()
# Step 7 — fail-fast models (phase 41): ``ModelUnavailableError``
# lands in the ``failed`` state sanitized (the phase-49 503
# becomes a status state, A5). Nothing is rolled back — the
# folder/row are committed and the next sync/re-upload retries
# idempotently.
llm = LLMClient()
await check_models(llm)
# Step 8 — scan — single source, prune (dropped files leave
# the KB), with the phase-64 progress hook feeding the status,
# then the change-gated overview refresh (phases 31/32). The
# closure captures the module ``_upload_status`` exactly like
# the state assignments above.
def _hook(source: str, rel: str, done: int, total: int) -> None:
_upload_status.current_file = f"{source}/{rel}"
_upload_status.files_done = done
_upload_status.files_total = total
summary = await import_sources([final_dir], llm, prune=True, progress=_hook)
overview = False overview = False
if summary.added + summary.updated > 0: if summary.added + summary.updated > 0:
overview = await regenerate_overview(llm) overview = await regenerate_overview(llm)
# Step 9 — per-upload log line (PLAN §9 / AGENTS.md rule 10)
# — moved with the scan: ``total_ms`` is the background run's
# duration.
logger.info(
"upload: name=%s file=%s bytes_in=%d files=%d added=%d updated=%d "
"unchanged=%d pruned=%d errors=%d overview=%s total_ms=%d",
name,
filename,
total_bytes,
summary.files,
summary.added,
summary.updated,
summary.unchanged,
summary.pruned,
summary.errors,
overview,
round((time.monotonic() - started) * 1000),
)
# Step 10 — success: the ``UploadOut`` fields ride in the
# status ``detail`` (the UI renders the same result line from
# the status that the sync button renders from its own).
_upload_status.state = "success"
_upload_status.finished_at = datetime.now(UTC)
_upload_status.current_file = None # phase 64: keep the final counts
_upload_status.detail = {
"source": name,
"files": summary.files,
"added": summary.added,
"updated": summary.updated,
"unchanged": summary.unchanged,
"pruned": summary.pruned,
"errors": summary.errors,
"chunks": summary.chunks,
"overview": overview,
}
except Exception as e: # noqa: BLE001 — a background task dies in state, see above
logger.exception("upload: failed")
_upload_status.state = "failed"
_upload_status.finished_at = datetime.now(UTC)
_upload_status.error = _sanitize_error(str(e))
_upload_status.current_file = None # phase 64: keep the final counts
finally: finally:
_upload_in_progress = False _upload_in_progress = False
# No temp may survive any failure path (defensive — each step # No temp may survive any failure path (defensive — each step
@@ -380,35 +562,6 @@ async def upload_archive(
temp_upload.unlink(missing_ok=True) temp_upload.unlink(missing_ok=True)
shutil.rmtree(temp_unpack, ignore_errors=True) shutil.rmtree(temp_unpack, ignore_errors=True)
# 9. Per-upload log line (PLAN §9 / AGENTS.md rule 10).
logger.info(
"upload: name=%s file=%s bytes_in=%d files=%d added=%d updated=%d "
"unchanged=%d pruned=%d errors=%d overview=%s total_ms=%d",
name,
filename,
total,
summary.files,
summary.added,
summary.updated,
summary.unchanged,
summary.pruned,
summary.errors,
overview,
round((time.monotonic() - started) * 1000),
)
# 10. Respond 200 with the sync-style counts.
return UploadOut(
source=name,
files=summary.files,
added=summary.added,
updated=summary.updated,
unchanged=summary.unchanged,
pruned=summary.pruned,
errors=summary.errors,
chunks=summary.chunks,
overview=overview,
)
@router.delete("/{source_id}", status_code=204) @router.delete("/{source_id}", status_code=204)
def delete_git_source( def delete_git_source(
+42 -2
View File
@@ -51,7 +51,12 @@ decisions):
run aborts in the ``failed`` state before this step. run aborts in the ``failed`` state before this step.
Status is in memory: a restart mid-sync loses the running state Status is in memory: a restart mid-sync loses the running state
(accepted — the next click re-syncs idempotently). (accepted — the next click re-syncs idempotently). The status also
carries the phase-64 per-file progress — ``current_file`` (the
``source/relative/path`` the import is processing right now) plus
``files_done`` / ``files_total`` — null/0/0 before the import starts
(clone/pull reports no file yet) and in terminal states, which clear
``current_file`` but keep the run's final counts.
""" """
from __future__ import annotations from __future__ import annotations
@@ -107,6 +112,13 @@ class SyncStatus:
``running``, ``success``, ``failed``. Terminal states carry the run's ``running``, ``success``, ``failed``. Terminal states carry the run's
``detail`` (success) or ``error`` (failure) so the UI can render the ``detail`` (success) or ``error`` (failure) so the UI can render the
last result after a page reload (task 02's re-attach behavior). last result after a page reload (task 02's re-attach behavior).
Phase 64 (task 02) progress fields: ``current_file`` is the
``source/relative/path`` the import is processing right now (null
outside the import phase — clone/pull first, terminal states
after); ``files_done`` / ``files_total`` carry the hook's
done/total position and survive a terminal state (the run's last
position is useful context next to the error).
""" """
state: Literal["idle", "running", "success", "failed"] = "idle" state: Literal["idle", "running", "success", "failed"] = "idle"
@@ -114,6 +126,11 @@ class SyncStatus:
finished_at: datetime | None = None finished_at: datetime | None = None
detail: dict[str, Any] = field(default_factory=dict) detail: dict[str, Any] = field(default_factory=dict)
error: str | None = None error: str | None = None
# Phase 64 (task 02): per-file progress — the file the import is
# processing right now and the hook's done/total position.
current_file: str | None = None
files_done: int = 0
files_total: int = 0
_status = SyncStatus() _status = SyncStatus()
@@ -125,6 +142,10 @@ def sync_status() -> dict[str, Any]:
"""Current sync state (the UI polls this every 2 s — task 02). """Current sync state (the UI polls this every 2 s — task 02).
``started_at`` / ``finished_at`` are ISO-8601 strings or null. ``started_at`` / ``finished_at`` are ISO-8601 strings or null.
``current_file`` (phase 64) is the ``source/relative/path`` the
import is processing right now — null during the clone/pull phase
and in terminal states; ``files_done`` / ``files_total`` carry the
hook's position (0/0 idle).
""" """
return { return {
"state": _status.state, "state": _status.state,
@@ -132,6 +153,9 @@ def sync_status() -> dict[str, Any]:
"finished_at": _status.finished_at.isoformat() if _status.finished_at else None, "finished_at": _status.finished_at.isoformat() if _status.finished_at else None,
"detail": _status.detail, "detail": _status.detail,
"error": _status.error, "error": _status.error,
"current_file": _status.current_file,
"files_done": _status.files_done,
"files_total": _status.files_total,
} }
@@ -165,6 +189,11 @@ async def _run_sync() -> None:
_status.finished_at = None _status.finished_at = None
_status.detail = {} _status.detail = {}
_status.error = None _status.error = None
# Phase 64 (task 02): the progress fields reset with the run — no
# current file until the import starts (the clone/pull phase).
_status.current_file = None
_status.files_done = 0
_status.files_total = 0
try: try:
settings = get_settings() settings = get_settings()
# Step 1 (phase 41): fail fast — verify both models the sync # Step 1 (phase 41): fail fast — verify both models the sync
@@ -208,7 +237,16 @@ async def _run_sync() -> None:
if not path.is_dir(): if not path.is_dir():
raise GitSyncError(f"local source missing: {path}") raise GitSyncError(f"local source missing: {path}")
sources.append(path) sources.append(path)
summary: ImportSummary = await import_sources(sources, llm, prune=True) # Phase 64 (task 02): the per-file progress hook — the status
# endpoint reports the file being processed right now. The
# closure captures the module ``_status`` exactly like the state
# assignments above.
def _hook(source: str, rel: str, done: int, total: int) -> None:
_status.current_file = f"{source}/{rel}"
_status.files_done = done
_status.files_total = total
summary: ImportSummary = await import_sources(sources, llm, prune=True, progress=_hook)
overview = False overview = False
if summary.added + summary.updated > 0: if summary.added + summary.updated > 0:
overview = await regenerate_overview(llm) overview = await regenerate_overview(llm)
@@ -234,6 +272,7 @@ async def _run_sync() -> None:
db.close() db.close()
_status.state = "success" _status.state = "success"
_status.finished_at = datetime.now(UTC) _status.finished_at = datetime.now(UTC)
_status.current_file = None # phase 64: keep the final counts
_status.detail = { _status.detail = {
"files": summary.files, "files": summary.files,
"added": summary.added, "added": summary.added,
@@ -253,3 +292,4 @@ async def _run_sync() -> None:
_status.state = "failed" _status.state = "failed"
_status.finished_at = datetime.now(UTC) _status.finished_at = datetime.now(UTC)
_status.error = _sanitize_error(str(e)) _status.error = _sanitize_error(str(e))
_status.current_file = None # phase 64: keep the final counts
+32
View File
@@ -26,11 +26,15 @@ no longer exist **or no longer match the format filter** — this is how
previously-imported junk (e.g. dot-dir READMEs) leaves the index. Per-file previously-imported junk (e.g. dot-dir READMEs) leaves the index. Per-file
logging uses the verbs ``added | updated | unchanged | pruned`` plus a logging uses the verbs ``added | updated | unchanged | pruned`` plus a
summary line with per-format counts (PLAN §9). summary line with per-format counts (PLAN §9).
``import_sources`` accepts an optional per-file ``progress`` callback
(phase 64, task 01) reporting the file being processed right now.
""" """
from __future__ import annotations from __future__ import annotations
import hashlib import hashlib
import logging import logging
from collections.abc import Callable
from dataclasses import dataclass, field from dataclasses import dataclass, field
from datetime import UTC, datetime from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
@@ -148,12 +152,26 @@ async def import_sources(
prune: bool = False, prune: bool = False,
limit: int | None = None, limit: int | None = None,
session: Session | None = None, session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
) -> ImportSummary: ) -> ImportSummary:
"""Import every A9-format file under *sources* (see module docstring). """Import every A9-format file under *sources* (see module docstring).
``session`` may be supplied (tests); a private one is opened and closed ``session`` may be supplied (tests); a private one is opened and closed
otherwise. ``limit`` caps the number of files processed (debug only) and otherwise. ``limit`` caps the number of files processed (debug only) and
disables pruning, since an incomplete walk must not drive deletions. disables pruning, since an incomplete walk must not drive deletions.
``progress`` (phase 64, task 01) is an optional per-file hook called
once per importable file, immediately before that file's
``_index_file`` — with ``(source, rel_posix_path, done, total)``: the
same POSIX *rel* the document rows use, ``done`` = the 1-based index of
the current file **across all sources**, and ``total`` = the combined
pre-walk count of importable files across all *sources* roots. The
pre-walk (same extension/exclusion rules, directory stats only, no file
reads) happens **only when *progress* is provided**: callers passing
nothing pay no extra walk and behave exactly as before. Under
``limit``, the hook still fires per processed file only — ``done`` never
exceeds the limit, but ``total`` stays the full pre-walk count (an
incomplete walk must not misreport the denominator).
""" """
if limit is not None and limit <= 0: if limit is not None and limit <= 0:
raise ValueError("limit must be >= 1") raise ValueError("limit must be >= 1")
@@ -163,6 +181,14 @@ async def import_sources(
session = SessionLocal() session = SessionLocal()
seen: set[tuple[str, str]] = set() seen: set[tuple[str, str]] = set()
source_names: set[str] = set() source_names: set[str] = set()
# phase 64 (task 01): the hook's combined denominator, walked with the
# exact same rules as the processing loop below (directory stats only,
# no file reads). Skipped entirely for ``progress=None`` callers — no
# extra pass, byte-identical behaviour and cost.
total = 0
if progress is not None:
for root in sources:
total += len(iter_importable_files(root, llm.settings.import_extension_set))
try: try:
for root in sources: for root in sources:
if not root.is_dir(): if not root.is_dir():
@@ -180,6 +206,12 @@ async def import_sources(
summary.files += 1 summary.files += 1
ext = path.suffix.lower().lstrip(".") or "unknown" ext = path.suffix.lower().lstrip(".") or "unknown"
summary.formats[ext] = summary.formats.get(ext, 0) + 1 summary.formats[ext] = summary.formats.get(ext, 0) + 1
if progress is not None:
# phase 64: report the file *before* indexing it — a
# file that then errors or turns out unchanged was
# already the "current file". No try/except around the
# call: the hooks in this repo only assign fields.
progress(source, rel, summary.files, total)
try: try:
await _index_file( await _index_file(
session, source=source, rel=rel, full_path=path, llm=llm, session, source=source, rel=rel, full_path=path, llm=llm,
+20 -2
View File
@@ -304,9 +304,12 @@ class GitSourceList(BaseModel):
class UploadOut(BaseModel): class UploadOut(BaseModel):
"""``POST /api/git-sources/upload`` response (phase 49, task 02). """The upload run's result fields (phase 49, task 02; phase 64, task 03).
The uploaded source's name (filename minus the archive suffix) plus Phase 64 (task 03): ``POST /api/git-sources/upload`` answers 202 the
moment the archive is on disk; these fields become the shape of
``GET /api/git-sources/upload/status`` ``detail`` on ``success`` —
the uploaded source's name (filename minus the archive suffix) plus
the SAME count keys as the admin sync's success ``detail`` the SAME count keys as the admin sync's success ``detail``
(``files``, ``added``, ``updated``, ``unchanged``, ``pruned``, (``files``, ``added``, ``updated``, ``unchanged``, ``pruned``,
``errors``, ``chunks`` — ``app.api.sync._run_sync``) and the ``errors``, ``chunks`` — ``app.api.sync._run_sync``) and the
@@ -325,6 +328,21 @@ class UploadOut(BaseModel):
overview: bool overview: bool
class UploadAccepted(BaseModel):
"""``POST /api/git-sources/upload`` 202 response (phase 64, task 03).
The archive is **safely on disk** — this is the "successfully
uploaded" moment the Sources page toasts on (owner-locked A2). The
scan itself (unpack → swap → row upsert → model check → import →
overview) runs in a background task behind
``GET /api/git-sources/upload/status``, whose ``success`` ``detail``
carries the :class:`UploadOut` fields.
"""
detail: str = "upload received"
name: str
class ToolCall(BaseModel): class ToolCall(BaseModel):
"""One agent tool-call record (the phase-37 ``tools`` record shape). """One agent tool-call record (the phase-37 ``tools`` record shape).
+275 -41
View File
@@ -34,20 +34,38 @@
* shape-aware like the tuning forms) and keeps the input — the * shape-aware like the tuning forms) and keeps the input — the
* instruction survives. 409/422 details are fixed generic strings * instruction survives. 409/422 details are fixed generic strings
* (credential safety — the URL is never echoed). * (credential safety — the URL is never echoed).
* • upload — #archive-upload-form submit (phase 49) → POST * • upload — #archive-upload-form submit (phase 49, reworked to the
* phase-64 202 contract in task 05 — the phase-49 synchronous
* 200 paragraph is superseded): POST
* /api/git-sources/upload with a FormData file (NO manual * /api/git-sources/upload with a FormData file (NO manual
* Content-Type — the browser sets the multipart boundary). The * Content-Type — the browser sets the multipart boundary). The
* SAME §7.4 never-stale lifecycle: the button disables + * §7.4 never-stale lifecycle keeps its shape — the button
* relabels "Uploading…" while the request is out and is restored * disables + relabels "Uploading…" while the request is out —
* on success AND failure. 200 clears the file input, shows the * but the transfer is now short: the 202 arrives the moment the
* sync-style count line ("2 added · 1 pruned" — fmtUploadResult, * archive is safely on disk (A1). 202 → the page-local
* sources.js's fmtSyncResult convention) in the role=status * "Successfully uploaded — <file>" toast fires (showUploadToast,
* result line, announces "Archive uploaded: …" and reloads the * the phase-55 share-toast pattern; A2: safe to navigate away),
* list (the new/updated row lands with the Local badge; a * the file input clears, and the button hands over to the scan —
* re-upload simply refreshes the row — no duplicate). Non-2xx * the processing state ("Processing…", disabled, title cleared)
* inlines the server detail (422 format/name/traversal, 413 * plus startUploadPolling(): a 2 s poll of
* size, 409 busy — the messages are already user-safe) and KEEPS * GET /api/git-sources/upload/status renders the live
* the file selection — the fix is one re-pick, not a re-type. * "Processing… <file> (n/m)" label (A4 — bare during unpack; the
* full path rides the button title) and settles it: success →
* the sync-style count line (fmtUploadResult, the role=status
* result line) + the "Archive uploaded: …" announce +
* loadSources (the new/updated row lands with the Local badge;
* a re-upload refreshes the row — no duplicate; NO second toast
* — A2); failure → the sanitized server error in the role=alert
* banner + loadSources, the file selection KEPT for a one-click
* re-upload. 409 (an upload is already in progress) raises NO
* error banner — it re-attaches to the in-flight run (processing
* state + poll, never stale). Other non-2xx (422 format/name,
* 413 cap, 5xx) keep the phase-49 error banner + the kept file
* selection. The submit finally restores the button ONLY when no
* poll is active (§7.4). Boot re-attach (initUploadStatus, admin
* branch): a running scan re-enters the processing state + poll
* (a reload mid-scan re-attaches — no second upload), a terminal
* run re-renders its result line / error banner.
* • remove — a row's Remove button asks window.confirm first * • remove — a row's Remove button asks window.confirm first
* (removal prunes the documents only on the NEXT sync — the * (removal prunes the documents only on the NEXT sync — the
* confirm says so). Cancel → nothing; ok → the row button * confirm says so). Cancel → nothing; ok → the row button
@@ -64,8 +82,8 @@
* removing a source does NOT clone, import, or prune — the sync * removing a source does NOT clone, import, or prune — the sync
* service (server-side) performs that; the page's hint box says so. * service (server-side) performs that; the page's hint box says so.
* The phase-49 upload is the exception: it unpacks and scans the * The phase-49 upload is the exception: it unpacks and scans the
* single source in place, and its response counts render as the * single source in place (the phase-64 background task — 202 +
* result line. * status endpoint), and its counts render as the result line.
* *
* The shared header module loads through this script's own relative * The shared header module loads through this script's own relative
* import ("./header.js") — a hoisted import evaluated before this body * import ("./header.js") — a hoisted import evaluated before this body
@@ -354,22 +372,42 @@ wireAddForm({
idleLabel: "Add source", idleLabel: "Add source",
}); });
/* ---------- upload (POST /api/git-sources/upload) — phase 49 ------- /* ---------- upload (POST /api/git-sources/upload) — phase 64 (task 05) -------
* The archive upload form: the file input's selection is posted as * The archive upload form follows the phase-64 202 contract (A1):
* FormData (the browser sets the multipart boundary — no manual * the file input's selection is posted as FormData (the browser sets
* Content-Type). §7.4 never-stale: "Uploading…" while in flight, * the multipart boundary — no manual Content-Type), and the 202
* restored in the finally block on success AND failure. 200 → the * answers the moment the archive is safely on disk — the "Uploading…"
* input clears, the sync-style counts land in the role=status result * label covers only that short receive. Then the button HANDS OVER to
* line, the announcer confirms, and loadSources() re-renders the row * the scan: 202 → the page-local "Successfully uploaded — <file>"
* (Local badge; a re-upload refreshes the existing row — no * toast (showUploadToast — A2, safe to navigate away), the file input
* duplicate). Non-2xx → the server detail inline (role=alert; 422 * clears, and the processing state ("Processing…", disabled, title
* format/name/traversal, 413 size, 409 busy — user-safe as-is) with * cleared) + startUploadPolling() own it — a 2 s poll of
* the file selection KEPT; network failure → the fixed line. */ * GET /api/git-sources/upload/status renders the live "Processing…
* <file> (n/m)" label (A4 — bare during unpack; the full path rides
* the button title) and settles it: success → the sync-style count
* line (fmtUploadResult) in the role=status result line + the
* "Archive uploaded: …" announce + loadSources (NO second toast — it
* already fired at the 202, A2); failure → the sanitized server
* error in the role=alert banner + loadSources, the file selection
* KEPT for a one-click re-upload. 409 (an upload is already in
* progress) raises NO error banner — it re-attaches to the in-flight
* run (processing state + poll, never stale); the phase-49 "server
* detail inline for 409" branch is superseded. Other non-2xx (422
* format/name, 413 cap, 5xx) keep the phase-49 error banner + the
* kept file selection; a network failure keeps the fixed line. The
* submit finally restores the button ONLY when no poll is active
* (PLAN §7.4 — while startUploadPolling owns the button it stays
* disabled / "Processing…"). Boot re-attach (initUploadStatus, the
* admin branch): a running scan re-enters the processing state + poll
* (no second upload, no error); a terminal run re-renders its result
* line (success) or error banner (failed); idle does nothing.
* (The phase-49 synchronous 200 paragraph is superseded by phase 64.) */
/* The success line's text — the sync-result shape (sources.js's /* The success line's text — the sync-result shape (sources.js's
fmtSyncResult convention): "N added" always leads, then updated / fmtSyncResult convention): "N added" always leads, then updated /
unchanged / pruned — zero parts omitted (unchanged is shown unchanged / pruned — zero parts omitted (unchanged is shown
when nothing was added or updated). */ when nothing was added or updated). Reads exactly the keys the
upload status's detail carries (task 03's UploadOut-shaped dict). */
function fmtUploadResult(detail) { function fmtUploadResult(detail) {
const d = detail || {}; const d = detail || {};
const added = d.added || 0; const added = d.added || 0;
@@ -383,6 +421,183 @@ function fmtUploadResult(detail) {
return parts.join(" · "); return parts.join(" · ");
} }
/* Upload-success toast (phase 64 task 05, A2 — owner-locked): the
* "successfully uploaded" confirmation, the phase-55 share-toast
* pattern (frontend/assets/app.js) made page-local. A SINGLE node —
* lazy-created on the first 202 and reused (toasts never stack): a
* new toast replaces a pending one (clear the prior dismiss timer,
* re-run the entry). role="status" aria-live="polite" — on THIS page
* the toast IS the a11y announcer for the 202 (there is no other
* live-region line for it). SUCCESS-ONLY (A2): failures are the
* #archive-upload-error banner, never a toast. The .toast CSS ships
* as-is (styles.css, phase 55). */
const UPLOAD_TOAST_MS = 5000; // ~5 s auto-dismiss (A2)
let uploadToastEl = null; // the single toast node — lazy-created, reused
let uploadToastTimer = 0; // the pending auto-dismiss (replaced by a new toast)
function showUploadToast(message) {
if (!uploadToastEl) {
uploadToastEl = document.createElement("div");
uploadToastEl.className = "toast";
uploadToastEl.setAttribute("role", "status");
uploadToastEl.setAttribute("aria-live", "polite");
document.body.appendChild(uploadToastEl);
}
uploadToastEl.textContent = message; // XSS-safe text assignment
// Re-trigger the entry even when a toast is already up (a second
// upload accepted while the first toast is showing): clear the
// pending dismiss, drop the visible state, force a reflow
// (restarts the CSS transition), then show again.
clearTimeout(uploadToastTimer);
uploadToastEl.classList.remove("is-visible");
void uploadToastEl.offsetWidth; // force reflow — the entry transition restarts
uploadToastEl.classList.add("is-visible");
uploadToastTimer = setTimeout(() => {
uploadToastEl.classList.remove("is-visible"); // auto-dismiss ~5 s
}, UPLOAD_TOAST_MS);
}
/* The scan poll (phase 64 task 05): a 2 s cadence — the SYNC_POLL_MS
* house value. Single timer, one loop at a time (the guard makes a
* double-start a no-op, and the submit finally reads this same
* variable to know whether the poll OWNS the button). Each tick
* fetches GET /api/git-sources/upload/status: running → the live
* "Processing… <file> (n/m)" label (A4 — bare "Processing…" during
* the unpack phase, before any file is indexed; the full untruncated
* path rides the button title) + reschedule; success → stop + the
* result line + the announcement + the row reload (NO toast — it
* fired at the 202, A2); failed → stop + the sanitized server error
* banner + the row reload (a post-swap failure keeps the row — the
* list state may have changed), the file selection kept for a
* one-click re-upload; idle → stop + the button restored
* (defensive — a started run never returns to idle). A network blip
* retries next tick. */
const UPLOAD_POLL_MS = 2000; // the SYNC_POLL_MS house value
let uploadPollTimer = null; // null = no poll active (the finally's guard)
function stopUploadPolling() {
if (uploadPollTimer !== null) {
clearTimeout(uploadPollTimer);
uploadPollTimer = null;
}
}
/* The button's processing entry (the 202 + the 409 re-attach): from
* here the poll OWNS it — disabled, "Processing…", title cleared (a
* live file lands on it at the first tick). */
function enterUploadProcessingState() {
uploadBtn.disabled = true;
uploadBtn.textContent = "Processing…";
uploadBtn.title = "";
}
/* The idle restore (the poll's terminal branches + the submit
* finally, which calls this ONLY when no poll is active — PLAN §7.4). */
function restoreUploadButton() {
uploadBtn.disabled = false; // never stale — success OR failure
uploadBtn.textContent = "Upload & scan";
uploadBtn.removeAttribute("title");
}
function startUploadPolling() {
if (uploadPollTimer !== null) return; // one poll loop at a time
const tick = async () => {
let status = null;
try {
const r = await fetch("/api/git-sources/upload/status");
if (r.ok) status = await r.json();
} catch { /* network blip — retry next tick */ }
if (!status) {
uploadPollTimer = setTimeout(tick, UPLOAD_POLL_MS);
return;
}
// running: the live file label (A4 — bare "Processing…" during
// the unpack phase, before any file is indexed).
if (status.state === "running") {
uploadBtn.textContent =
"Processing…" +
(status.current_file ? ` ${status.current_file}` : "") +
(status.files_total > 0 ? ` (${status.files_done}/${status.files_total})` : "");
uploadBtn.title = status.current_file || ""; // full path on hover
uploadPollTimer = setTimeout(tick, UPLOAD_POLL_MS);
return;
}
stopUploadPolling();
if (status.state === "success") {
// The scan finished: the result line (the existing helper reads
// exactly these keys), the announcement, the row lands. NO toast
// here — it already fired at the 202 (A2).
const detail = status.detail || {};
if (uploadResult) {
uploadResult.textContent = fmtUploadResult(detail);
uploadResult.hidden = false;
}
announce(`Archive uploaded: ${detail.source}.`);
uploadFileInput.value = "";
restoreUploadButton();
loadSources(); // the row lands / refreshes
return;
}
if (status.state === "failed") {
// Sanitized server-side (task 03's _sanitize_error): the banner
// is the failure UI (A2), the selection stays KEPT for a
// one-click re-upload, and the list reloads (a post-swap failure
// keeps the row — the list state may have changed).
if (uploadError) {
uploadError.textContent = status.error || "The upload scan failed.";
uploadError.hidden = false;
}
restoreUploadButton();
loadSources(); // the list state may have changed
return;
}
// idle: defensive — a started run never returns to idle; just
// settle the button (the poll stopped above).
restoreUploadButton();
};
uploadPollTimer = setTimeout(tick, UPLOAD_POLL_MS);
}
/* Boot re-attach (phase 64 task 05, the admin branch only): fetch the
* upload status ONCE — a running scan re-enters the processing state
* + the poll (a reload mid-scan re-attaches instead of dead-ending —
* no second upload, no error); a finished run re-renders its result
* line ONLY (no announce, no toast — the toast fired at the 202, A2);
* a failed run re-renders its error banner; idle does nothing (and a
* blip is a no-op — the page boots honest either way). */
async function initUploadStatus() {
if (!uploadBtn) return;
let status;
try {
const r = await fetch("/api/git-sources/upload/status");
if (!r.ok) return;
status = await r.json();
} catch {
/* network blip — boot without the re-attach */
}
if (!status) return;
if (status.state === "running") {
enterUploadProcessingState();
startUploadPolling(); // the first tick carries the live file
return;
}
if (status.state === "success") {
// The last run's result line only — no announce, no toast (A2).
if (uploadResult) {
uploadResult.textContent = fmtUploadResult(status.detail);
uploadResult.hidden = false;
}
return;
}
if (status.state === "failed") {
if (uploadError) {
uploadError.textContent = status.error || "The upload scan failed.";
uploadError.hidden = false;
}
}
// idle: nothing to re-attach.
}
if (uploadFormEl && uploadFileInput && uploadBtn) { if (uploadFormEl && uploadFileInput && uploadBtn) {
uploadFormEl.addEventListener("submit", async (e) => { uploadFormEl.addEventListener("submit", async (e) => {
e.preventDefault(); e.preventDefault();
@@ -399,7 +614,7 @@ if (uploadFormEl && uploadFileInput && uploadBtn) {
if (uploadError) uploadError.hidden = true; if (uploadError) uploadError.hidden = true;
if (uploadResult) uploadResult.hidden = true; // a new attempt starts clean if (uploadResult) uploadResult.hidden = true; // a new attempt starts clean
uploadBtn.disabled = true; // §7.4: one upload per click uploadBtn.disabled = true; // §7.4: one upload per click
uploadBtn.textContent = "Uploading…"; uploadBtn.textContent = "Uploading…"; // the transfer is now short — the 202
try { try {
// Multipart from the form itself (the file input's name is // Multipart from the form itself (the file input's name is
// "file") — the browser sets the boundary; NO manual // "file") — the browser sets the boundary; NO manual
@@ -408,23 +623,35 @@ if (uploadFormEl && uploadFileInput && uploadBtn) {
method: "POST", method: "POST",
body: new FormData(uploadFormEl), body: new FormData(uploadFormEl),
}); });
if (r.ok) { if (r.status === 202) {
let data = {}; // The archive is safely on disk (A1) — the "successfully
// uploaded" moment: the toast fires NOW (A2), the file input
// clears, and the scan's poll takes over the button. The 202
// body (UploadAccepted) carries the safe source name; a body
// parse failure degrades to the picked file's name.
let name = file.name;
try { try {
data = await r.json(); const data = await r.json();
if (data && typeof data.name === "string" && data.name) name = data.name;
} catch { } catch {
/* the body is advisory — the counts line degrades gracefully */ /* body parse failure — the picked file's name degrades fine */
} }
uploadFileInput.value = ""; // 200: the archive is unpacked + scanned showUploadToast(`Successfully uploaded — ${name}`);
if (uploadResult) { uploadFileInput.value = ""; // 202: the archive is on the server
uploadResult.textContent = fmtUploadResult(data); enterUploadProcessingState();
uploadResult.hidden = false; startUploadPolling();
}
announce(`Archive uploaded: ${data.source || file.name}.`);
await loadSources(); // the new/updated row lands (Local badge)
return; return;
} }
// 422 (format/name/traversal), 413 (size), 409 (busy): the server // 409 (an upload is already in progress): NO error banner —
// re-attach to the in-flight run (never stale): the processing
// state + the poll track it to the terminal. The phase-49
// "server detail inline" branch does not apply to 409 anymore.
if (r.status === 409) {
enterUploadProcessingState();
startUploadPolling();
return;
}
// Other non-2xx (422 format/name, 413 cap, 5xx): the server
// detail inline, the file selection KEPT — the fix is one // detail inline, the file selection KEPT — the fix is one
// re-pick, not a re-type. // re-pick, not a re-type.
if (uploadError) { if (uploadError) {
@@ -437,8 +664,11 @@ if (uploadFormEl && uploadFileInput && uploadBtn) {
uploadError.hidden = false; uploadError.hidden = false;
} }
} finally { } finally {
uploadBtn.disabled = false; // never stale — success OR failure // Never stale (PLAN §7.4) — but ONLY when no poll owns the
uploadBtn.textContent = "Upload & scan"; // button: while startUploadPolling tracks the scan (202 / 409)
// it stays disabled / "Processing…", so a finally restore here
// would race the poll. No poll → the button is ours to restore.
if (uploadPollTimer === null) restoreUploadButton();
} }
}); });
} }
@@ -474,4 +704,8 @@ if (retryBtn) retryBtn.addEventListener("click", () => loadSources());
if (gateEl) gateEl.hidden = true; if (gateEl) gateEl.hidden = true;
if (contentEl) contentEl.hidden = false; if (contentEl) contentEl.hidden = false;
await loadSources(); await loadSources();
// Phase 64 (task 05): re-attach a running scan (a reload mid-scan
// resumes the Processing state) or re-render a terminal run's
// result line / error banner.
await initUploadStatus();
})(); })();
+159 -28
View File
@@ -14,6 +14,13 @@
* page "new chat" means going to the chat, fresh — the module clears * page "new chat" means going to the chat, fresh — the module clears
* the phase-14 conversation key and navigates to "/"). * the phase-14 conversation key and navigates to "/").
* *
* Phase 64 (task 04): the sync button is also the live progress face
* of the BACKGROUND archive scan (task 03) — while an upload import
* is in flight the button animates with the upload's current file
* ("Importing <file>"), settles + refreshes the catalog when it
* finishes, and re-attaches to it on page load. See the sync-button
* block below for the full two-job contract.
*
* Phase 26: the table's path links open the document in the * Phase 26: the table's path links open the document in the
* almost-fullscreen modal overlay (assets/document-modal.js) on the * almost-fullscreen modal overlay (assets/document-modal.js) on the
* SAME page — no new tab, no navigation. The link keeps its * SAME page — no new tab, no navigation. The link keeps its
@@ -36,6 +43,33 @@ import { fetchIsAdmin, initSharedHeader } from "./header.js";
* opens an error modal (same as the former header.js module — recreated * opens an error modal (same as the former header.js module — recreated
* here since the navbar button is gone). * here since the navbar button is gone).
* *
* Phase 64 (task 04): the button reports the FILE being processed, not
* just "Syncing…" — TWO jobs drive it. Each poll tick fetches BOTH
* status endpoints — GET /api/sync/status +
* GET /api/git-sources/upload/status — and applies this decision
* tree, in order (startSyncPolling):
* 1. sync running → "Syncing… <file> (n/m)" — bare "Syncing…" until
* the import's first file (clone/pull, A4);
* 2. upload running → "Importing <file> (n/m)" — the background
* archive scan (the "clicked upload, then opened
* sources" contract, A3);
* 3. sync success → the phase-32 settle (counts + catalog refresh);
* 4. sync failed → the phase-32 failure (banner + modal);
* 5. upload success → settle "Sync sources" + catalog refresh
* (loadDocs — the new documents must appear); the
* upload's counts live on the Sources page, never
* in #sync-result (A3);
* 6. upload failed → settle "Sync sources" — the failure is the
* Sources page's error banner, never this page's (A3);
* 7. both idle → retry-ready idle.
* The live label is the status endpoint's full source/relative/path
* (A4): CSS ellipsizes #sync-label; the full untruncated path also
* rides the button title (hover) and #sync-result (the aria-live
* announcer — screen readers hear it). The load-time re-attach
* (initSyncButton) re-enters a RUNNING upload the same way; a terminal
* upload is a no-op there (the boot-time loadDocs() already shows the
* current catalog).
*
* Elements: #sync-btn (the button), #sync-label (the text), * Elements: #sync-btn (the button), #sync-label (the text),
* #sync-icon (the spinner icon), #sync-result (aria-live result * #sync-icon (the spinner icon), #sync-result (aria-live result
* line), #sync-error-banner / #sync-error-text (error banner). * line), #sync-error-banner / #sync-error-text (error banner).
@@ -70,6 +104,22 @@ function fmtSyncTime(iso) {
return `${pad(d.getHours())}:${pad(d.getMinutes())}`; return `${pad(d.getHours())}:${pad(d.getMinutes())}`;
} }
/* Phase 64 (task 04): the live-file label. `kind` picks the prefix —
* "sync" → "Syncing…", "upload" → "Importing" (the background scan's
* word, A3). The current file — the status endpoint's full
* source/relative/path (A4) — is appended while one is being processed;
* the BARE prefix shows during the clone/pull (sync) or unpack (upload)
* phase, before any file is indexed. The counts appear only once the
* import has started (total > 0). CSS ellipsizes the button label; the
* same untruncated text goes to the button title + #sync-result (the
* aria-live announcer). */
function fmtSyncLabel(kind, currentFile, done, total) {
const prefix = kind === "upload" ? "Importing" : "Syncing…";
let label = currentFile ? `${prefix} ${currentFile}` : prefix;
if (total > 0) label += ` (${done}/${total})`;
return label;
}
function fmtSyncResult(detail) { function fmtSyncResult(detail) {
const d = detail || {}; const d = detail || {};
const added = d.added || 0; const added = d.added || 0;
@@ -88,15 +138,25 @@ function sanitizeSyncError(message) {
return text.length > 200 ? `${text.slice(0, 200)}…` : text; return text.length > 200 ? `${text.slice(0, 200)}…` : text;
} }
function enterSyncRunningState() { /* The single running-state entry point (phase 64 task 04: the label
* carries the live file — `kind` "sync" | "upload", the status
* endpoint's current_file + done/total). Same mechanics as before
* (disabled, aria-busy, spinning icon, no is-error) plus: the full
* untruncated path on the button title (removed when null — no file
* yet) and in #sync-result (the aria-live announcer reads the full
* live path; CSS ellipsizes the button's label span only). */
function enterSyncRunningState(kind, currentFile, done, total) {
if (!syncBtn) return; if (!syncBtn) return;
syncBtn.disabled = true; syncBtn.disabled = true;
syncBtn.setAttribute("aria-busy", "true"); syncBtn.setAttribute("aria-busy", "true");
syncBtn.removeAttribute("title"); if (currentFile) syncBtn.title = currentFile;
else syncBtn.removeAttribute("title");
syncBtn.setAttribute("aria-label", "Sync sources"); syncBtn.setAttribute("aria-label", "Sync sources");
syncBtn.classList.remove("is-error"); syncBtn.classList.remove("is-error");
if (syncIcon) syncIcon.classList.add("is-spinning"); if (syncIcon) syncIcon.classList.add("is-spinning");
if (syncLabel) syncLabel.textContent = "Syncing…"; const label = fmtSyncLabel(kind, currentFile, done, total);
if (syncLabel) syncLabel.textContent = label;
if (syncResult) syncResult.textContent = label;
} }
function settleSyncButton(label) { function settleSyncButton(label) {
@@ -193,15 +253,22 @@ function applySyncIdle(status) {
emitSyncStatus(status || { state: "idle" }); emitSyncStatus(status || { state: "idle" });
} }
/* The 2 s poll (phase 64 task 04): each tick fetches BOTH jobs — the
* sync AND the background upload scan — and applies the two-job
* decision tree in order (see the section header). The 403 on the SYNC
* fetch hides the button (the whoami backstop); a 403 on the UPLOAD
* fetch is simply "no upload" (never a hide), and a network blip on
* either fetch retries next tick. */
function startSyncPolling() { function startSyncPolling() {
if (syncPollTimer !== null) return; if (syncPollTimer !== null) return;
const tick = async () => { const tick = async () => {
let status = null; let syncStatus = null;
let uploadStatus = null;
let notAdmin = false; let notAdmin = false;
try { try {
const r = await fetch("/api/sync/status"); const r = await fetch("/api/sync/status");
if (r.status === 403) notAdmin = true; if (r.status === 403) notAdmin = true;
else if (r.ok) status = await r.json(); else if (r.ok) syncStatus = await r.json();
} catch { /* network blip — retry next tick */ } } catch { /* network blip — retry next tick */ }
if (notAdmin) { if (notAdmin) {
stopSyncPolling(); stopSyncPolling();
@@ -209,28 +276,66 @@ function startSyncPolling() {
applySyncIdle(); applySyncIdle();
return; return;
} }
if (!status) { // The SECOND job: the background upload scan (admin-only surface).
try {
const ur = await fetch("/api/git-sources/upload/status");
if (ur.ok) uploadStatus = await ur.json();
} catch { /* network blip — retry next tick */ }
if (!syncStatus) {
syncPollTimer = setTimeout(tick, SYNC_POLL_MS); syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
return; return;
} }
if (status.state === "success") { // 1. sync running: the live sync file (bare "Syncing…" until the
stopSyncPolling(); // import's first file — A4).
applySyncSuccess(status); if (syncStatus.state === "running") {
enterSyncRunningState(
"sync", syncStatus.current_file, syncStatus.files_done, syncStatus.files_total
);
syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
return; return;
} }
if (status.state === "failed") { // 2. upload running: the same animation, the upload's file (A3).
stopSyncPolling(); if (uploadStatus && uploadStatus.state === "running") {
applySyncFailure(status); enterSyncRunningState(
"upload", uploadStatus.current_file, uploadStatus.files_done, uploadStatus.files_total
);
syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
return; return;
} }
if (status.state === "idle") { if (syncStatus.state === "success") {
stopSyncPolling(); stopSyncPolling();
applySyncIdle(status); applySyncSuccess(syncStatus);
return; return;
} }
// Still running: keep button state honest and re-schedule. if (syncStatus.state === "failed") {
enterSyncRunningState(); stopSyncPolling();
syncPollTimer = setTimeout(tick, SYNC_POLL_MS); applySyncFailure(syncStatus);
return;
}
// 5. upload success: settle + catalog refresh (A3 — the upload's
// counts live on the Sources page; #sync-result stays empty).
if (uploadStatus && uploadStatus.state === "success") {
stopSyncPolling();
settleSyncButton("Sync sources");
if (syncResult) syncResult.textContent = "";
hideSyncError();
emitSyncStatus({ state: "idle" });
loadDocs();
return;
}
// 6. upload failed: settle only — the failure is the Sources page's
// error banner, never this page's (A3).
if (uploadStatus && uploadStatus.state === "failed") {
stopSyncPolling();
settleSyncButton("Sync sources");
if (syncResult) syncResult.textContent = "";
hideSyncError();
emitSyncStatus({ state: "idle" });
return;
}
// 7. both idle: settle retry-ready.
stopSyncPolling();
applySyncIdle(syncStatus);
}; };
syncPollTimer = setTimeout(tick, SYNC_POLL_MS); syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
} }
@@ -253,8 +358,10 @@ async function startSync() {
return; return;
} }
if (r.status === 202 || r.status === 409) { if (r.status === 202 || r.status === 409) {
enterSyncRunningState(); // Phase 64: the run is just starting (model check / clone-pull) —
if (syncResult) syncResult.textContent = ""; // bare "Syncing…" until the first polled file (A4); entering the
// running state also clears #sync-result with the same label.
enterSyncRunningState("sync", null, 0, 0);
hideSyncError(); hideSyncError();
if (lastSyncState !== "running") emitSyncStatus({ state: "running" }); if (lastSyncState !== "running") emitSyncStatus({ state: "running" });
startSyncPolling(); startSyncPolling();
@@ -268,8 +375,12 @@ async function startSync() {
}); });
} }
/* Load-time re-attach (ADMIN ONLY): a running run re-enters running state, /* Load-time re-attach (ADMIN ONLY): a running run re-enters running
* a terminal run renders its last result. */ * state, a terminal run renders its last result. Phase 64 (A3): with
* the sync IDLE, an in-flight background upload scan adopts the button
* the same way — the "user clicked upload, then opened sources" case;
* a terminal upload is a no-op (the boot-time loadDocs() already shows
* the current catalog). */
async function initSyncButton() { async function initSyncButton() {
if (!syncBtn) return; if (!syncBtn) return;
if (!(await fetchIsAdmin())) return; if (!(await fetchIsAdmin())) return;
@@ -281,16 +392,36 @@ async function initSyncButton() {
status = await r.json(); status = await r.json();
} catch { return; } } catch { return; }
if (status.state === "running") { if (status.state === "running") {
enterSyncRunningState(); enterSyncRunningState(
"sync", status.current_file, status.files_done, status.files_total
);
emitSyncStatus(status); emitSyncStatus(status);
startSyncPolling(); startSyncPolling();
} else if (status.state === "success") { return;
applySyncSuccess(status);
} else if (status.state === "failed") {
applySyncFailure(status);
} else {
applySyncIdle(status);
} }
if (status.state === "success") {
applySyncSuccess(status);
return;
}
if (status.state === "failed") {
applySyncFailure(status);
return;
}
// Sync idle: check the SECOND job — an in-flight upload scan re-attaches.
let upload;
try {
const ur = await fetch("/api/git-sources/upload/status");
if (ur.ok) upload = await ur.json();
} catch { /* network blip — the idle settle below is still honest */ }
if (upload && upload.state === "running") {
enterSyncRunningState(
"upload", upload.current_file, upload.files_done, upload.files_total
);
emitSyncStatus({ state: "running" });
startSyncPolling();
return;
}
applySyncIdle(status);
} }
if (syncBtn) { if (syncBtn) {
+12 -1
View File
@@ -1496,7 +1496,18 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
@media (prefers-reduced-motion: reduce) { @media (prefers-reduced-motion: reduce) {
.sync-btn .sync-icon.is-spinning { animation: none; } .sync-btn .sync-icon.is-spinning { animation: none; }
} }
.sync-label { display: inline; } /* Phase 64: the live-file label ("Syncing… <file> (n/m)" / "Importing
<file> (n/m)") — a long source/relative/path ellipsizes inside the
pill; the full path lives in the button title + #sync-result (the
aria-live announcer). */
.sync-label {
display: inline-block;
max-width: min(16rem, 40vw);
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
vertical-align: bottom;
}
/* Last-result announcer — soft ink, small mono */ /* Last-result announcer — soft ink, small mono */
.sync-result { .sync-result {
display: block; display: block;
+21 -10
View File
@@ -179,16 +179,27 @@
<p class="git-source-error" id="git-source-error" role="alert" hidden></p> <p class="git-source-error" id="git-source-error" role="alert" hidden></p>
</form> </form>
<!-- Phase 49 (owner permission 2026-08-28): the archive upload form <!-- Phase 49 (owner permission 2026-08-28): the archive upload
replaces the phase-38 local-directory form — an uploaded form replaces the phase-38 local-directory form — an
.tar/.tar.gz/.tgz/.zip is unpacked under BOR_UPLOAD_DIR and uploaded .tar/.tar.gz/.tgz/.zip is unpacked under
scanned immediately; the same filename replaces the source in BOR_UPLOAD_DIR and scanned; the same filename replaces the
place (no new folder, no duplicate row). The file control is source in place (no new folder, no duplicate row). The file
labeled (visible <label for=…> — WCAG input-label rule); the control is labeled (visible <label for=…> — WCAG
button runs the §7.4 never-stale lifecycle ("Uploading…" input-label rule); the button runs the §7.4 never-stale
while the POST is out, restored on success AND failure); lifecycle ("Uploading…" while the POST is out). Phase 64
non-2xx shows the server detail inline (role=alert), 200 (task 05) reworks the rest to the 202 contract (the
shows the sync-style counts (role=status). --> phase-49 synchronous 200 paragraph is superseded): the 202
arrives the moment the archive is safely on disk (A1) — a
JS-created "Successfully uploaded — <file>" toast fires
then (A2 — the phase-55 .toast node, no markup here; safe
to navigate away) and the button settles into the live
"Processing… <file> (n/m)" label (A4 — the full path rides
the button title) driven by the 2 s poll of
GET /api/git-sources/upload/status, until the success line
(role=status) or the sanitized error banner (role=alert)
lands; 409 re-attaches to the in-flight run — no error
banner; the other non-2xx still show the server detail
inline. -->
<form id="archive-upload-form"> <form id="archive-upload-form">
<label for="archive-upload-file">Upload a source archive (.tar, .tar.gz, .tgz, .zip)</label> <label for="archive-upload-file">Upload a source archive (.tar, .tar.gz, .tgz, .zip)</label>
<input id="archive-upload-file" name="file" type="file" <input id="archive-upload-file" name="file" type="file"
+8 -2
View File
@@ -113,8 +113,14 @@
to pull the latest and re-import. to pull the latest and re-import.
</p> </p>
</div> </div>
<!-- #sync-result is the aria-live announcer for the last sync <!-- #sync-result is the aria-live announcer: the last sync
result ("N added · …") — renders off the "bor:sync-status" event. --> result ("N added · …") when a sync settles, and — phase 64 —
the LIVE file label while either job runs ("Syncing… <file>
(n/m)" / "Importing <file> (n/m)"), UNTRUNCATED (the button's
label span ellipsizes; screen readers hear the full
source/relative path, which also rides the button title).
After an upload settles it stays empty — the upload's counts
live on the Sources page (A3). -->
<span class="sync-result" id="sync-result" role="status" aria-live="polite"></span> <span class="sync-result" id="sync-result" role="status" aria-live="polite"></span>
<!-- Sync failure banner — role="alert" so a failed sync is announced. --> <!-- Sync failure banner — role="alert" so a failed sync is announced. -->
<div class="kb-banner is-error" id="sync-error-banner" role="alert" hidden> <div class="kb-banner is-error" id="sync-error-banner" role="alert" hidden>
+93
View File
@@ -0,0 +1,93 @@
"""Phase 64 E2E helper — a delay-injecting reverse proxy in front of the
mock LLM.
The mock LLM (``mock_llm.py``) answers instantly: a real archive scan of
dozens of files finishes in well under a second and never outlives the
UI's 2 s status poll — the phase-64 live file labels ("Processing…
<file>", "Importing <file>", "Syncing… <file>"), the at-202 toast →
navigate-away contract, and the mid-scan reload re-attach would all be
races against the mock. This proxy sits between the app under test and
the mock LLM and sleeps ``SLOW_LLM_DELAY_S`` seconds (default 0.15)
before forwarding each request, so a scan's duration is deterministic
(≈ the run's number of LLM requests × the delay — for an N-file
archive/sync that is N + 3: the ``check_models`` embed + chat probe,
one embed per file, and the change-gated overview chat). Both the
tests' ~100 ms status polling (the deterministic layer) and the UI's
2 s poll (the UI layer) then observe the running state, the current
file, and the counts reliably.
Everything else is byte-transparent: method, path, query, headers, and
body are forwarded verbatim; the upstream response's status and body
come back as-is (``content-encoding`` / ``content-length`` are dropped
— httpx has already decoded the body and starlette recomputes the
length). The mock's responses are all finite (its SSE streams end with
``[DONE]``), so the proxy reads each body to completion before
answering.
Run it the conftest way (a suite's module ``app_server`` fixture spawns
it as a subprocess):
uv run python -m uvicorn tests.e2e.slow_llm:app --port 8902
with ``E2E_MOCK_PORT`` (upstream, default 8901) and ``SLOW_LLM_DELAY_S``
(delay, default 0.15) in its environment.
"""
from __future__ import annotations
import asyncio
import os
import httpx
from fastapi import FastAPI, Request, Response
#: The mock LLM this proxy forwards to (the conftest's MOCK_PORT).
MOCK_PORT = int(os.environ.get("E2E_MOCK_PORT", "8901"))
UPSTREAM = f"http://127.0.0.1:{MOCK_PORT}"
#: Per-request delay in seconds — each suite's proxy fixture picks its
#: own (passed through the subprocess env).
DELAY_S = float(os.environ.get("SLOW_LLM_DELAY_S", "0.15"))
app = FastAPI()
_client: httpx.AsyncClient | None = None
def _get_client() -> httpx.AsyncClient:
global _client
if _client is None:
_client = httpx.AsyncClient(base_url=UPSTREAM, timeout=60.0)
return _client
@app.on_event("shutdown")
async def _close_client() -> None:
global _client
if _client is not None:
await _client.aclose()
_client = None
@app.api_route("/{path:path}", methods=["GET", "POST", "PUT", "DELETE", "PATCH"])
async def proxy(path: str, request: Request) -> Response:
"""Sleep ``DELAY_S``, then forward the request to the mock LLM."""
await asyncio.sleep(DELAY_S)
body = await request.body()
headers = {
k: v for k, v in request.headers.items() if k.lower() not in ("host", "content-length")
}
upstream = await _get_client().request(
request.method,
f"/{path}",
content=body,
headers=headers,
params=dict(request.query_params),
)
resp_headers = {
k: v
for k, v in upstream.headers.items()
if k.lower()
not in ("content-length", "content-encoding", "transfer-encoding", "connection")
}
return Response(
content=upstream.content, status_code=upstream.status_code, headers=resp_headers
)
+169 -39
View File
@@ -11,10 +11,26 @@ form is gone, replaced by this form): an uploaded ``.tar``/``.tar.gz``/
``.tgz``/``.zip`` is safely unpacked under ``BOR_UPLOAD_DIR/<name>/`` ``.tgz``/``.zip`` is safely unpacked under ``BOR_UPLOAD_DIR/<name>/``
(name = filename minus the archive suffix), the ``git_sources`` row is (name = filename minus the archive suffix), the ``git_sources`` row is
upserted (``kind='local'``, no duplicates), and the source is **scanned upserted (``kind='local'``, no duplicates), and the source is **scanned
synchronously in the request** (single-source ``import_sources`` with in a background task** (phase 64, task 03 — owner-locked A1: the POST
``prune=True`` + the change-gated overview refresh) — the real pipeline, answers **202 the moment the archive is safely on disk** — the
against the deterministic mock LLM (no real models, no network beyond "Successfully uploaded — <source>" toast fires then and the user may
the app itself). navigate away — while unpack → swap → row upsert → model check →
single-source ``import_sources`` with ``prune=True`` + the change-gated
overview refresh run server-side behind
``GET /api/git-sources/upload/status``, the phase-32 ``SyncStatus``
pattern with the phase-64 ``current_file``/counts). The result line and
the row land from the status ``success`` (same ``UploadOut`` counts,
uncompressed in shape) — the real pipeline, against the deterministic
mock LLM (no real models, no network beyond the app itself).
**Timing fixture (phase 64):** the mock LLM answers instantly, so this
module's app boots behind ``tests/e2e/slow_llm.py`` — a delay-injecting
reverse proxy in front of it (``SLOW_DELAY_S`` per request). A 2-file
scan is 5 LLM requests ≈ 5 × 0.6 s ≈ 3 s: long enough to outlive the
UI's 2 s status poll, so the button's literal
"Uploading… → Processing… → restored" lifecycle is observable
(the "Processing…" tick even carries the live file, A4) instead of
racing the mock.
The archives are **built in-test** with Python's ``tarfile`` over The archives are **built in-test** with Python's ``tarfile`` over
``tmp_path`` fixture files carrying markdown sentinels (``ALPHA-…`` / ``tmp_path`` fixture files carrying markdown sentinels (``ALPHA-…`` /
@@ -38,24 +54,33 @@ Contract under test:
the upload form is in its place with the labeled file input (accept the upload form is in its place with the labeled file input (accept
= the four archive extensions), the "Upload & scan" button, and the = the four archive extensions), the "Upload & scan" button, and the
hint explains unpack/scan + in-place replace; hint explains unpack/scan + in-place replace;
* **upload → scan → list** (§7.4 never-stale): the button shows * **upload → 202 + toast → background scan → list** (§7.4 never-stale,
"Uploading…" while the POST is in flight (the request is held in the phase-64 A1/A2): the button shows "Uploading…" while the POST is in
browser via ``page.route`` so the in-flight state is deterministic), flight (the request is held in the browser via ``page.route`` so the
then restores; the result line shows the added count; the list gains in-flight state is deterministic); at the 202 the "Successfully
exactly one row for ``e2e-upload`` with the **Local** badge; uploaded — <source>" toast fires (``.toast.is-visible``,
``GET /api/docs`` lists both sentinel files under source ``role="status"``) WHILE the scan is still running, and the button
``e2e-upload``; the RAG catalog (``/sources.html``) shows them; hands over to the scan — "Processing…" (the live-file tick carries
the current file, A4) — then restores when the status ``success``
lands: the result line shows the added count; the list gains exactly
one row for ``e2e-upload`` with the **Local** badge; ``GET
/api/docs`` lists both sentinel files under source ``e2e-upload``;
the RAG catalog (``/sources.html``) shows them;
* **re-upload, same filename** → in-place replace: the result line * **re-upload, same filename** → in-place replace: the result line
shows the prune, the list still has exactly ONE ``e2e-upload`` row shows the prune, the SECOND RUN'S STATUS ``detail`` carries the
(no duplicate), the KB shows the changed ``alpha`` + the new prune/refresh counts, the list still has exactly ONE ``e2e-upload``
row (no duplicate), the KB shows the changed ``alpha`` + the new
``gamma`` and NOT the dropped ``beta``, and the on-disk folder holds ``gamma`` and NOT the dropped ``beta``, and the on-disk folder holds
only the new archive's files; only the new archive's files;
* **bad file** → inline 422 (role=alert) naming the accepted formats, * **bad file** → inline 422 (role=alert) naming the accepted formats
button restored, the file selection kept, the list unchanged, and a (UNCHANGED — the name/format/cap gates are inline, pre-202, exactly
subsequent good upload still works (the form is not wedged); as before), button restored, the file selection kept, the list
unchanged, and a subsequent good upload still works (the form is not
wedged);
* **anonymous** → the sign-in gate (``#git-sources-gate``) shows, the * **anonymous** → the sign-in gate (``#git-sources-gate``) shows, the
manager (and thus the upload form) stays hidden, and manager (and thus the upload form) stays hidden, and
``POST /api/git-sources/upload`` is 403. ``POST /api/git-sources/upload`` is 403 — as is the phase-64
``GET /api/git-sources/upload/status`` (same wall).
Test → story mapping (Playwright Mapping Rule): Test → story mapping (Playwright Mapping Rule):
1. ``test_form_swapped`` 1. ``test_form_swapped``
@@ -86,6 +111,7 @@ from e2e.auth_helpers import login
from e2e.conftest import ( from e2e.conftest import (
ADMIN_PASSWORD, ADMIN_PASSWORD,
APP_PORT, APP_PORT,
MOCK_PORT,
SESSION_SECRET, SESSION_SECRET,
USE_REAL_LLM, USE_REAL_LLM,
_wait_http, _wait_http,
@@ -94,6 +120,17 @@ from e2e.conftest import (
REPO = Path(__file__).resolve().parents[2] REPO = Path(__file__).resolve().parents[2]
APP_URL = f"http://127.0.0.1:{APP_PORT}" APP_URL = f"http://127.0.0.1:{APP_PORT}"
#: The slow-LLM proxy's port (the conftest's mock LLM stays on MOCK_PORT).
SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT", "8902"))
SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}"
#: Per-LLM-request delay on the proxy — a 2-file scan is 5 LLM requests
#: (the check_models embed + chat probe, one embed per file, the
#: change-gated overview chat) ≈ 5 × 0.6 s ≈ 3 s: the scan outlives the
#: UI's 2 s status poll, so the button's "Uploading… → Processing… →
#: restored" lifecycle (with the live-file tick, A4) is observable.
SLOW_DELAY_S = "0.6"
GIT_SOURCES_URL = "/git-sources.html" GIT_SOURCES_URL = "/git-sources.html"
SOURCES_URL = "/sources.html" SOURCES_URL = "/sources.html"
@@ -184,17 +221,47 @@ def tarball_v2(tmp_path_factory: pytest.TempPathFactory) -> Path:
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V2_FILES) return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V2_FILES)
@pytest.fixture(scope="module")
def slow_llm(mock_llm: int) -> Iterator[int]:
"""The delay-injecting reverse proxy in front of the mock LLM
(tests/e2e/slow_llm.py) — this suite's timing fixture: the phase-64
button lifecycle ("Uploading… → Processing… → restored") needs the
2-file scan to outlive the UI's 2 s status poll (see
``SLOW_DELAY_S``)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S
env["E2E_MOCK_PORT"] = str(MOCK_PORT)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app",
"--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{SLOW_URL}/v1/models")
yield SLOW_PORT
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module") @pytest.fixture(scope="module")
def app_server( def app_server(
mock_llm: int, mock_llm: int,
slow_llm: int,
upload_dir: Path, upload_dir: Path,
tmp_path_factory: pytest.TempPathFactory, tmp_path_factory: pytest.TempPathFactory,
) -> Iterator[str]: ) -> Iterator[str]:
"""The real app under test — per-module env: uploads unpack into a """The real app under test — per-module env: the LLM base URL is the
scratch dir and the env git list is forced empty (the dev ``.env``'s SLOW PROXY in front of the mock (the timing fixture), uploads unpack
``BOR_GIT_SOURCES`` must not render as env rows on the initially into a scratch dir and the env git list is forced empty (the dev
empty table). No sync is triggered here — the upload's own scan is ``.env``'s ``BOR_GIT_SOURCES`` must not render as env rows on the
the pipeline under test.""" initially empty table). No sync is triggered here — the upload's own
scan is the pipeline under test."""
env = dict(os.environ) env = dict(os.environ)
env.pop("DEBUGPY", None) env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e" env["BOR_ENVIRONMENT"] = "e2e"
@@ -202,7 +269,7 @@ def app_server(
env["BOR_LLM_BASE_URL"] = ( env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1" "https://aipi.reeseapps.com/v1"
if USE_REAL_LLM if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1" else f"{SLOW_URL}/v1"
) )
# Mock-calibrated threshold (conftest pattern) — no chat turn is # Mock-calibrated threshold (conftest pattern) — no chat turn is
# ever sent in this suite, but the app boots with the same env shape. # ever sent in this suite, but the app boots with the same env shape.
@@ -286,9 +353,10 @@ def _docs(page: Page, app_url: str) -> list[tuple[str, str]]:
def _upload_via_page(page: Page, archive: Path) -> str: def _upload_via_page(page: Page, archive: Path) -> str:
"""Pick the archive, submit the form, and wait for the result line """Pick the archive, submit the form, and wait for the result line
(the 200 path) — returns its text. The failing path is asserted (the phase-64 202 path: toast at the 202, then the button's status
explicitly by the bad-file test, so any non-result outcome here is polling renders the line from the run's ``success``) — returns its
a test error.""" text. The failing path is asserted explicitly by the bad-file test,
so any non-result outcome here is a test error."""
page.set_input_files("#archive-upload-file", str(archive)) page.set_input_files("#archive-upload-file", str(archive))
page.click("#archive-upload-btn") page.click("#archive-upload-btn")
result = page.locator("#archive-upload-result") result = page.locator("#archive-upload-result")
@@ -298,6 +366,24 @@ def _upload_via_page(page: Page, archive: Path) -> str:
return text return text
def _wait_upload_running(page: Page, app_url: str, timeout_s: float = 15.0) -> dict[str, Any]:
"""Poll (cookie-authenticated) the upload status endpoint until the
run is ``running`` — the phase-64 single source of truth for the
background scan (A1)."""
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = page.request.get(f"{app_url}/api/git-sources/upload/status")
assert r.status == 200, r.text
body = r.json()
if body["state"] == "running":
return body
if body["state"] in ("success", "failed"):
raise AssertionError(f"the scan settled too fast to observe: {body}")
time.sleep(0.1)
raise AssertionError(f"the scan never entered running: {body}")
def _hold_upload_request(page: Page, hold_s: float) -> None: def _hold_upload_request(page: Page, hold_s: float) -> None:
"""Intercept the upload POST and hold the REQUEST in the browser for """Intercept the upload POST and hold the REQUEST in the browser for
``hold_s`` seconds before letting it reach the server. While it is ``hold_s`` seconds before letting it reach the server. While it is
@@ -365,11 +451,15 @@ def test_upload_scans_and_lists(
page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path
) -> None: ) -> None:
"""One real upload through the page: while the POST is in flight the """One real upload through the page: while the POST is in flight the
button is disabled and reads "Uploading…"; on the 200 it restores, button is disabled and reads "Uploading…"; at the 202 the
the result line shows the added count (2), the file input clears, "Successfully uploaded — <source>" toast fires (A2) WHILE the
the list gains exactly ONE row for ``e2e-upload`` with the Local background scan is still running and the button hands over to it —
badge, ``/api/docs`` lists both sentinel files under the source, and "Processing…" (the 2 s poll tick carries the live file, A4); when
the RAG catalog shows them where the admin expects them.""" the status ``success`` lands the button restores, the result line
shows the added count (2), the file input clears, the list gains
exactly ONE row for ``e2e-upload`` with the Local badge,
``/api/docs`` lists both sentinel files under the source, and the
RAG catalog shows them where the admin expects them."""
page.set_default_timeout(30_000) page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url) _admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(0) expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
@@ -378,7 +468,7 @@ def test_upload_scans_and_lists(
result = page.locator("#archive-upload-result") result = page.locator("#archive-upload-result")
# Hold the upload request in the browser: the in-flight state below # Hold the upload request in the browser: the in-flight state below
# cannot race the (fast) mock-LLM scan while it is held. # cannot race the receive while it is held.
_hold_upload_request(page, hold_s=0.8) _hold_upload_request(page, hold_s=0.8)
page.set_input_files("#archive-upload-file", str(tarball_v1)) page.set_input_files("#archive-upload-file", str(tarball_v1))
btn.click() btn.click()
@@ -388,12 +478,35 @@ def test_upload_scans_and_lists(
expect(btn).to_have_text("Uploading…") expect(btn).to_have_text("Uploading…")
expect(result).to_be_hidden() expect(result).to_be_hidden()
# The request goes out, the server unpacks + scans (mock LLM) and # The request goes out; the server stores the archive and answers
# answers 200 → the result line shows the added count. # 202 the moment it is safely on disk (A1) → the toast fires NOW
# (A2) — while the scan is still running — and the button hands
# over to the scan (bare "Processing…" — A4: no file yet during the
# unpack phase).
toast = page.locator(".toast")
expect(toast).to_have_count(1, timeout=UPLOAD_TIMEOUT_MS)
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
assert toast.get_attribute("role") == "status"
expect(toast).to_have_text(f"Successfully uploaded — {SOURCE_NAME}")
expect(result).to_be_hidden()
expect(btn).to_be_disabled()
expect(btn).to_have_text("Processing…", timeout=5_000)
# The scan is running server-side (the status endpoint is the
# single source of truth, A1) — the run the UI's poll tracks.
_wait_upload_running(page, app_url)
# …and the button's 2 s poll tick renders the live file label
# ("Processing… <file> (n/m)", A4).
expect(btn).to_have_text(
re.compile(rf"Processing… {re.escape(SOURCE_NAME)}/.+\.md"),
timeout=UPLOAD_TIMEOUT_MS,
)
# The status success lands → the result line (the same UploadOut
# counts) + the never-stale restore (input cleared).
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS) expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
expect(result).to_have_text("2 added") expect(result).to_have_text("2 added")
# Never stale: the button restored on success and the input cleared.
expect(btn).to_be_enabled() expect(btn).to_be_enabled()
expect(btn).to_have_text("Upload & scan") expect(btn).to_have_text("Upload & scan")
expect(page.locator("#archive-upload-file")).to_have_value("") expect(page.locator("#archive-upload-file")).to_have_value("")
@@ -434,7 +547,9 @@ def test_reupload_replaces_in_place(
upload_dir: Path, upload_dir: Path,
) -> None: ) -> None:
"""v1 then v2 under the SAME filename (``e2e-upload.tar.gz``): the """v1 then v2 under the SAME filename (``e2e-upload.tar.gz``): the
result line shows the prune, the list still has exactly ONE result line shows the prune, the SECOND RUN'S STATUS ``detail``
carries the prune/refresh counts (phase 64 — the line is rendered
from the status success), the list still has exactly ONE
``e2e-upload`` row (the row count for that source is invariant — no ``e2e-upload`` row (the row count for that source is invariant — no
duplicate), the KB shows the changed ``alpha`` + the new ``gamma`` duplicate), the KB shows the changed ``alpha`` + the new ``gamma``
and NOT the dropped ``beta``, and the on-disk folder holds only the and NOT the dropped ``beta``, and the on-disk folder holds only the
@@ -442,7 +557,7 @@ def test_reupload_replaces_in_place(
page.set_default_timeout(30_000) page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url) _admin_git_sources_page(page, app_url)
# Baseline: v1 through the page (200 → "2 added", one row). # Baseline: v1 through the page (202 → "2 added", one row).
assert _upload_via_page(page, tarball_v1) == "2 added" assert _upload_via_page(page, tarball_v1) == "2 added"
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1) expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
@@ -452,6 +567,19 @@ def test_reupload_replaces_in_place(
result = page.locator("#archive-upload-result") result = page.locator("#archive-upload-result")
expect(result).to_have_text(re.compile(r"\d+ pruned")) expect(result).to_have_text(re.compile(r"\d+ pruned"))
# The SECOND RUN's status ``detail`` shows the prune/refresh counts
# (phase 64: the result line is rendered from this success).
r = page.request.get(f"{app_url}/api/git-sources/upload/status")
assert r.status == 200, r.text
status = r.json()
assert status["state"] == "success", status
detail = status["detail"]
assert detail["source"] == SOURCE_NAME
assert detail["files"] == 2
assert detail["added"] == 1 # gamma — new in v2
assert detail["updated"] == 1 # alpha — changed in v2
assert detail["pruned"] == 1 # beta — dropped in v2
# No duplicate: exactly ONE row for that source (and one row total). # No duplicate: exactly ONE row for that source (and one row total).
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1) expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1) expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
@@ -552,6 +680,8 @@ def test_anonymous_gate(page: Page, app_url: str, db_ready: None) -> None:
# The upload route 403s anonymous callers (require_admin runs before # The upload route 403s anonymous callers (require_admin runs before
# the multipart body is parsed — the body is a stand-in, the # the multipart body is parsed — the body is a stand-in, the
# test_local_directory_sources.py pattern for this route). # test_local_directory_sources.py pattern for this route)…
r = page.request.post(f"{app_url}/api/git-sources/upload", data={"file": ""}) r = page.request.post(f"{app_url}/api/git-sources/upload", data={"file": ""})
assert r.status == 403 assert r.status == 403
# …and so does the phase-64 upload STATUS endpoint (same wall).
assert page.request.get(f"{app_url}/api/git-sources/upload/status").status == 403
+758
View File
@@ -0,0 +1,758 @@
"""Phase 64 story E2E (Playwright): real-time progress for sync + upload.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_sync_upload_progress.py -v --no-cov
The story gate for the phase's executable proof (owner-locked A1–A5):
both long-running KB jobs report **which file is being processed right
now** — not just "Syncing…"/"Uploading…" — and the archive upload is
fully **backgrounded**: ``POST /api/git-sources/upload`` answers 202 the
moment the archive is on disk (the "Successfully uploaded — <file>"
toast fires — the user may navigate away), the unpack/scan continues
server-side behind ``GET /api/git-sources/upload/status`` (the phase-32
``SyncStatus`` pattern), and the RAG-page sync button
(``/sources.html``) animates with the upload's current file while that
scan runs.
**Timing fixture (the phase's fixture note):** the mock LLM indexes
fast — the in-progress state is real but brief (a 25-file scan against
it takes ≈0.4 s, well under the UI's 2 s status poll). This module's
app therefore boots behind ``tests/e2e/slow_llm.py`` — a delay-injecting
reverse proxy in front of the mock LLM (``SLOW_DELAY_S`` per request →
a 25-file scan is 28 LLM requests ≈ 4.2 s), so the scan outlives the
2 s poll and the live-file label is asserted at BOTH layers the task
pins:
* **deterministic** — the status endpoints (``page.request`` / the
concurrent recorder, ~100 ms cadence): ``state == "running"`` with a
non-null ``current_file`` (``source/relative/path``) observed at some
tick, the counts advancing, and the file-less ticks (unpack/row/
model probe — A4) preceding the first file tick;
* **UI** — polling the button labels for the ``Importing`` /
``Syncing…`` / ``Processing…`` prefix plus a file path (generous
timeout), which the pages' own 2 s poll ticks render.
The upload archive is built in-test with Python's ``tarfile`` from
**25 small ``.md`` files** (``e2e-prog.tar.gz`` → source ``e2e-prog``);
the sync subject is a host temp dir (``sync-corpus/``, 25 small
``.md`` files under ``notes/``) registered as a ``kind=local`` row —
the ``test_sync_button.py`` / ``test_local_directory_sources.py``
fixture styles. Per-module app env (the conftest pattern):
``BOR_UPLOAD_DIR`` scratch, ``BOR_GIT_SOURCES`` forced empty (the sync
sources are this suite's own local row), ``BOR_LLM_BASE_URL`` the slow
proxy.
Contract under test:
* **toast → navigate away (A2 + A3)**: on ``/git-sources.html`` the
"Successfully uploaded — <source>" toast (``.toast.is-visible``,
``role="status"``) appears while the scan is still running;
navigating to ``/sources.html`` shows the sync button animating
(spinner + ``aria-busy``) with the ``Importing <file>`` label; on
completion the button settles to "Sync sources" (no error UI,
``#sync-result`` stays empty — the upload's counts never render
there, A3) and the catalog shows the uploaded documents (the
phase-63 listing, untouched);
* **upload progress (A4)**: during the scan the status endpoint
reports a non-null ``current_file`` (``source/relative/path`` shape,
full denominator, advancing counts) at running ticks, the upload
button shows "Processing… <file>" (bare "Processing…" during unpack)
before the result line lands, and the toast fired earlier in the run
— the result line itself comes from the status ``success``;
* **sync live file (A4)**: a multi-file local source; clicking
**Sync sources** on ``/sources.html`` shows "Syncing…" (bare, the
pre-64 click state) then "Syncing… <file> (n/m)" (both layers), then
the pre-64 success settle — "Synced HH:MM" + the counts result line —
preserved, plus the file in the label;
* **reload re-attach (A1 + A2)**: starting an upload and reloading
``/git-sources.html`` mid-scan leaves the button in the Processing
state (disabled) with no error banner and NO second upload (the
status endpoint's single run is still the one from before the
reload — pinned on its ``started_at``); it then settles with the
result line and the list shows exactly one row for the archive.
Test → story mapping (Playwright Mapping Rule):
1. ``test_upload_toast_then_navigate_away``
2. ``test_upload_progress_shows_current_file``
3. ``test_sync_live_file_label``
4. ``test_upload_reattach_after_reload``
"""
from __future__ import annotations
import io
import os
import re
import subprocess
import sys
import tarfile
import threading
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import httpx
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.db import SessionLocal
from app.models import GitSource
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
APP_PORT,
MOCK_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
APP_URL = f"http://127.0.0.1:{APP_PORT}"
GIT_SOURCES_URL = "/git-sources.html"
SOURCES_URL = "/sources.html"
#: The slow-LLM proxy's port (the conftest's mock LLM stays on MOCK_PORT).
SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT", "8902"))
SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}"
#: Per-LLM-request delay on the proxy — the scan's duration becomes
#: deterministic: an N-file archive scan issues N + 3 LLM requests
#: (the check_models embed + chat probe, one embed per file, the
#: change-gated overview chat), so a 25-file upload takes ≈ 28 × 0.15 s
#: ≈ 4.2 s — long enough to outlive the UI's 2 s status poll (see the
#: module docstring's timing-fixture note).
SLOW_DELAY_S = "0.15"
#: The uploaded archive: 25 small docs under ``docs/`` (the phase's
#: fixture note — 20+ files so the scan outlives the 2 s poll).
UPLOAD_NAME = "e2e-prog"
UPLOAD_ARCHIVE = f"{UPLOAD_NAME}.tar.gz"
N_FILES = 25
UPLOAD_FILES: dict[str, str] = {
f"docs/{i:02d}.md": f"# Doc {i:02d}\n\nDeterministic upload content {i:02d}.\n"
for i in range(N_FILES)
}
#: The local sync source (host temp dir — the app runs on the same
#: machine): 25 small docs under ``notes/`` (the current_file's
#: source/relative/path shape has a directory level).
SYNC_SOURCE_DIR = "sync-corpus"
SYNC_FILES: dict[str, str] = {
f"notes/{i:02d}.md": f"# Sync doc {i:02d}\n\nDeterministic sync content {i:02d}.\n"
for i in range(N_FILES)
}
#: "Synced HH:MM" — the local-time last-result label (sources.js's
#: fmtSyncTime), any hour/minute (test_sync_button.py's pattern).
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}")
#: Generous settle budget: a 25-file scan against the slowed LLM is
#: ≈4.2 s; the UI's 2 s poll settles at most one tick after the
#: terminal state lands.
SETTLE_TIMEOUT_MS = 45_000
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
def _build_targz(path: Path, files: dict[str, str]) -> Path:
"""A deterministic ``.tar.gz`` (mtime 0) over the given files."""
with tarfile.open(path, "w:gz") as tf:
for rel, content in files.items():
data = content.encode("utf-8")
info = tarfile.TarInfo(rel)
info.size = len(data)
info.mtime = 0
tf.addfile(info, io.BytesIO(data))
return path
@pytest.fixture(scope="module")
def slow_llm(mock_llm: int) -> Iterator[int]:
"""The delay-injecting reverse proxy in front of the mock LLM
(tests/e2e/slow_llm.py) — this suite's timing fixture: the live-file
contract needs the scan to outlive the UI's 2 s poll (see
``SLOW_DELAY_S``)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S
env["E2E_MOCK_PORT"] = str(MOCK_PORT)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app",
"--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{SLOW_URL}/v1/models")
yield SLOW_PORT
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def upload_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The app's ``BOR_UPLOAD_DIR`` for this suite — a scratch dir the
host-side assertions inspect (the app server runs on the same
machine)."""
return tmp_path_factory.mktemp("bor_uploads") / "uploads"
@pytest.fixture(scope="module")
def upload_archive(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The 25-file upload archive (``e2e-prog.tar.gz``)."""
root = tmp_path_factory.mktemp("bor_archive")
return _build_targz(root / UPLOAD_ARCHIVE, UPLOAD_FILES)
@pytest.fixture(scope="module")
def sync_local_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The host temp dir the sync test registers as a ``kind=local``
source — ``sync-corpus/notes/NN.md`` (25 files)."""
root = tmp_path_factory.mktemp("bor_sync_local") / SYNC_SOURCE_DIR
for rel, content in SYNC_FILES.items():
path = root / rel
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8")
return root
@pytest.fixture(scope="module")
def app_server(
mock_llm: int,
slow_llm: int,
upload_dir: Path,
tmp_path_factory: pytest.TempPathFactory,
) -> Iterator[str]:
"""The real app under test — per-module env: the LLM base URL is the
SLOW PROXY in front of the mock (the timing fixture), uploads unpack
into a scratch dir, and the env git list is forced empty (the sync
sources are this suite's own ``kind=local`` row, seeded per test)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"{SLOW_URL}/v1"
)
# Mock-calibrated threshold (conftest pattern) — no chat turn is
# ever sent in this suite, but the app boots with the same env shape.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
env["BOR_GIT_SOURCES"] = ""
env["BOR_UPLOAD_DIR"] = str(upload_dir)
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB per test (the E2E isolation pattern): the
upload's/sync's counts and every ``/api/docs`` assertion must be
this test's own doing. The E2E suites share one Postgres, and a
leftover git_sources row or document would corrupt the row-count
and doc-list assertions (and a leftover row would be a SECOND sync
source, skewing the per-file counts)."""
with SessionLocal() as db:
db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview, git_sources"))
db.commit()
def _wait_no_running_jobs(app_url: str) -> None:
"""No background job may leak across tests (the run states live in
the app's memory, and a still-running scan would keep importing
into the NEXT test's truncated KB): wait for both status endpoints
to be non-running BEFORE the truncate. Own admin session (the
endpoints are admin-only) — usually a no-op: the tests settle only
after their run's terminal state."""
with httpx.Client(timeout=5.0) as client:
client.post(f"{app_url}/api/login", json={"password": ADMIN_PASSWORD})
for path in ("/api/sync/status", "/api/git-sources/upload/status"):
body: dict[str, Any] = {}
deadline = time.monotonic() + 90
while time.monotonic() < deadline:
r = client.get(f"{app_url}{path}")
if r.status_code == 200:
body = r.json()
if body["state"] != "running":
break
time.sleep(0.2)
assert body["state"] != "running", (
f"a background job was still running at test boundary: {path} {body}"
)
@pytest.fixture(autouse=True)
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
_wait_no_running_jobs(app_url)
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _seed_local_source(path: str) -> None:
"""Register a ``kind=local`` source row directly (deterministic —
the phase-49 page form is upload-only; a plain directory is an
API/DB-only operation, the test_local_directory_sources.py note)."""
with SessionLocal() as db:
db.add(GitSource(url=path, kind="local", path=path))
db.commit()
def _admin_git_sources_page(page: Page, app_url: str) -> None:
"""Real form login landing on the git sources page (admin settled:
Sign out visible, the manager revealed by the page module)."""
login(page, app_url, next=GIT_SOURCES_URL)
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
expect(page.locator("#git-sources-gate")).to_be_hidden()
expect(page.locator("#git-sources-content")).to_be_visible()
def _status(page: Page, app_url: str, path: str) -> dict[str, Any]:
"""One (cookie-authenticated) status-endpoint fetch."""
r = page.request.get(f"{app_url}{path}")
assert r.status == 200, r.text
return r.json()
def _wait_running_started_at(
page: Page, app_url: str, path: str, timeout_s: float = 15.0
) -> str:
"""Poll the status endpoint until the run is ``running``; return its
``started_at`` (the run's identity — a second run would reset it)."""
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
body = _status(page, app_url, path)
if body["state"] == "running":
assert body["started_at"] is not None
return str(body["started_at"])
if body["state"] in ("success", "failed"):
raise AssertionError(f"the run settled too fast to observe: {body}")
time.sleep(0.1)
raise AssertionError(f"the run never entered running: {body}")
class _TickRecorder:
"""The deterministic layer, concurrent with the UI assertions.
A daemon thread that tight-polls (~100 ms cadence) the status
endpoint with its OWN admin session (``httpx`` — the browser page
drives itself in the meantime; the Playwright sync API is not
thread-safe, so the thread never touches it), recording every tick
from the first poll: the idle prelude, the running ticks (file-less
unpack/row/probe phase, then the per-file ticks), and the terminal
body. The thread only READS the same endpoint the UI's 2 s poll
reads — it starts no jobs and cannot skew the run."""
def __init__(self, app_url: str, path: str) -> None:
self._url = f"{app_url}{path}"
self._login_url = f"{app_url}/api/login"
self._ticks: list[dict[str, Any]] = []
self._terminal: dict[str, Any] | None = None
self._stop = threading.Event()
self._thread: threading.Thread | None = None
def start(self) -> None:
def run() -> None:
with httpx.Client(timeout=5.0) as client:
# Its own admin session — the recorder's reads must not
# depend on (or disturb) the browser context's cookie.
client.post(self._login_url, json={"password": ADMIN_PASSWORD})
saw_running = False
while not self._stop.is_set():
try:
r = client.get(self._url)
if r.status_code == 200:
body = r.json()
self._ticks.append(body)
# The run status is in the app's memory and
# SURVIVES across this module's tests: a
# residual terminal state (a previous test's
# run) must not be mistaken for this test's
# own terminal — accept it only AFTER this
# run's "running" has been observed.
if body["state"] == "running":
saw_running = True
elif body["state"] in ("success", "failed") and saw_running:
self._terminal = body
return
except Exception: # noqa: BLE001 — blip: retry next tick
pass
self._stop.wait(0.1)
self._thread = threading.Thread(target=run, daemon=True)
self._thread.start()
def stop(self, timeout_s: float = 60.0) -> dict[str, Any]:
"""Join until a terminal tick is recorded (or fail the test)."""
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
if self._terminal is not None:
break
time.sleep(0.05)
self._stop.set()
if self._thread is not None:
self._thread.join(timeout=5)
assert self._terminal is not None, (
"the recorder saw no terminal state "
f"(last ticks: {self._ticks[-3:] if self._ticks else 'none'})"
)
return self._terminal
@property
def running_ticks(self) -> list[dict[str, Any]]:
return [t for t in self._ticks if t["state"] == "running"]
def _assert_live_file_ticks(
ticks: list[dict[str, Any]], source: str, n_files: int
) -> None:
"""A4 against the recorded running ticks (the deterministic layer):
* the file-less ticks (unpack/row/model probe — before any file is
indexed) come FIRST (the label is the bare prefix then);
* SOME tick reports a non-null ``current_file`` in the
``source/relative/path`` shape;
* ``files_total`` is the full pre-walk count from the first file
tick, and ``files_done`` advances monotonically to it.
"""
with_file = [t for t in ticks if t["current_file"]]
assert with_file, f"no running tick carried a current_file: {ticks[:6]}"
first_with = with_file[0]
assert first_with["current_file"].startswith(f"{source}/"), first_with
assert re.fullmatch(
rf"{re.escape(source)}/.+\.(md|markdown)", first_with["current_file"]
), first_with
assert first_with["files_total"] == n_files, first_with
assert 1 <= first_with["files_done"] <= n_files, first_with
dones = [t["files_done"] for t in with_file]
assert dones == sorted(dones), f"files_done not monotonic: {dones}"
# The last file tick is (n-1, n): the hook fires before the final
# file's index, so the final count itself can land in the terminal
# body instead of a recorded tick (100 ms cadence vs ≈160 ms file).
assert max(dones) >= n_files - 1, f"files_done never advanced: {dones}"
pre = [t for t in ticks if t["current_file"] is None]
assert pre, f"no file-less running tick (the unpack phase): {ticks[:6]}"
assert ticks.index(pre[0]) < ticks.index(first_with), (
"a file tick preceded the file-less unpack ticks"
)
# ---------------------------------------------------------------------------
# 1. Toast on 202 → navigate away → sync button animates with the
# upload's current file → settle + catalog refresh (A2 + A3)
# ---------------------------------------------------------------------------
def test_upload_toast_then_navigate_away(
page: Page, app_url: str, db_ready: None, upload_archive: Path
) -> None:
"""On ``/git-sources.html``: pick the multi-file archive, submit →
the "Successfully uploaded — <source>" toast appears WHILE the scan
is still running; immediately navigate to ``/sources.html`` → the
sync button is present, animating (icon ``is-spinning``,
``aria-busy``) with the ``Importing`` label; wait for the settle →
button idle ("Sync sources"), no error UI, and the catalog table
shows the uploaded documents (the phase-63 listing, untouched)."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
# (A previous test's terminal run may re-render its result line at
# boot — the task-05 re-attach contract; the submit below clears
# it, and that is the "clean start" asserted after the click.)
page.set_input_files("#archive-upload-file", str(upload_archive))
page.click("#archive-upload-btn")
# The 202 moment (A2): a single .toast node, visible, role=status,
# naming the safe source name…
toast = page.locator(".toast")
expect(toast).to_have_count(1, timeout=SETTLE_TIMEOUT_MS)
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
assert toast.get_attribute("role") == "status"
expect(toast).to_have_text(f"Successfully uploaded — {UPLOAD_NAME}")
# …and the scan is still running — the result line is not up yet
# (the toast precedes the scan's completion, A2).
expect(page.locator("#archive-upload-result")).to_be_hidden()
started_at = _wait_running_started_at(page, app_url, "/api/git-sources/upload/status")
assert started_at is not None
# Navigate away immediately (A1: the scan no longer dies with the
# page).
page.goto(app_url + SOURCES_URL)
# The RAG-page sync button re-attaches to the in-flight upload scan
# (A3): present, animating (spinner + aria-busy), disabled, with
# the live "Importing <file>" label — no error UI on this page
# (the upload's failure UI lives on the Sources page, A3).
btn = page.locator("#sync-btn")
expect(btn).to_be_visible(timeout=30_000)
expect(btn).to_be_disabled()
expect(btn).to_have_attribute("aria-busy", "true")
expect(btn.locator(".sync-icon")).to_have_class(re.compile(r"\bis-spinning\b"))
expect(page.locator("#sync-label")).to_have_text(
re.compile(rf"Importing {re.escape(UPLOAD_NAME)}/"), timeout=SETTLE_TIMEOUT_MS
)
expect(page.locator("#sync-error-banner")).to_be_hidden()
# Settle: the button returns to idle, the sync-result line never
# rendered the upload's counts (A3), and the catalog refreshes with
# the uploaded documents — the phase-63 listing, untouched.
expect(page.locator("#sync-label")).to_have_text("Sync sources", timeout=SETTLE_TIMEOUT_MS)
expect(btn).to_be_enabled()
expect(btn).not_to_have_attribute("aria-busy")
expect(btn.locator(".sync-icon")).not_to_have_class(re.compile(r"\bis-spinning\b"))
expect(page.locator("#sync-result")).to_have_text("")
expect(page.locator("#docs-tbody tr")).to_have_count(N_FILES, timeout=30_000)
expect(page.locator("#docs-tbody tr", has_text="docs/00.md")).to_have_count(1)
expect(page.locator("#docs-tbody tr", has_text="docs/24.md")).to_have_count(1)
expect(page.locator("#docs-tbody tr", has_text=UPLOAD_NAME)).to_have_count(N_FILES)
# ---------------------------------------------------------------------------
# 2. Upload progress: live current file at BOTH layers; the toast fired
# earlier than the result (A4)
# ---------------------------------------------------------------------------
def test_upload_progress_shows_current_file(
page: Page, app_url: str, db_ready: None, upload_archive: Path
) -> None:
"""During the scan: the status endpoint reports a non-null
``current_file`` (``source/relative/path`` shape) at some running
tick; the upload button label shows "Processing…" with a file path
(UI layer) BEFORE the result line lands; the toast fired earlier in
the run (not after the result). The result line + the settle come
from the status ``success``."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
recorder = _TickRecorder(app_url, "/api/git-sources/upload/status")
recorder.start()
page.set_input_files("#archive-upload-file", str(upload_archive))
page.click("#archive-upload-btn")
# A new attempt starts clean (the submit handler hides the result
# line — any previous run's re-rendered line is gone by now).
expect(page.locator("#archive-upload-result")).to_be_hidden()
# The toast fires at the 202 — earlier in the run, NOT after the
# result (the result line is still down when the toast is up).
toast = page.locator(".toast")
expect(toast).to_have_count(1, timeout=SETTLE_TIMEOUT_MS)
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
expect(toast).to_have_text(f"Successfully uploaded — {UPLOAD_NAME}")
expect(page.locator("#archive-upload-result")).to_be_hidden()
# UI layer: the button hands over to the scan — bare "Processing…"
# at the 202 (A4: no file yet during the unpack phase)…
btn = page.locator("#archive-upload-btn")
expect(btn).to_be_disabled()
expect(btn).to_have_text("Processing…", timeout=5_000)
# …then the live file label ("Processing… <file> (n/m)") at the
# page's next 2 s poll tick, still before the result line lands.
expect(btn).to_have_text(
re.compile(rf"Processing… {re.escape(UPLOAD_NAME)}/.+\.md \(\d+/{N_FILES}\)"),
timeout=SETTLE_TIMEOUT_MS,
)
expect(page.locator("#archive-upload-result")).to_be_hidden()
# Deterministic layer: the recorder's full tick series (the same
# endpoint the UI's 2 s poll reads) — file-less unpack ticks first,
# then the per-file ticks with the full denominator.
terminal = recorder.stop()
assert terminal["state"] == "success", terminal
_assert_live_file_ticks(recorder.running_ticks, UPLOAD_NAME, N_FILES)
# Phase 64: current_file is null in terminal states (the final
# counts survive).
assert terminal["current_file"] is None
assert terminal["files_done"] == N_FILES
assert terminal["files_total"] == N_FILES
# The result line is rendered from the status success (the
# UploadOut-shaped detail, counts unchanged in shape).
assert terminal["detail"]["files"] == N_FILES
assert terminal["detail"]["added"] == N_FILES
assert terminal["detail"]["source"] == UPLOAD_NAME
# Settle: the result line lands, the button restores, the input
# cleared, and the list has exactly one row for the archive.
result = page.locator("#archive-upload-result")
expect(result).to_have_text(f"{N_FILES} added", timeout=30_000)
expect(btn).to_be_enabled()
expect(btn).to_have_text("Upload & scan")
expect(page.locator("#archive-upload-file")).to_have_value("")
expect(page.locator("#git-sources-tbody tr", has_text=UPLOAD_NAME)).to_have_count(1)
# ---------------------------------------------------------------------------
# 3. Sync live file label: both layers, then the preserved pre-64
# success settle (A4)
# ---------------------------------------------------------------------------
def test_sync_live_file_label(
page: Page, app_url: str, db_ready: None, sync_local_dir: Path
) -> None:
"""A multi-file local source (the ``test_sync_button.py`` fixture
style): on ``/sources.html`` click **Sync sources** → the label
shows "Syncing…" (bare, the pre-64 click state) then "Syncing…
<file> (n/m)" while running (endpoint layer: ``current_file``
non-null at running ticks; UI layer: the label poll), then the
success settle with the counts result line — the pre-phase-64 sync
UX is preserved, plus the file."""
page.set_default_timeout(30_000)
_seed_local_source(str(sync_local_dir))
login(page, app_url) # lands on /sources.html (the button's home)
btn = page.locator("#sync-btn")
expect(btn).to_be_visible(timeout=30_000)
expect(page.locator("#sync-label")).to_have_text("Sync sources")
recorder = _TickRecorder(app_url, "/api/sync/status")
recorder.start()
btn.click()
# The click's immediate state is the pre-64 one (A4 — the bare
# prefix until the import's first file): disabled, aria-busy,
# spinning icon, no error…
expect(btn).to_be_disabled()
expect(btn).to_have_attribute("aria-busy", "true")
expect(btn.locator(".sync-icon")).to_have_class(re.compile(r"\bis-spinning\b"))
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-error-banner")).to_be_hidden()
# UI layer: the label gains the live file at the page's 2 s poll
# tick ("Syncing… <source/relative/path> (n/m)").
expect(page.locator("#sync-label")).to_have_text(
re.compile(rf"Syncing… {re.escape(SYNC_SOURCE_DIR)}/.+\.md \(\d+/{N_FILES}\)"),
timeout=SETTLE_TIMEOUT_MS,
)
# Deterministic layer: the recorder's full tick series — file-less
# model-check ticks first, then the per-file ticks (full
# denominator, advancing counts).
terminal = recorder.stop()
assert terminal["state"] == "success", terminal
_assert_live_file_ticks(recorder.running_ticks, SYNC_SOURCE_DIR, N_FILES)
assert terminal["current_file"] is None
assert terminal["files_done"] == N_FILES
assert terminal["files_total"] == N_FILES
assert terminal["detail"]["added"] == N_FILES
# The pre-64 success settle, preserved: "Synced HH:MM" + the counts
# result line, button re-enabled, no error — and the catalog lists
# the imported docs.
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
expect(btn).to_be_enabled()
expect(btn).not_to_have_attribute("aria-busy")
expect(page.locator("#sync-result")).to_have_text(f"{N_FILES} added")
expect(page.locator("#docs-tbody tr")).to_have_count(N_FILES, timeout=30_000)
expect(page.locator("#docs-tbody tr", has_text="notes/00.md")).to_have_count(1)
expect(page.locator("#docs-tbody tr", has_text=SYNC_SOURCE_DIR)).to_have_count(N_FILES)
# ---------------------------------------------------------------------------
# 4. Reload mid-scan → re-attach: no error, no second upload (A1 + A2)
# ---------------------------------------------------------------------------
def test_upload_reattach_after_reload(
page: Page, app_url: str, db_ready: None, upload_archive: Path
) -> None:
"""Start the upload and, DURING the scan, reload
``/git-sources.html`` → the button is in the Processing state
(disabled) with no error banner and NO second upload (the status
endpoint's single run is still the one from before the reload —
pinned on its ``started_at``); it then settles with the result line
and the list shows exactly one row for the archive (in-place
identity preserved)."""
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
page.set_input_files("#archive-upload-file", str(upload_archive))
page.click("#archive-upload-btn")
# The 202 toast (the run is in flight)…
toast = page.locator(".toast")
expect(toast).to_have_count(1, timeout=SETTLE_TIMEOUT_MS)
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
# …and the run's identity: the status's started_at (a second run
# would reset it — the single-run claim is pinned on it).
started_at = _wait_running_started_at(page, app_url, "/api/git-sources/upload/status")
# Reload mid-scan — the page must re-attach, not dead-end.
page.reload()
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
expect(page.locator("#git-sources-content")).to_be_visible()
# The boot re-attach (task 05): the button is in the Processing
# state (disabled, "Processing…") with no error banner and no
# result line yet…
btn = page.locator("#archive-upload-btn")
expect(btn).to_be_disabled(timeout=15_000)
expect(btn).to_have_text(re.compile(r"Processing…"), timeout=15_000)
expect(page.locator("#archive-upload-error")).to_be_hidden()
expect(page.locator("#archive-upload-result")).to_be_hidden()
# …and it settles with the result line from the status success…
expect(page.locator("#archive-upload-result")).to_have_text(
f"{N_FILES} added", timeout=SETTLE_TIMEOUT_MS
)
expect(btn).to_be_enabled()
expect(btn).to_have_text("Upload & scan")
# …and the list shows exactly one row for the archive.
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
row = page.locator("#git-sources-tbody tr", has_text=UPLOAD_NAME)
expect(row).to_have_count(1)
expect(row.locator("span.git-source-kind")).to_have_text("Local")
# No second upload: the terminal status is the SAME run — its
# started_at is the one from before the reload.
terminal = _status(page, app_url, "/api/git-sources/upload/status")
assert terminal["state"] == "success", terminal
assert str(terminal["started_at"]) == started_at, (
f"the run's identity changed (a second upload ran): {terminal['started_at']}"
)
assert terminal["current_file"] is None
File diff suppressed because it is too large Load Diff
+15 -1
View File
@@ -63,7 +63,7 @@ from __future__ import annotations
import asyncio import asyncio
import logging import logging
import time import time
from collections.abc import Iterator from collections.abc import Callable, Iterator
from datetime import UTC, datetime from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
@@ -235,6 +235,9 @@ class FakeImportSources:
self.sources: list[list[Path]] = [] self.sources: list[list[Path]] = []
self.llms: list[LLMClient] = [] self.llms: list[LLMClient] = []
self.prune_flags: list[bool] = [] self.prune_flags: list[bool] = []
# Phase 64 (task 02): the progress hook the runner passes (a live
# closure while wired, None if the wiring regresses).
self.progress_hooks: list[object] = []
async def __call__( async def __call__(
self, self,
@@ -244,10 +247,12 @@ class FakeImportSources:
prune: bool = False, prune: bool = False,
limit: int | None = None, limit: int | None = None,
session: Session | None = None, session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
) -> ImportSummary: ) -> ImportSummary:
self.sources.append(list(sources)) self.sources.append(list(sources))
self.llms.append(llm) self.llms.append(llm)
self.prune_flags.append(prune) self.prune_flags.append(prune)
self.progress_hooks.append(progress)
if self.delay: if self.delay:
await asyncio.sleep(self.delay) await asyncio.sleep(self.delay)
return self.summary return self.summary
@@ -324,6 +329,10 @@ def test_admin_sync_success_reports_full_detail(
"finished_at": None, "finished_at": None,
"detail": {}, "detail": {},
"error": None, "error": None,
# Phase 64 (task 02): the per-file progress keys — null/0/0 idle.
"current_file": None,
"files_done": 0,
"files_total": 0,
} }
r = sync_client.post("/api/sync") r = sync_client.post("/api/sync")
@@ -353,6 +362,10 @@ def test_admin_sync_success_reports_full_detail(
assert fake_import.prune_flags == [True] assert fake_import.prune_flags == [True]
assert len(fake_import.llms) == 1 assert len(fake_import.llms) == 1
assert isinstance(fake_import.llms[0], LLMClient) assert isinstance(fake_import.llms[0], LLMClient)
# Phase 64 (task 02): the runner wires the per-file progress hook
# (the live closure the status endpoint reads while the import runs).
assert len(fake_import.progress_hooks) == 1
assert callable(fake_import.progress_hooks[0])
# Overview: refreshed (added + updated > 0) with the same client. # Overview: refreshed (added + updated > 0) with the same client.
assert fake_overview.llms == [fake_import.llms[0]] assert fake_overview.llms == [fake_import.llms[0]]
# Phase 41: the probe ran first and got the very client the import # Phase 41: the probe ran first and got the very client the import
@@ -733,6 +746,7 @@ def test_import_error_is_reported_with_credentials_masked(
prune: bool = False, prune: bool = False,
limit: int | None = None, limit: int | None = None,
session: Session | None = None, session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
) -> ImportSummary: ) -> ImportSummary:
raise EmbeddingError( raise EmbeddingError(
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 " "embeddings request to https://user:secret@aipi.reeseapps.com/v1 "

Some files were not shown because too many files have changed in this diff Show More