feat(sources): real-time file progress for sync and upload — background upload with success toast
This commit is contained in:
@@ -0,0 +1,54 @@
|
|||||||
|
# Phase 63 — Unambiguous Document Listing for Agent Tools
|
||||||
|
|
||||||
|
**Source:** Live troubleshooting report (2026-09-01) — the `brain-of-reese` agent at `https://brain.experimental.reeseapps.com/` cannot read documents because the `list_documents` tool output uses an ambiguous `source/path — title` format that the LLM cannot reliably parse when paths contain `/` characters.
|
||||||
|
|
||||||
|
**Story:** `.agent/user_stories/agent-document-tools.md` (phase 37, extended)
|
||||||
|
|
||||||
|
**Context:** The agent's grounded-turn loop uses two tools: `list_documents` (returns the catalog) and `read_document` (reads one document). The catalog output is formatted as `source/path — title` (one line per document), but when a file lives inside a subdirectory of its source, the output looks like `brain-of-reese-main/homelab/aws-route53.md — aws-route53`. The LLM cannot reliably determine where `source` ends and `path` begins because both contain `/` separators. The thinking trace shows the LLM cycling through multiple failed attempts, each time guessing a different split.
|
||||||
|
|
||||||
|
The fix changes the catalog format to unambiguous `source: X | path: Y | title: Z` lines and updates the tool descriptions to tell the LLM how to parse them.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
The agent can always extract the correct `(source, path)` pair from the `list_documents` output, regardless of how many `/` characters the path contains.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
|
||||||
|
- `62_ui_customization` (todo, preceding — functional: no code changes in this phase touch UI or config)
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
|
||||||
|
1. `01_agent_list_format.md` — change `list_catalog` output from `source/path — title` to `source: X | path: Y | title: Z`; update the `read_document` tool descriptions to reference the new format.
|
||||||
|
2. `02_mock_and_unit_tests.md` — update `tests/e2e/mock_llm.py::_catalog_docs` to parse the new format; update `tests/unit/test_agent.py` (the "No document at" refusal line is unchanged — it names the lookup failure, not the list format).
|
||||||
|
3. `03_e2e_document_tools.md` — validate that `tests/e2e/test_agent_document_tools.py` and `tests/e2e/test_agent_unlimited_tools.py` still pass (they assert on SSE frames and UI rendering, not on the catalog text format — the tool call `argument` field `source/path` is unchanged).
|
||||||
|
4. `04_docs.md` — update any inline documentation that references the old format.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
|
||||||
|
- Unit: `tests/unit/test_agent.py` — the "No document at" refusal message is unchanged (it reports the lookup failure, not the list format). The `_catalog_docs` mock parsing must produce identical `(source, path)` tuples for all existing test catalogs.
|
||||||
|
- Integration: `tests/integration/test_agent_tools.py` — `list_catalog` ordering assertions are unchanged (the function returns `[(source, path, title), ...]` tuples; only the string formatting in `_execute_tool` changes).
|
||||||
|
- E2E (mandatory, house rule): `tests/e2e/test_agent_document_tools.py` and `tests/e2e/test_agent_unlimited_tools.py` run in isolation — the SSE `argument` field (`source/path`) is unchanged; the mock's `_catalog_docs` parsing produces identical results; the tool lines and source chips render identically.
|
||||||
|
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
- [ ] `list_catalog` output uses `source: X | path: Y | title: Z` format — verifiable by a unit test that patches `list_catalog` and checks the formatted string.
|
||||||
|
- [ ] The `read_document` tool descriptions tell the LLM to extract `source` and `path` from the `source:` / `path:` labels.
|
||||||
|
- [ ] `tests/e2e/mock_llm.py::_catalog_docs` parses the new format and produces identical `(source, path)` tuples for all existing test catalogs.
|
||||||
|
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] `uv run pytest tests/e2e/test_agent_document_tools.py tests/e2e/test_agent_unlimited_tools.py -v --no-cov` green in isolation (DB up).
|
||||||
|
- [ ] Regression E2E suites green in isolation: `test_chat_api.py`, `test_sse_events.py`, `test_whole_document_context.py`.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agent/phases/complete/` (`.agent/` stays untracked).
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
|
||||||
|
- **Owner-locked (2026-09-01, troubleshooting report, A1):** the new format uses `source: X | path: Y | title: Z` — one line per document, pipe-delimited fields, no ambiguity even with deeply nested paths.
|
||||||
|
- **Owner-locked (2026-09-01, troubleshooting report, A2):** the `read_document` tool descriptions are updated to reference the new format (tell the LLM how to parse `source:` and `path:` labels).
|
||||||
|
- **Owner-locked (2026-09-01, troubleshooting report, A3):** the SSE `argument` field (`source/path`) is unchanged — it is the tool call argument, not the list output. The frontend tool-line rendering is unchanged.
|
||||||
|
- **Owner-locked (2026-09-01, troubleshooting report, A4):** the "No document at …" refusal message is unchanged — it reports the lookup failure, not the list format.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add app/ tests/ && git commit --no-gpg-sign -m "fix(agent): unambiguous document listing format for LLM parsing"
|
||||||
|
```
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
# Task 01 — Change `list_catalog` output format + update tool descriptions
|
||||||
|
|
||||||
|
**Goal:** Make the `list_documents` catalog output unambiguous for LLM parsing, and update the `read_document` tool descriptions to tell the LLM how to extract `source` and `path`.
|
||||||
|
|
||||||
|
## Changes
|
||||||
|
|
||||||
|
### `app/rag/agent.py`
|
||||||
|
|
||||||
|
#### 1. Change `list_catalog` output format
|
||||||
|
|
||||||
|
In `_execute_tool`, the `list_documents` handler formats the catalog as:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BEFORE (ambiguous when path contains /)
|
||||||
|
listing = f"{len(rows)} documents:\n" + "\n".join(
|
||||||
|
f"{source}/{path} — {title}" for source, path, title in rows
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Change to:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# AFTER (unambiguous — each field is labeled)
|
||||||
|
listing = f"{len(rows)} documents:\n" + "\n".join(
|
||||||
|
f"source: {source} | path: {path} | title: {title}"
|
||||||
|
for source, path, title in rows
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 2. Update `read_document` tool descriptions
|
||||||
|
|
||||||
|
In `AGENT_TOOLS`, update the `source` and `path` parameter descriptions to reference the new format:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BEFORE
|
||||||
|
"source": {
|
||||||
|
"description": (
|
||||||
|
"The document's source (a directory basename, e.g. 'Homelab')."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"path": {
|
||||||
|
"description": (
|
||||||
|
"The document's path relative to its source directory."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
|
||||||
|
# AFTER
|
||||||
|
"source": {
|
||||||
|
"description": (
|
||||||
|
"The document's source, as shown after 'source: ' in the "
|
||||||
|
"list_documents output (e.g. 'Homelab' from "
|
||||||
|
"'source: Homelab | path: homelab/aws-route53.md')."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
"path": {
|
||||||
|
"description": (
|
||||||
|
"The document's path, as shown after 'path: ' in the "
|
||||||
|
"list_documents output (e.g. 'homelab/aws-route53.md' from "
|
||||||
|
"'source: Homelab | path: homelab/aws-route53.md')."
|
||||||
|
),
|
||||||
|
},
|
||||||
|
```
|
||||||
|
|
||||||
|
### `app/rag/prompts.py`
|
||||||
|
|
||||||
|
No changes needed — the prompt text references the tool names but not the output format.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `app/rag/agent.py` — `list_catalog` formatting + `AGENT_TOOLS` descriptions
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- `uv run pytest tests/unit/test_agent.py -v --no-cov` — the tool shape tests (`test_agent_tools_names_and_parameters`) check the description strings, so they must be updated to match.
|
||||||
|
- `uv run ruff check app/rag/agent.py && uv run pyright app/rag/agent.py` — lint + types clean.
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
# Task 02 — Update mock parsing + unit tests for the new format
|
||||||
|
|
||||||
|
**Goal:** Update `tests/e2e/mock_llm.py::_catalog_docs` to parse the new `source: X | path: Y | title: Z` format, and update `tests/unit/test_agent.py` to match the new tool descriptions.
|
||||||
|
|
||||||
|
## Changes
|
||||||
|
|
||||||
|
### `tests/e2e/mock_llm.py` — `_catalog_docs` function
|
||||||
|
|
||||||
|
The current parsing uses `head.rpartition("/")` on the `source/path` head:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BEFORE
|
||||||
|
def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
|
||||||
|
docs: list[tuple[str, str]] = []
|
||||||
|
for m in _messages(body):
|
||||||
|
if m.get("role") != "tool":
|
||||||
|
continue
|
||||||
|
content = str(m.get("content") or "")
|
||||||
|
if content.startswith(_READ_RESULT_PREFIX):
|
||||||
|
continue
|
||||||
|
for line in content.splitlines():
|
||||||
|
head = line.split(" — ", 1)[0].strip()
|
||||||
|
if "/" in head:
|
||||||
|
source, _, path = head.rpartition("/")
|
||||||
|
if source and path:
|
||||||
|
docs.append((source, path))
|
||||||
|
return docs
|
||||||
|
```
|
||||||
|
|
||||||
|
Change to parse the new labeled format:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# AFTER
|
||||||
|
def _catalog_docs(body: dict[str, Any]) -> list[tuple[str, str]]:
|
||||||
|
docs: list[tuple[str, str]] = []
|
||||||
|
for m in _messages(body):
|
||||||
|
if m.get("role") != "tool":
|
||||||
|
continue
|
||||||
|
content = str(m.get("content") or "")
|
||||||
|
if content.startswith(_READ_RESULT_PREFIX):
|
||||||
|
continue
|
||||||
|
for line in content.splitlines():
|
||||||
|
# New format: "source: X | path: Y | title: Z"
|
||||||
|
if not line.startswith("source: "):
|
||||||
|
continue
|
||||||
|
parts = line.split(" | ", 2)
|
||||||
|
if len(parts) < 3:
|
||||||
|
continue
|
||||||
|
source = parts[0].removeprefix("source: ").strip()
|
||||||
|
path_part = parts[1]
|
||||||
|
if not path_part.startswith("path: "):
|
||||||
|
continue
|
||||||
|
path = path_part.removeprefix("path: ").strip()
|
||||||
|
if source and path:
|
||||||
|
docs.append((source, path))
|
||||||
|
return docs
|
||||||
|
```
|
||||||
|
|
||||||
|
### `tests/unit/test_agent.py` — tool description assertions
|
||||||
|
|
||||||
|
The test `test_agent_tools_names_and_parameters` asserts the exact description strings:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# BEFORE (line ~109)
|
||||||
|
assert by_name["read_document"]["function"]["description"] == (
|
||||||
|
"Add the full content of one more indexed document to your context"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
This assertion is about the function description (not parameter descriptions), so it stays the same. But the test file may have inline assertions about the `source` and `path` parameter descriptions — check and update if any exist.
|
||||||
|
|
||||||
|
### `tests/unit/test_agent.py` — refusal message
|
||||||
|
|
||||||
|
The refusal message `"No document at S/ghost.md — check the list_documents output."` is unchanged — it reports the lookup failure, not the list format. No changes needed here.
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `tests/e2e/mock_llm.py` — `_catalog_docs` parsing
|
||||||
|
- `tests/unit/test_agent.py` — only if inline assertions reference the old format (verify)
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- `uv run pytest tests/unit/test_agent.py -v --no-cov` — all agent unit tests pass.
|
||||||
|
- `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov` — the mock's `_catalog_docs` must produce identical `(source, path)` tuples for the test catalogs.
|
||||||
|
- `uv run ruff check tests/ && uv run pyright tests/` — lint + types clean.
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Task 03 — E2E document tools validation
|
||||||
|
|
||||||
|
**Goal:** Run the E2E suites for agent document tools and verify they still pass with the new catalog format.
|
||||||
|
|
||||||
|
## What to verify
|
||||||
|
|
||||||
|
The E2E tests assert on:
|
||||||
|
1. **SSE frames** — the `argument` field of `tool` events is `source/path` (e.g., `"Deployments/example-record-file.json"`). This is the tool call argument, NOT the list output format. **Unchanged.**
|
||||||
|
2. **UI rendering** — tool lines show "Listing documents" and "Reading <source/path>". **Unchanged.**
|
||||||
|
3. **Source chips** — display `${s.source}/${s.path}`. **Unchanged.**
|
||||||
|
4. **Answer content** — the mock's deterministic answer quoting the read document. **Unchanged** (the mock reads the same `(source, path)` tuples from the catalog).
|
||||||
|
5. **Query log** — sources field stores `"source/path, source/path"`. **Unchanged.**
|
||||||
|
|
||||||
|
The only thing that changes is the **text of the catalog** that the mock parses. The mock's `_catalog_docs` function must produce identical `(source, path)` tuples for the test catalogs, which means the tool flow classification (`_tool_flow`) and the read step will work identically.
|
||||||
|
|
||||||
|
## Run commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# DB must be up: podman compose up -d db
|
||||||
|
uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov
|
||||||
|
uv run pytest tests/e2e/test_agent_unlimited_tools.py -v --no-cov
|
||||||
|
```
|
||||||
|
|
||||||
|
## Regression suites to run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run pytest tests/e2e/test_chat_api.py -v --no-cov
|
||||||
|
uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov
|
||||||
|
uv run pytest tests/e2e/test_sse_events.py -v --no-cov
|
||||||
|
```
|
||||||
|
|
||||||
|
## Expected outcome
|
||||||
|
|
||||||
|
All tests pass without modification (aside from the mock parsing change in task 02). The catalog text format is an internal detail of the agent loop — the tool call arguments, SSE frames, UI rendering, and answer content are all unchanged.
|
||||||
|
|
||||||
|
## If tests fail
|
||||||
|
|
||||||
|
1. Check that `_catalog_docs` produces the same `(source, path)` tuples as before for the test catalogs.
|
||||||
|
2. Verify the mock's `_tool_flow` classification still works (it calls `_catalog_docs`).
|
||||||
|
3. Check that the SSE `argument` field is still `source/path` (it should be — `_execute_tool` builds it from `call.arguments["source"] + "/" + call.arguments["path"]`, which is unchanged).
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- None (verification only — all changes were in tasks 01 and 02)
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Task 04 — Update documentation
|
||||||
|
|
||||||
|
**Goal:** Update all documentation that references the old `source/path — title` catalog format to the new `source: X | path: Y | title: Z` format.
|
||||||
|
|
||||||
|
## Files to update
|
||||||
|
|
||||||
|
### `README.md` — Agent document tools section (~L168)
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# BEFORE
|
||||||
|
* **`list_documents`** — lists every indexed document, one
|
||||||
|
`source/path — title` line each (the same order as the Sources page);
|
||||||
|
|
||||||
|
# AFTER
|
||||||
|
* **`list_documents`** — lists every indexed document, one
|
||||||
|
`source: X | path: Y | title: Z` line each (the same order as the
|
||||||
|
Sources page);
|
||||||
|
```
|
||||||
|
|
||||||
|
### `.agent/user_stories/agent-document-tools.md` — Acceptance criteria (~L38)
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
# BEFORE
|
||||||
|
3. `app/rag/agent.py`: the loop — budgets from
|
||||||
|
`BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` (default 1/1); the
|
||||||
|
`list_documents` tool returns the DB catalog
|
||||||
|
(`source/path — title` lines, `/api/docs` order);
|
||||||
|
|
||||||
|
# AFTER
|
||||||
|
3. `app/rag/agent.py`: the loop — budgets from
|
||||||
|
`BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` (default 1/1); the
|
||||||
|
`list_documents` tool returns the DB catalog
|
||||||
|
(`source: X | path: Y | title: Z` lines, `/api/docs` order);
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- `grep -n "source/path — title\|source/path - title" README.md .agent/user_stories/*.md` should return no results.
|
||||||
|
- `uv run ruff check README.md .agent/` — lint clean (markdown files are not linted by ruff, but the command should not error).
|
||||||
|
|
||||||
|
## Files changed
|
||||||
|
|
||||||
|
- `README.md` — Agent document tools section
|
||||||
|
- `.agent/user_stories/agent-document-tools.md` — Acceptance criteria item 3
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
All verification passes green. Phase 61 is fully complete — final verification pass only, no code changes needed.
|
||||||
|
|
||||||
|
**Phase 61 (stale_ui_copy) — final verification report**
|
||||||
|
|
||||||
|
Verified (all 4 task files already in `complete/`; no work remaining, no defects found):
|
||||||
|
- Chat page: meta description, empty-state sub, composer placeholder, and the 4 default chips in `app/config.py` match the locked (A1/A2) copy exactly
|
||||||
|
- KB page-sub (`sources.html`) reads the locked current-source-model text (git repos + local dirs + uploaded archives, Sync pulls/re-imports); git-sources placeholder is `https://github.com/you/your-repo.git` (A3)
|
||||||
|
- All 9 pages carry `<span class="footer-text">Powered by self-hosted models</span>` (1 each); `grep -rn "Reese's" frontend/` empty; only remaining "homelab" is the A3-exempt L176 code comment in `index.html`
|
||||||
|
- Tests: `tests/unit/test_stale_ui_copy.py` + `tests/e2e/test_stale_ui_copy.py` exist with real pins (body-text scan, DOM assertions)
|
||||||
|
|
||||||
|
Results:
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing`: **1201 passed**, coverage TOTAL **99%** (>90% ✓)
|
||||||
|
- `uv run ruff check . && uv run pyright`: clean (0 errors)
|
||||||
|
- `uv run pytest tests/e2e/test_stale_ui_copy.py -v --no-cov`: **5 passed** in isolation (DB up)
|
||||||
|
- Regression E2E in isolation: `test_smoke` 3 ✓, `test_configurable_brand` 5 ✓, `test_header_consistency` 3 ✓, `test_dark_tech_theme` 6 ✓
|
||||||
|
- Commit `baefcde` `--no-gpg-sign` exists, staged paths only `app/ frontend/ tests/` ✓
|
||||||
|
|
||||||
|
Completion criteria: all 7 met (E2E body scan, KB page-sub, 9 footers, suite/coverage/lint, phase E2E, regression suites, single commit).
|
||||||
|
Next pending phase: **62_ui_customization**.
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
................................................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 101 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2458 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
**Task 01 complete — report**
|
||||||
|
|
||||||
|
Implemented:
|
||||||
|
- `frontend/index.html`: meta description, `.empty-state-sub`, and `#message-input` placeholder → locked neutral copy (A1); brand-embedding strings left untouched for the phase-39 brand layer.
|
||||||
|
- `app/config.py`: `suggestions` default → the four locked chips (A2); `BOR_SUGGESTIONS` override path unchanged.
|
||||||
|
- Defect fix (pre-existing — 3 `tests/integration/test_api.py` failures even on the pristine tree): the operator's local gitignored `.env` leaked into test apps via pydantic `env_file=".env"`. Made the test env hermetic — `tests/conftest.py` + `tests/e2e/conftest.py` now force `BOR_DOCS_REPO=""` and `BOR_SUGGESTIONS`=code default (derived from `Settings.model_fields` so it can't drift); `test_response_to_docs.py` "unconfigured" spawn forces empty docs repo (dotenv leak re-enabled the docs feature there too).
|
||||||
|
|
||||||
|
Results:
|
||||||
|
- `uv run pytest --cov=app` → 1193 passed; TOTAL 99% (>90%)
|
||||||
|
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
|
||||||
|
- `bash .agent/validate.sh` → validation OK
|
||||||
|
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov` → 3 passed (validates the modified shared E2E server)
|
||||||
|
- Task grep criteria verified: `homelab` in `index.html` only at the L176 code comment (A3); no "deployment" text; one user-visible "Ask me anything"
|
||||||
|
|
||||||
|
Decisions: conftest changes are a required defect fix (suite was red before this task on this machine); no commit made (task 04 owns it).
|
||||||
|
|
||||||
|
Next pending: `02_sources_pages_copy.md` (KB page-sub + remaining page-subs + git-sources example URL).
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 78%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
......................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 101 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2458 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
All gates green. Task 02 complete.
|
||||||
|
|
||||||
|
**Implemented**
|
||||||
|
- `frontend/sources.html` KB page-sub → locked (A1) copy: "Every file indexed from your configured sources — git repositories, local directories, and uploaded archives. Press **Sync sources** to pull the latest and re-import."
|
||||||
|
- `frontend/git-sources.html` example placeholder → `https://github.com/you/your-repo.git` (A3)
|
||||||
|
- `tests/unit/test_sync_button.py` — updated the stale copy-pin (it asserted the retired "clone the repos and re-import") to pin the new locked copy incl. the source-model wording
|
||||||
|
- Verification read-through, all **no change** (accurate today): `sources.js` sync result/error strings (server-driven, incl. "no sources configured (git or local)" from `app/api/sync.py:190`); page-subs in `tuning.html`, `history.html`, `doc-edit.html`, `git-sources.html`; `#sources-gate` copy (verified `GET /api/docs` admin-only, `GET /api/documents/content` public)
|
||||||
|
|
||||||
|
**Results**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term` → 1193 passed, TOTAL **99%** (>90%)
|
||||||
|
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
|
||||||
|
- `uv run pytest tests/e2e/test_smoke.py -v --no-cov` → 3 passed; `tests/e2e/test_header_consistency.py -v --no-cov` → 3 passed
|
||||||
|
- `grep "Homelab\|Deployments" frontend/sources.html` and `grep "homelab" frontend/git-sources.html` both empty; only A3-exempt comments remain
|
||||||
|
|
||||||
|
**Decisions**
|
||||||
|
- One pre-existing test pinned the retired copy (defect in previously completed work) — updated to the locked replacement rather than the HTML reverted
|
||||||
|
- No commit made: the phase's single atomic commit belongs to task 04
|
||||||
|
|
||||||
|
**Next pending task:** `.agent/phases/todo/61_stale_ui_copy/03_footer_defaults.md`
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 78%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
......................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 101 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2458 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
Task 03 is complete and green.
|
||||||
|
|
||||||
|
**Task 03 — Neutral footer default on all 9 pages (`.footer-text` hook)**
|
||||||
|
|
||||||
|
- Replaced `<span>Powered by Reese's self-hosted models</span>` → `<span class="footer-text">Powered by self-hosted models</span>` on all 9 pages (`index`, `sources`, `tuning`, `document`, `git-sources`, `history`, `login`, `shared`, `doc-edit`), exactly at the audited line numbers; `.footer-version` siblings untouched
|
||||||
|
- Verified `grep -c 'class="footer-text"' frontend/*.html` → 1 per file, 9 total; `grep -rn "Reese's" frontend/` → empty
|
||||||
|
- No CSS/JS changes — the class is a selector-only hook for phase 62's `BOR_FOOTER_TEXT`; no test pinned the old footer text
|
||||||
|
- Tests: `uv run pytest --cov=app` → 1193 passed (exit 0), coverage TOTAL **99%** (>90% gate); `uv run ruff check . && uv run pyright` → clean
|
||||||
|
- Brand layer unaffected: `uv run pytest tests/unit/test_frontend_brand.py tests/e2e/test_configurable_brand.py -v --no-cov` → 15 passed (DB up, so run now rather than deferred to task 04)
|
||||||
|
- The 9-page unit pin is deliberately deferred to task 04 per the task file; no new tests added in this task (no logic changes)
|
||||||
|
- Not committed — the single atomic phase commit is task 04's job; working tree carries tasks 01–03 changes
|
||||||
|
- Note: `.agent/PLAN.md` is absent from the repo; owner-locked copy was taken from the phase's `00_phase.md` locked decisions (A1)
|
||||||
|
|
||||||
|
**Next pending task:** `.agent/phases/todo/61_stale_ui_copy/04_tests_and_commit.md`
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 78%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
......................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 101 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2458 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
Task 04 complete. Report:
|
||||||
|
|
||||||
|
**Implemented**
|
||||||
|
- `tests/unit/test_stale_ui_copy.py` (8 tests, house text-pin pattern): old homelab-era strings GONE from `index.html`/`sources.html`/`git-sources.html`; locked A1 strings present exactly once; `class="footer-text"` + neutral footer on all 9 pages; A2 chips in `app/config.py` in order; old chips absent from all of `app/` + `frontend/` (retired, not relocated).
|
||||||
|
- `tests/e2e/test_stale_ui_copy.py` (5 tests, shared conftest server, DB up): chat placeholder/meta/footer pins, full body scan for `homelab`/`deployment`, A2 chip list in order, sources `.page-sub` (no Homelab/Deployments, has "configured sources"), git-sources `#git-source-url` placeholder (A3) — all from the rendered DOM.
|
||||||
|
|
||||||
|
**Verification (all green)**
|
||||||
|
- `uv run pytest --cov=app` → 1201 passed, coverage TOTAL **99%** (>90% gate)
|
||||||
|
- `uv run pytest tests/e2e/test_stale_ui_copy.py -v --no-cov` → 5 passed (isolation, DB up)
|
||||||
|
- Regression E2E in isolation: `test_smoke.py` 3 ✓, `test_configurable_brand.py` 5 ✓, `test_header_consistency.py` 3 ✓, `test_dark_tech_theme.py` 6 ✓
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean
|
||||||
|
|
||||||
|
**Decisions/deviations**
|
||||||
|
- Old git example URL also pinned (A3, "where they lived"); old-chip absence scan scoped to `app/`+`frontend/` (a repo-wide scan would hit the pin file itself).
|
||||||
|
- Atomic commit `baefcde` on `main` — staged `frontend/ tests/ app/` only, `--no-gpg-sign`, body carries the task-02 read-through outcome (all 6 checked locations: no change needed).
|
||||||
|
|
||||||
|
**Next pending task:** `62_ui_customization/01_config_settings.md`
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
................................................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 101 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2458 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
All completion criteria verified. Report:
|
||||||
|
|
||||||
|
**Phase 62 — UI Customization: final verification pass (all tasks already complete, commit `c738105` present)**
|
||||||
|
|
||||||
|
- Verified implementation: 3 new `Settings` fields + fail-loud `theme` validator, six-key `/api/config`, `brand.js` placeholder/footer/theme-link passes, `themes/indigo.css` + README, Containerfile L29 `cp -r ./assets/themes`, `.env.example` + README docs.
|
||||||
|
- Live spot-checks: default `/api/config` = exactly the six keys with phase-61 defaults; `/assets/themes/indigo.css` served 200 from dev static dir; `../evil.css`, `a/b.css`, `INDIGO.CSS`, `indigo` all refused at startup, each naming the value.
|
||||||
|
- `uv run pytest --cov=app` — all green; **coverage TOTAL 99%** (>90% gate).
|
||||||
|
- `uv run ruff check . && uv run pyright` — clean (0 errors).
|
||||||
|
- `uv run pytest tests/e2e/test_ui_customization.py -v --no-cov` — **5 passed** in isolation (DB up).
|
||||||
|
- Regression E2E in isolation: `test_configurable_brand` 5✓, `test_stale_ui_copy` 5✓, `test_smoke` 3✓, `test_dark_tech_theme` 6✓.
|
||||||
|
- Commit criterion: single `--no-gpg-sign` commit `c738105 feat(web): customizable placeholder, footer text, and color theme via BOR_* env vars` (stages only `app/ frontend/ tests/ .env.example README.md Containerfile`); `.agent/` phase files already filed under `complete/` by the harness.
|
||||||
|
- All 9 completion criteria: **met**. No defects found; no fixes needed.
|
||||||
|
- Note: `.agent/PLAN.md` referenced by AGENTS.md does not exist in the repo — verified against the phase overview's locked decisions (A4–A7), all honored.
|
||||||
|
- Next pending phase: **`63_unambiguous_document_listing`**.
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
.................................................................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2469 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
All green. Task complete — final verification summary:
|
||||||
|
|
||||||
|
**Implemented (task 01 — Settings + `/api/config` keys + validator):**
|
||||||
|
- `app/config.py`: `input_placeholder` / `footer_text` / `theme` (phase-61 neutral defaults) + `theme` field_validator — empty ok, bare `^[a-z0-9_-]+\.css$` only, malformed fails startup naming the value (phase-56 style)
|
||||||
|
- `app/api/config.py`: `GET /api/config` now returns exactly the six locked keys (values verbatim, same public posture)
|
||||||
|
- Contract tests moved to six keys: `tests/integration/test_api.py` (both key-set tests + new `test_config_serves_ui_customization_overrides` + default-value pins), `tests/e2e/test_configurable_brand.py` (both instances)
|
||||||
|
- Unit pins: `tests/unit/test_config.py` — validator accept/reject cases (each rejection names the value), env-override + `BOR_THEME=../evil.css` startup failure
|
||||||
|
|
||||||
|
**Test / lint / coverage results:**
|
||||||
|
- `uv run pytest` → **1211 passed**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term` → **TOTAL 99%** (>90% gate)
|
||||||
|
- `uv run ruff check .` → clean · `uv run pyright` → 0 errors
|
||||||
|
- `bash .agent/validate.sh` → "validation OK" · e2e file collects (5 tests; run lands in task 05)
|
||||||
|
|
||||||
|
**Notable decisions:**
|
||||||
|
- Also updated `tests/unit/test_save_as_doc_button.py` L54 key-set pin (not in the task's inventory — suite would go red without it) and added the phase-61-style `.env`-leak guard for the three new vars in `tests/conftest.py`
|
||||||
|
- No commit made — the phase's single atomic commit belongs to task 05
|
||||||
|
|
||||||
|
**Next pending task:** `02_brand_layer_customization.md` (brand.js applies placeholder/footer/theme from the same boot fetch)
|
||||||
+71
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
........................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2469 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+23
@@ -0,0 +1,23 @@
|
|||||||
|
Task 02 complete. All gates green.
|
||||||
|
|
||||||
|
**Implemented**
|
||||||
|
- `frontend/assets/brand.js`: the three phase-62 keys apply from the SAME settled-config `.then` (no second fetch), placed after the app_name block and independent of it (the name passes are now wrapped in `if (name)` — behavior byte-identical to the old early-return):
|
||||||
|
- `input_placeholder` → `#message-input` placeholder (null-guard no-op off the chat page)
|
||||||
|
- `footer_text` → every `.footer-text` node via `textContent` (no markup injection)
|
||||||
|
- `theme` → `<link id="theme-override" href="/assets/themes/<name>">` inserted immediately after the `styles.css` link (cascade wins), idempotent via `#theme-override`, A5 degradation: `onerror` → `console.warn`, built-in theme stands
|
||||||
|
- Header contract comment: new "Phase 62 (owner-locked 2026-09-01, TODO L3)" items 5–7 + updated no-op property (all 4 env vars unset ⇒ byte-identical)
|
||||||
|
- `tests/unit/test_frontend_brand.py`: new pin `test_brand_js_applies_phase_62_customization_from_the_same_fetch` — single-fetch marker, all three application markers, empty-skip guards, phase-62-after-app_name ordering, app_name default literal still holds
|
||||||
|
- Existing app_name passes 1–4 and `BOR_DOCS_REPO_CONFIGURED` logic untouched (re-indent only)
|
||||||
|
|
||||||
|
**Results**
|
||||||
|
- `uv run pytest` → 1212 passed
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL 99% (>90%)
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||||
|
- `bash .agent/validate.sh` → validation OK
|
||||||
|
- `node --check frontend/assets/brand.js` → syntax OK
|
||||||
|
|
||||||
|
**Decisions**
|
||||||
|
- Keyed the empty-skip guards on `typeof … === "string"` + truthiness, so a fetch failure (`cfg` null) is also a clean no-op (house style: page never breaks).
|
||||||
|
- No defects found in prior work — baseline was green before my changes.
|
||||||
|
|
||||||
|
**Next pending task:** `.agent/phases/todo/62_ui_customization/03_theme_assets.md`
|
||||||
+71
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
............................................................ [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2469 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
All done and green. Final report:
|
||||||
|
|
||||||
|
**Implemented (task 03 — theme assets)**
|
||||||
|
- `frontend/assets/themes/indigo.css` — WCAG-AA dark indigo/slate theme, single `:root` block, exactly the 8 identity overrides (semantic accent/ok/err untouched)
|
||||||
|
- `frontend/assets/themes/README.md` — authoring guide (load path, variable table w/ built-in refs, filename regex, 4.5:1 bar, never-white-on-brand trap, A7 deployment)
|
||||||
|
- `Containerfile` — `&& cp -r ./assets/themes /out/assets/themes \` added in stage 1, after the styles.css minify (line 29)
|
||||||
|
- `tests/unit/test_themes.py` — 6 new pins: files exist, single leading `:root` block + nothing else, exactly the 8 locked overrides, 5 identity-pair ratios ≥4.5:1 computed from the file's own hex values, Containerfile cp line (position + no esbuild), guide contract markers
|
||||||
|
|
||||||
|
**Test / lint / coverage**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term -q` → rc=0, 1218 tests, TOTAL **99%** (>90%)
|
||||||
|
- `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors
|
||||||
|
- `uv run pytest tests/unit/test_themes.py tests/integration/test_containerfile_assets.py -v --no-cov` → 12 passed
|
||||||
|
- Ratios (WCAG calc): ink/bg 15.84 · ink/surface 14.71 · ink-soft/surface 8.27 · bg-on-brand 6.46 · brand-ink/surface 11.99 — all ≥4.5:1
|
||||||
|
- Dev sanity: `BOR_THEME=indigo.css` → `GET /assets/themes/indigo.css` 200, `/api/config` theme="indigo.css", served HTML has 0 theme refs; default env → theme="", built-in `--brand: #f43f5e` stands (opt-in via env var, no dir scanning)
|
||||||
|
|
||||||
|
**Decisions / notes**
|
||||||
|
- Task's comment ratios were slightly off (14.6/8.2/6.4/11.9); wrote the computed values (14.7/8.3/6.5/12.0) — no palette values changed
|
||||||
|
- Required grep spot-check: `f43f5e|#f0e6e6|#0f0a0a` matches only the `:root` block (L9/11/14) + one comment (L1271) as expected; broader scan found decorative `rgb()` literals (bg grid, glow spots, `.send-btn` steps) — noted, not refactored per task
|
||||||
|
- Commit deferred to task 05's atomic phase commit (per phase plan)
|
||||||
|
|
||||||
|
**Next pending task:** `.agent/phases/todo/62_ui_customization/04_docs.md`
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
.................................................................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2469 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
All gates green. Task complete.
|
||||||
|
|
||||||
|
**Report**
|
||||||
|
|
||||||
|
- `.env.example` — added the three commented vars (`BOR_INPUT_PLACEHOLDER`, `BOR_FOOTER_TEXT`, `BOR_THEME`) in the `# --- App ---` block directly after the `BOR_APP_NAME` line, task's exact text/defaults, phase-cited.
|
||||||
|
- `README.md` env table — three rows added immediately after the `BOR_APP_NAME` row (what it affects, default, unset ⇒ no-op contract; `BOR_THEME` row documents the startup validator regex + missing-file degradation).
|
||||||
|
- `README.md` — new `### Customizing the look` subsection (4 sentences) placed at the end of "Configuration reference", directly **below** the env table, before "## Troubleshooting" — mirrors the house pattern of `###` feature subsections under a parent `##`; links to `frontend/assets/themes/README.md` (verified exists, link resolves) and names `indigo.css` as the example.
|
||||||
|
- No code changes (task is docs-only, per spec).
|
||||||
|
- Tests/lint: `uv run pytest` → 1218 passed; `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% gate); `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors.
|
||||||
|
- No commit made — the phase's single atomic commit is defined in task 05 (harness/handoff per phase overview).
|
||||||
|
|
||||||
|
Next pending task: `.agent/phases/todo/62_ui_customization/05_e2e_customization.md` (E2E suite + regression suites + atomic commit).
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
.................................................................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2469 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
All gates green and the atomic commit landed. Final report:
|
||||||
|
|
||||||
|
## Phase 62 / Task 05 — complete
|
||||||
|
|
||||||
|
**Implemented**
|
||||||
|
- `tests/e2e/test_ui_customization.py` — 5 mapped tests: `/api/config` six-key overrides + dev static serving of the theme; chat page custom placeholder/footer/`link#theme-override` + computed `--brand` `#818cf8`; multi-page footer (login, no composer); default server byte-identical (settled-fetch wait before the absence assert); malformed `BOR_THEME="../evil.css"` refuses boot via real uvicorn subprocess, stderr names the value. Own instance on `APP_PORT+2`, brand suite's env-block pattern.
|
||||||
|
- **Defect fix (task 02 work):** the phase-33/54 cache-bust rewrite serves `styles.css?v=<token>`, so brand.js's `el.href.endsWith("styles.css")` never matched and the theme link was never inserted — now matches the raw attribute path with query/fragment stripped; unit pin added for the exact regression.
|
||||||
|
- E2E conftest hardening: shared `app_server` pins the three new `BOR_` vars to code defaults (same leak-class fix as phase 61).
|
||||||
|
|
||||||
|
**Results**
|
||||||
|
- `uv run pytest tests/e2e/test_ui_customization.py -v --no-cov` → 5 passed (isolation, DB up)
|
||||||
|
- Regressions in isolation: `test_configurable_brand` 5✓, `test_stale_ui_copy` 5✓, `test_smoke` 3✓, `test_dark_tech_theme` 6✓
|
||||||
|
- `.agent/validate.sh` → unit+integration green, coverage TOTAL **99%** (>90%), ruff clean, pyright 0 errors
|
||||||
|
- Commit `c738105` on `main`, staged exactly `app/ frontend/ tests/ .env.example README.md Containerfile`, `--no-gpg-sign`
|
||||||
|
|
||||||
|
**Decisions:** `.agent/` left unstaged per task file (harness moves the phase dir). Next pending: `63_unambiguous_document_listing/01` (or the harness's queue order).
|
||||||
+71
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
.................................................................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2469 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+17
@@ -0,0 +1,17 @@
|
|||||||
|
All verification complete. The one remaining `source/path —` match is an explanatory comment in a test describing the historical bug (introduced by this phase's own commit) — not stale.
|
||||||
|
|
||||||
|
**Phase 63 — Final verification pass: ALL GREEN** (all 4 tasks already in `complete/`; no code changes needed)
|
||||||
|
|
||||||
|
- Verified `app/rag/agent.py`: `_execute_tool` emits `source: X | path: Y | title: Z` lines; `AGENT_TOOLS` `read_document` param descriptions reference the `source:`/`path:` labels; "No document at …" refusal unchanged
|
||||||
|
- Verified `tests/e2e/mock_llm.py::_catalog_docs` parses the new format via labeled-field regex; README + user story updated to new format; no stale old-format references remain
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → **1219 passed**, TOTAL coverage **99%** (>90% gate)
|
||||||
|
- `uv run ruff check . && uv run pyright` → **All checks passed / 0 errors**
|
||||||
|
- `uv run pytest tests/e2e/test_agent_document_tools.py tests/e2e/test_agent_unlimited_tools.py -v --no-cov` → **8 passed** (DB up)
|
||||||
|
- Regression in isolation: `test_whole_document_context.py` **3 passed**; `tests/integration/test_chat_api.py` **17 passed**; `tests/unit/test_sse_events.py` **12 passed**; `tests/integration/test_agent_tools.py` **4 passed**
|
||||||
|
- Commit `15a16a8 fix(agent): unambiguous document listing format for LLM parsing` already present (`--no-gpg-sign`); harness has moved task files to `complete/`
|
||||||
|
|
||||||
|
Completion criteria: (1) new format + unit test — ✓ (2) tool descriptions — ✓ (3) mock parsing — ✓ (4) pytest/coverage/lint — ✓ (5) E2E document tools — ✓ (6) regression suites — ✓ (7) commit + phase move — ✓ (commit exists; `00_phase.md` move handled by harness)
|
||||||
|
|
||||||
|
Deviation: `test_chat_api.py`/`test_sse_events.py` don't exist under `tests/e2e/`; ran their actual locations (integration/unit) in isolation — all green.
|
||||||
|
|
||||||
|
Next pending phase: none — `todo/` holds only phase 63 itself.
|
||||||
+71
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2469 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+11
@@ -0,0 +1,11 @@
|
|||||||
|
**Task 01 complete — report**
|
||||||
|
|
||||||
|
- `app/rag/agent.py`: `list_documents` output now `source: X | path: Y | title: Z` (labeled, pipe-delimited — owner-locked A1); `read_document` `source`/`path` param descriptions now point the LLM at the `source:`/`path:` labels (A2, verbatim from task file); `list_documents` tool description + module docstring updated to match the new format (old wording would have misdescribed the output). "No document at …" refusal and SSE `argument` unchanged (A3/A4).
|
||||||
|
- `tests/unit/test_agent.py`: tool-shape test now asserts the new param descriptions; all three listing-format assertions updated.
|
||||||
|
- `tests/unit/test_mock_tool_flow.py`: `CATALOG_*` wire fixtures updated to the new format (they must stay byte-identical to `agent.py` output).
|
||||||
|
- Deviation: `tests/e2e/mock_llm.py::_catalog_docs` (nominally task 02) was pulled into this task — the old `rsplit("/", 1)` parse would have mis-read the new lines and broken the phase's mandatory E2E gate; now regex-parses the labeled fields, producing identical `(source, path)` tuples (verified round-trip incl. the `brain-of-reese-main/homelab/aws-route53.md` case from the bug report).
|
||||||
|
- `uv run pytest --cov=app` → 1218 passed, TOTAL coverage **99%** (>90%)
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||||
|
- `uv run pytest tests/e2e/test_agent_document_tools.py tests/e2e/test_agent_unlimited_tools.py -v --no-cov` → 8 passed (DB up); regression `test_whole_document_context.py` → 3 passed. (`test_chat_api.py`/`test_sse_events.py` listed in the phase don't exist in this repo.)
|
||||||
|
- No commit made — the phase (tasks 02–04 remaining) gets one atomic commit at phase completion per AGENTS.md rule 8.
|
||||||
|
- Next pending task: `63_unambiguous_document_listing/02_mock_and_unit_tests.md` (its mock/unit work is already done here — expect verification only).
|
||||||
+71
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
.................................................................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2469 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+19
@@ -0,0 +1,19 @@
|
|||||||
|
All gates green. Task 02 complete.
|
||||||
|
|
||||||
|
**Task 02 report — mock + unit tests for the new catalog format**
|
||||||
|
|
||||||
|
- **State found:** task 01's session had already applied the task-02 code (noted as a deviation in its report); this session verified it end-to-end and closed the one test gap.
|
||||||
|
- `tests/e2e/mock_llm.py::_catalog_docs` — verified: regex-parses the labeled `source: X | path: Y | title: Z` lines; produces identical `(source, path)` tuples for all test catalogs.
|
||||||
|
- `tests/unit/test_agent.py` — verified: `test_agent_tools_names_and_parameters` asserts the new A2 `source:`/`path:` parameter descriptions; all three listing-format assertions use the new format; "No document at S/ghost.md" refusal unchanged (A4).
|
||||||
|
- **Added** `test_read_step_nested_path_stays_intact` in `tests/unit/test_mock_tool_flow.py` — pins the actual bug: the `brain-of-reese-main` / `homelab/aws-route53.md` nested path (multiple `/`) parses intact; the old `rpartition` parse misread it.
|
||||||
|
|
||||||
|
**Gates (exact commands → outcomes)**
|
||||||
|
- `uv run pytest tests/unit/test_agent.py -v --no-cov` → 22 passed
|
||||||
|
- `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov` → 4 passed (DB up)
|
||||||
|
- `uv run pytest tests/e2e/test_agent_document_tools.py tests/e2e/test_agent_unlimited_tools.py -v --no-cov` → 8 passed in isolation
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → **1219 passed**, coverage TOTAL **99%** (>90%)
|
||||||
|
- `uv run ruff check tests/ && uv run pyright tests/` → clean; `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||||
|
|
||||||
|
**Decisions/deviations:** none — mock and unit work was already in the working tree from task 01's session (its report documents the pull-in); no commit made (one atomic commit lands at phase completion per AGENTS.md rule 8).
|
||||||
|
|
||||||
|
**Next pending task:** `63_unambiguous_document_listing/03_e2e_document_tools.md`
|
||||||
+71
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2469 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+16
@@ -0,0 +1,16 @@
|
|||||||
|
All gates green. Task 03 is verification-only (no code changes needed, consistent with its "Files changed: None"). Following the phase convention (one atomic commit at phase completion, per AGENTS.md rule 8), no commit is made here.
|
||||||
|
|
||||||
|
**Task 03 complete — E2E document tools validation**
|
||||||
|
|
||||||
|
- **Verified (no code changes):** SSE `tool` frames carry `argument` = `source/path`; UI renders "Listing documents"/"Reading …"; source chips `(source, path)` pairs; mock's deterministic answer content; query log `sources` = `"source/path, source/path"` — all asserted by the passing E2E suites against the new `source: X | path: Y | title: Z` catalog format.
|
||||||
|
- **Gates (DB up, all in isolation):**
|
||||||
|
- `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov` → 4 passed
|
||||||
|
- `uv run pytest tests/e2e/test_agent_unlimited_tools.py -v --no-cov` → 4 passed
|
||||||
|
- `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov` → 3 passed
|
||||||
|
- `uv run pytest tests/e2e/test_chat_rag.py -v --no-cov` → 3 passed
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → **1219 passed**, coverage TOTAL **99%** (>90%)
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||||
|
- **Deviations:** `test_chat_api.py` / `test_sse_events.py` named in the task never existed in this repo (verified via `git log --all`); ran `test_chat_rag.py` as the chat-API/SSE regression equivalent — same finding as task 01's report.
|
||||||
|
- **No defects found** in prior work; no commit (phase-level commit at completion).
|
||||||
|
|
||||||
|
**Next pending task:** `63_unambiguous_document_listing/04_docs.md`
|
||||||
+71
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2469 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+18
@@ -0,0 +1,18 @@
|
|||||||
|
All green. Phase 63 is now fully complete.
|
||||||
|
|
||||||
|
**Task 04 — documentation update**
|
||||||
|
|
||||||
|
- `README.md` — "Agent document tools" section now documents the `list_documents` output as `source: X | path: Y | title: Z` lines
|
||||||
|
- `.agent/user_stories/agent-document-tools.md` — acceptance criterion 3 updated to the labeled format (reflowed, meaning unchanged)
|
||||||
|
- Verified: `grep "source/path — title\|source/path - title"` over `README.md` + `.agent/user_stories/*.md` → no matches; `uv run ruff check README.md .agent/` → no error. Other `source/path` hits (SSE `argument`, source chips, `app/config.py` KB-overview note) are locked unchanged per A3 / are a different format
|
||||||
|
- Committed phase work (tasks 01–04) as `15a16a8 fix(agent): unambiguous document listing format for LLM parsing` (`--no-gpg-sign`, staged `app/ tests/ README.md`; `.agent/` tree left for the harness, matching phases 61/62)
|
||||||
|
|
||||||
|
**Test / lint / coverage results**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → exit 0, 1231 results, 0 failures; coverage TOTAL **99%** (>90% gate)
|
||||||
|
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
|
||||||
|
- E2E isolated (DB up): `test_agent_document_tools.py` 4 passed · `test_agent_unlimited_tools.py` 4 passed · `test_whole_document_context.py` 3 passed
|
||||||
|
- Regression isolated: `tests/integration/test_chat_api.py` 17 passed · `tests/unit/test_sse_events.py` 12 passed (the phase's `tests/e2e/` paths for these two don't exist — actual locations used)
|
||||||
|
|
||||||
|
**Deviations:** none — phase commit command's `app/ tests/` extended with `README.md` (task 04's deliverable; phase-62 precedent).
|
||||||
|
|
||||||
|
**Next pending task:** none — task 04 was the last task of phase 63; `todo/` is now empty aside from this phase's harness-managed files.
|
||||||
+71
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 173 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2469 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+15
@@ -0,0 +1,15 @@
|
|||||||
|
All green. Task complete.
|
||||||
|
|
||||||
|
**Report — phase 64, task 01 (importer progress hook)**
|
||||||
|
|
||||||
|
- `app/rag/importer.py`: `import_sources` gains optional kwarg `progress: Callable[[str, str, int, int], None] | None`; called once per importable file immediately before `_index_file` with `(source, rel_posix_path, done, total)` — `done` = 1-based index across all sources (`summary.files`), `total` = combined count from a pre-walk via the existing `iter_importable_files` (stats only, no file reads), done **only when `progress` is provided**; `limit` path unchanged (hook fires per processed file, `total` stays full); module docstring notes the hook (phase 64).
|
||||||
|
- `tests/unit/test_importer.py`: 4 new tests — exact multi-root sequence (`total` = combined), no-pre-walk sentinel (monkeypatched counting wrapper: 1 walk without hook vs 2 with), unchanged + `EmbeddingError` files still reported as current, and `limit=` (2 events, `total` = 3).
|
||||||
|
|
||||||
|
**Results**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term`: **1223 passed**, `app/` coverage **99%** (>90% gate)
|
||||||
|
- `uv run ruff check . && uv run pyright`: clean (0 errors)
|
||||||
|
- Existing callers (`scripts/import_docs.py`, `app/api/sync.py`, `app/api/git_sources.py`) untouched and byte-identical in behavior — all pre-existing tests green.
|
||||||
|
|
||||||
|
**Notes:** `.agent/PLAN.md` no longer exists in the repo (removed in commit `281f355`); the phase files served as binding context — no deviations. No commit made (phase commit belongs to task 06).
|
||||||
|
|
||||||
|
**Next pending task:** `64_sync_upload_progress/02_sync_status_current_file.md`
|
||||||
+71
@@ -0,0 +1,71 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 58%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
....................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 89 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 180 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2476 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+17
@@ -0,0 +1,17 @@
|
|||||||
|
Everything is green and the diff is clean. Task complete.
|
||||||
|
|
||||||
|
## Report
|
||||||
|
|
||||||
|
**Implemented (task 02 — sync status reports the current file):**
|
||||||
|
- `app/api/sync.py`: `SyncStatus` gains `current_file`/`files_done`/`files_total`; `GET /api/sync/status` returns them (null/0/0 idle, all pre-existing keys unchanged); `_run_sync` resets them at start, passes the task-01 hook to `import_sources(..., prune=True, progress=_hook)` assigning `f"{source}/{rel}"` + counts, and clears `current_file` in both terminal states while keeping final counts; module + endpoint + dataclass docstrings updated (clone/pull phase reports no file per A4).
|
||||||
|
- `tests/unit/test_sync_button.py`: 3 new unit tests — full idle dict pinned (new keys null/0/0), mid-run status via the runner's real hook closure (worker-thread + gated clone/import, also pins A4 no-file-during-clone and success-terminal behavior), failed-terminal clears `current_file` keeping counts with sanitized error.
|
||||||
|
- `tests/integration/test_sync_api.py` (defect fix for prior work): `FakeImportSources`/`failing_import` now accept + record the `progress` kwarg, idle-shape pin extended with the three keys, success test pins the hook wiring.
|
||||||
|
|
||||||
|
**Results:**
|
||||||
|
- `uv run pytest` → 1226 passed
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → 1226 passed, TOTAL 99% (`app/api/sync.py` 100%)
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||||
|
|
||||||
|
**Decisions:** unit tests drive `_run_sync` directly (no DB/HTTP; seams monkeypatched on `app.api.sync` — the house mock-import pattern), since the task named `test_sync_button.py` though the API coverage lives in the integration suite; in-memory `GitSource` needs explicit `kind="git"` (Python default applies at INSERT flush). No UI changes — "Syncing…" stays until task 04. No commit (phase commit belongs to task 06).
|
||||||
|
|
||||||
|
**Next pending task:** `.agent/phases/todo/64_sync_upload_progress/03_upload_background_202.md`
|
||||||
+72
@@ -0,0 +1,72 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 46%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 58%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
.. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 143 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 101 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 180 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 198 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2488 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+20
@@ -0,0 +1,20 @@
|
|||||||
|
All green. Final report:
|
||||||
|
|
||||||
|
## Task 03 complete — background upload (202 + status endpoint)
|
||||||
|
|
||||||
|
**Implemented**
|
||||||
|
- `app/schemas.py`: new `UploadAccepted` (`detail` + safe `name`); `UploadOut` kept — now the shape of the status `detail` on success.
|
||||||
|
- `app/api/git_sources.py`: `POST /api/git-sources/upload` → **202** the moment the archive is on disk (`db` dependency dropped; inline gates unchanged: name 422, flag 409 set *before* the streamed receive, cap 413 with temp deleted + flag released on the pre-task exception path); unpack → swap → row upsert (short-lived `SessionLocal`, `IntegrityError` → failed naming the path) → `check_models` → `import_sources(..., progress=_hook)` → gated overview → moved INFO log line → `success` with `UploadOut` fields, all in `_run_upload` (phase-32 `_run_sync` shape; failures land sanitized in `failed`; `finally` cleans temps + clears the flag; `CancelledError` not caught). New `UploadStatus` dataclass + `GET /api/git-sources/upload/status` (identical key set to `/api/sync/status`, admin-only).
|
||||||
|
- `tests/integration/test_git_sources_upload.py`: all 29 scenarios kept, observation point moved to the status endpoint (polled to terminal) — every background branch has a dedicated test (unpack/zero-entry/swap/IntegrityError/model-down/import-fail/success/cancel-cleanup), mid-run `current_file`/counts via the task-02 hook seam, flag-not-state 409 pins, temp-on-disk + 202-before-scan pins, TRUNCATE regression.
|
||||||
|
|
||||||
|
**Results**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term` → **1233 passed**, app/ **99%** (`git_sources.py` 100%)
|
||||||
|
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
|
||||||
|
- `bash .agent/validate.sh` → validation OK; upload suite alone: 29 passed in ~1 s
|
||||||
|
|
||||||
|
**Notable decisions**
|
||||||
|
- The task's "adapt `tests/unit/test_archive_upload.py`" bullet misreferences the file: that suite covers the unchanged `archive_upload` *utility*; the 202-contract HTTP scenarios live in `tests/integration/test_git_sources_upload.py` (adapted there; unit file untouched, still green).
|
||||||
|
- Discovered: starlette's TestClient delivers the response only after the app's task-completion callback runs on the loop — a sync park inside `unpack_archive` stalls even the 202. So the temp-on-disk pin drives `_run_upload` directly (worker loop), and the HTTP "202 while in flight" pin uses the async import gate.
|
||||||
|
- `test_archive_upload_sources.py` E2E expected to fail until task 06 (noted, not fixed here).
|
||||||
|
|
||||||
|
**Next pending task:** `04_sync_button_live_file.md`
|
||||||
+72
@@ -0,0 +1,72 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 46%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 58%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
......... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 181 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 101 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 180 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 201 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2529 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+14
@@ -0,0 +1,14 @@
|
|||||||
|
Task 04 complete. All changes are confined to `frontend/` + the two test files (the `app/`/integration diffs were pre-existing uncommitted work from tasks 01–03, untouched by me).
|
||||||
|
|
||||||
|
**Implemented**
|
||||||
|
- `frontend/assets/sources.js`: `fmtSyncLabel(kind, currentFile, done, total)` (sync → "Syncing…", upload → "Importing", file appended only when present, counts only when `total > 0`); `enterSyncRunningState(kind, file, done, total)` writes the full untruncated path to button `title` (removed when null) + `#sync-result` aria-live; unified tick fetches both `/api/sync/status` and `/api/git-sources/upload/status` (upload 403 = "no upload", blips retry) with the 7-branch decision tree (sync running → upload running → sync success → sync failed → upload success → upload failed → idle); upload success settles + `loadDocs()` with no counts in `#sync-result` (A3); 202/409 click branch → `enterSyncRunningState("sync", null, 0, 0)`; `initSyncButton` re-attaches a running upload when sync is idle; header comment documents the contract
|
||||||
|
- `frontend/assets/styles.css`: `.sync-label` ellipsis (inline-block, `min(16rem, 40vw)`, ellipsis, nowrap)
|
||||||
|
- `frontend/sources.html`: comment block updated (no structural change)
|
||||||
|
- New `tests/unit/test_frontend_sync_upload.py` (9 pins); adapted 2 now-stale pins in `tests/unit/test_sync_button.py`
|
||||||
|
|
||||||
|
**Results**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term` → 1242 passed, `app/` 99% (>90% gate)
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||||
|
- E2E regressions in isolation: `test_sync_button.py` 3✓, `test_sync_model_down.py` 4✓, `test_git_sources_admin.py` 6✓, `test_local_directory_sources.py` 3✓; `node --check` + runtime `fmtSyncLabel` checks pass
|
||||||
|
|
||||||
|
**Notes**: no deviations from the task spec; no commit (task 06 owns it). Next pending task: `.agent/phases/todo/64_sync_upload_progress/05_upload_toast_progress.md`.
|
||||||
+72
@@ -0,0 +1,72 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 46%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 63%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
.................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 181 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 101 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 180 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 201 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2529 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
+19
@@ -0,0 +1,19 @@
|
|||||||
|
Everything is verified. All deletions in `git-sources.js` are confined to the phase-49 upload doc/paragraph (marked superseded) and the synchronous submit handler; load/add/remove behavior is untouched.
|
||||||
|
|
||||||
|
## Report — Task 05 complete
|
||||||
|
|
||||||
|
**Implemented** (`frontend/assets/git-sources.js` + `frontend/git-sources.html`):
|
||||||
|
- `showUploadToast` — page-local phase-55 toast: single lazy node, `role="status" aria-live="polite"`, ~5 s auto-dismiss, new toast replaces pending one; `.toast` CSS reused as-is
|
||||||
|
- Submit rework: 202 → toast `Successfully uploaded — <name>` (UploadAccepted body, file-name fallback) → clear input → processing state → poll; 409 → re-attach, NO error banner; other non-2xx/network keep phase-49 banners + kept selection; `finally` restores only when no poll is active (§7.4)
|
||||||
|
- `startUploadPolling` — 2 s single-timer poll of `GET /api/git-sources/upload/status`: running → live `"Processing… <file> (n/m)"` + full-path `title`; success → result line + announce + `loadSources()` (no second toast); failed → sanitized error banner + `loadSources()`; idle → defensive restore
|
||||||
|
- `initUploadStatus` boot re-attach (admin branch): running → processing + poll; success → result line only; failed → banner; idle → nothing; header/form comments rewritten (phase-49 paragraph superseded)
|
||||||
|
- `tests/unit/test_frontend_sync_upload.py` extended with 10 source pins (all task-04 pins intact)
|
||||||
|
|
||||||
|
**Results:**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → 1251 passed, coverage **99%** (>90 ✓)
|
||||||
|
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||||
|
- Ad-hoc Playwright sanity (temp files, deleted): toast-on-202 + auto-dismiss + single node, Processing→settle, reload re-render, 409 re-attach without banner, mid-scan reload (160 files) re-attaches to `Processing…` and settles — zero uncaught JS errors
|
||||||
|
|
||||||
|
**Notes:** `tests/e2e/test_archive_upload_sources.py` still expects the old 200 flow — expected-fail until task 06 (noted in task 03); no commit (phase commit lands with task 06).
|
||||||
|
|
||||||
|
**Next pending task:** `06_e2e_sync_upload_progress.md`
|
||||||
+72
@@ -0,0 +1,72 @@
|
|||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 46%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 63%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 80%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 97%]
|
||||||
|
........................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
-----------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 22 0 100%
|
||||||
|
app/api/chat.py 131 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 7 0 100%
|
||||||
|
app/api/doc_drafts.py 93 0 100%
|
||||||
|
app/api/docs.py 50 0 100%
|
||||||
|
app/api/git_sources.py 181 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 8 0 100%
|
||||||
|
app/api/sync.py 101 0 100%
|
||||||
|
app/config.py 112 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 20 0 100%
|
||||||
|
app/core/caching.py 108 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/db.py 21 0 100%
|
||||||
|
app/main.py 52 0 100%
|
||||||
|
app/models.py 86 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 78 0 100%
|
||||||
|
app/rag/archive_upload.py 128 0 100%
|
||||||
|
app/rag/chunker.py 206 4 98%
|
||||||
|
app/rag/git_sources.py 14 0 100%
|
||||||
|
app/rag/importer.py 180 3 98%
|
||||||
|
app/rag/llm.py 189 0 100%
|
||||||
|
app/rag/overview.py 71 0 100%
|
||||||
|
app/rag/prompts.py 65 0 100%
|
||||||
|
app/rag/retriever.py 94 3 97%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 24 0 100%
|
||||||
|
app/schemas.py 201 0 100%
|
||||||
|
-----------------------------------------------
|
||||||
|
TOTAL 2529 12 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
validation OK
|
||||||
@@ -35,7 +35,8 @@ addition to "thinking".
|
|||||||
3. `app/rag/agent.py`: the loop — budgets from
|
3. `app/rag/agent.py`: the loop — budgets from
|
||||||
`BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` (default 1/1); the
|
`BOR_AGENT_LIST_CALLS` / `BOR_AGENT_READ_CALLS` (default 1/1); the
|
||||||
`list_documents` tool returns the DB catalog
|
`list_documents` tool returns the DB catalog
|
||||||
(`source/path — title` lines, `/api/docs` order); `read_document`
|
(`source: X | path: Y | title: Z` lines, `/api/docs` order);
|
||||||
|
`read_document`
|
||||||
appends the **full** document text (A7-revised: never truncated); after
|
appends the **full** document text (A7-revised: never truncated); after
|
||||||
both budgets are spent the tools are dropped and the model must answer;
|
both budgets are spent the tools are dropped and the model must answer;
|
||||||
safety cap on model rounds; the deflection (LOW) path is byte-identical.
|
safety cap on model rounds; the deflection (LOW) path is byte-identical.
|
||||||
|
|||||||
+290
-137
@@ -17,13 +17,19 @@ Routes: ``GET`` (DB rows oldest-first, or the env list with
|
|||||||
``path``, git rows — and env rows — report ``path: null``), ``POST``
|
``path``, git rows — and env rows — report ``path: null``), ``POST``
|
||||||
(201, validated create; ``kind`` selects the validation: git → exactly
|
(201, validated create; ``kind`` selects the validation: git → exactly
|
||||||
the phase-35 URL contract, local → an existing absolute directory, else
|
the phase-35 URL contract, local → an existing absolute directory, else
|
||||||
422 naming the path), ``POST /upload`` (phase 49 — admin archive upload:
|
422 naming the path), ``POST /upload`` (phase 49, backgrounded in phase
|
||||||
``.tar``/``.tar.gz``/``.tgz``/``.zip`` streamed with a size cap, safely
|
64 task 03 — admin archive upload: the ``.tar``/``.tar.gz``/``.tgz``/
|
||||||
unpacked, atomically swapped in over an existing folder of the same
|
``.zip`` name/format gate + the 1 MiB-chunk receive with the
|
||||||
name, row upserted, then the synchronous single-source scan — see
|
``upload_max_mb`` cap run **inline** and answered 202 the moment the
|
||||||
:func:`upload_archive`), ``DELETE /{source_id}`` (204). The whole
|
archive is safely on disk; unpack → swap → row upsert → model check →
|
||||||
router sits behind :func:`app.core.auth.require_admin` — anonymous
|
single-source scan → change-gated overview then run in a **background
|
||||||
callers get 403 on every route.
|
task** — see :func:`upload_archive` and :func:`_run_upload`),
|
||||||
|
``GET /upload/status`` (the phase-32 ``SyncStatus``-shaped in-memory
|
||||||
|
state of that run — incl. the phase-64 ``current_file`` /
|
||||||
|
``files_done`` / ``files_total`` progress fields; navigating away from
|
||||||
|
the page mid-scan no longer aborts anything), ``DELETE /{source_id}``
|
||||||
|
(204). The whole router sits behind :func:`app.core.auth.require_admin`
|
||||||
|
— anonymous callers get 403 on every route.
|
||||||
|
|
||||||
No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's
|
No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's
|
||||||
masking discipline), so every git 409/422 detail is a fixed generic
|
masking discipline), so every git 409/422 detail is a fixed generic
|
||||||
@@ -33,21 +39,26 @@ owner sees exactly which directory failed.
|
|||||||
|
|
||||||
Scope boundary (phase locked decisions): the CRUD routes do NOT
|
Scope boundary (phase locked decisions): the CRUD routes do NOT
|
||||||
clone, import, or prune anything — the existing Sync button performs
|
clone, import, or prune anything — the existing Sync button performs
|
||||||
that (a removal prunes on the next sync, ``prune=True``). The phase-49
|
that (a removal prunes on the next sync, ``prune=True``). The upload
|
||||||
upload route is the exception: it unpacks the archive and then scans
|
route is the exception (phase 64, task 03): after the 202 receive
|
||||||
the single source synchronously in the request (``import_sources``
|
answer, its background task unpacks the archive, swaps it in, upserts
|
||||||
with ``prune=True`` + the change-gated overview refresh) and answers
|
the row, probes the models, scans the single source
|
||||||
with the sync-style counts.
|
(``import_sources`` with ``prune=True`` + the change-gated overview
|
||||||
|
refresh), and lands the sync-style counts (the ``UploadOut`` fields)
|
||||||
|
in the status ``detail``.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
import re
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
import time
|
import time
|
||||||
import uuid
|
import uuid
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import UTC, datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Literal, cast
|
from typing import Any, Literal, cast
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, File, HTTPException, Response, UploadFile
|
from fastapi import APIRouter, Depends, File, HTTPException, Response, UploadFile
|
||||||
from sqlalchemy import select
|
from sqlalchemy import select
|
||||||
@@ -57,7 +68,7 @@ from sqlalchemy.orm import Session
|
|||||||
from app.api.sync import _sanitize_error
|
from app.api.sync import _sanitize_error
|
||||||
from app.config import get_settings
|
from app.config import get_settings
|
||||||
from app.core.auth import require_admin
|
from app.core.auth import require_admin
|
||||||
from app.db import get_db
|
from app.db import SessionLocal, get_db
|
||||||
from app.models import GitSource
|
from app.models import GitSource
|
||||||
from app.rag.archive_upload import (
|
from app.rag.archive_upload import (
|
||||||
ARCHIVE_SUFFIXES,
|
ARCHIVE_SUFFIXES,
|
||||||
@@ -67,9 +78,15 @@ from app.rag.archive_upload import (
|
|||||||
unpack_archive,
|
unpack_archive,
|
||||||
)
|
)
|
||||||
from app.rag.importer import import_sources
|
from app.rag.importer import import_sources
|
||||||
from app.rag.llm import LLMClient, ModelUnavailableError, check_models
|
from app.rag.llm import LLMClient, check_models
|
||||||
from app.rag.overview import regenerate_overview
|
from app.rag.overview import regenerate_overview
|
||||||
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut, GitSourceRow, UploadOut
|
from app.schemas import (
|
||||||
|
GitSourceIn,
|
||||||
|
GitSourceList,
|
||||||
|
GitSourceOut,
|
||||||
|
GitSourceRow,
|
||||||
|
UploadAccepted,
|
||||||
|
)
|
||||||
|
|
||||||
logger = logging.getLogger("app.api.git_sources")
|
logger = logging.getLogger("app.api.git_sources")
|
||||||
|
|
||||||
@@ -79,18 +96,56 @@ router = APIRouter(
|
|||||||
dependencies=[Depends(require_admin)], # phase 16 pattern: admin-only surface
|
dependencies=[Depends(require_admin)], # phase 16 pattern: admin-only surface
|
||||||
)
|
)
|
||||||
|
|
||||||
#: One upload at a time (phase 49, task 02 — the phase-32 ``_task``
|
#: One upload at a time (phase 49, backgrounded in phase 64 task 03):
|
||||||
#: spirit): the flag is held from the name gate through the scan
|
#: the flag is checked and set with **no await in between, BEFORE the
|
||||||
#: response. A plain bool, not an ``asyncio.Lock`` — it is checked and
|
#: streaming receive** — the handler now awaits (the 1 MiB-chunk stream)
|
||||||
#: set with no await in between (a single app loop can never enter
|
#: long before the background task exists, so a task-done check alone
|
||||||
|
#: would let a concurrent POST slip through the receive window and start
|
||||||
|
#: a second run. A plain bool, not an ``asyncio.Lock``: it is checked
|
||||||
|
#: and set with no await in between (a single app loop can never enter
|
||||||
#: twice), and it stays correct across requests that run on separate
|
#: twice), and it stays correct across requests that run on separate
|
||||||
#: event loops (the TestClient convention).
|
#: event loops (the TestClient convention). Held until
|
||||||
|
#: :func:`_run_upload`'s ``finally`` (end of the background run) — or
|
||||||
|
#: cleared on the inline exception path where the task was never
|
||||||
|
#: created.
|
||||||
_upload_in_progress = False
|
_upload_in_progress = False
|
||||||
|
|
||||||
#: Streaming read size while counting compressed upload bytes (1 MiB
|
#: Streaming read size while counting compressed upload bytes (1 MiB
|
||||||
#: chunks — the task-02 cap check granularity).
|
#: chunks — the task-02 cap check granularity).
|
||||||
_STREAM_CHUNK = 1 << 20
|
_STREAM_CHUNK = 1 << 20
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class UploadStatus:
|
||||||
|
"""In-memory state of the (at most one) in-flight upload run.
|
||||||
|
|
||||||
|
Mirrors :class:`app.api.sync.SyncStatus` (the phase-32 pattern,
|
||||||
|
phase 64 task 03): ``state`` is the same four-state machine
|
||||||
|
(``idle`` / ``running`` / ``success`` / ``failed``); terminal states
|
||||||
|
carry the run's ``detail`` (success — the ``UploadOut`` fields) or
|
||||||
|
``error`` (failure — sanitized) so the UI can render the last result
|
||||||
|
after a page reload (the re-attach behavior, task 05).
|
||||||
|
|
||||||
|
Phase 64 (task 03) progress fields: ``current_file`` is the
|
||||||
|
``source/relative/path`` the scan is processing right now (null
|
||||||
|
outside the import phase — unpack/swap/row/model-check first — and
|
||||||
|
in terminal states); ``files_done`` / ``files_total`` carry the
|
||||||
|
hook's done/total position and survive a terminal state (the run's
|
||||||
|
last position is useful context next to the error).
|
||||||
|
"""
|
||||||
|
|
||||||
|
state: Literal["idle", "running", "success", "failed"] = "idle"
|
||||||
|
started_at: datetime | None = None
|
||||||
|
finished_at: datetime | None = None
|
||||||
|
current_file: str | None = None
|
||||||
|
files_done: int = 0
|
||||||
|
files_total: int = 0
|
||||||
|
detail: dict[str, Any] = field(default_factory=dict)
|
||||||
|
error: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
_upload_status = UploadStatus()
|
||||||
|
|
||||||
#: Accepted git URL shapes — the trimmed URL must *start* with one of them.
|
#: Accepted git URL shapes — the trimmed URL must *start* with one of them.
|
||||||
#: Covers the phase-28 real URLs (HTTPS + ``git@`` SSH); scp-style
|
#: Covers the phase-28 real URLs (HTTPS + ``git@`` SSH); scp-style
|
||||||
#: ``host:repo`` is deliberately rejected (422). ASSUMPTION (task 02): the
|
#: ``host:repo`` is deliberately rejected (422). ASSUMPTION (task 02): the
|
||||||
@@ -227,46 +282,55 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.post("/upload", response_model=UploadOut)
|
@router.post("/upload", response_model=UploadAccepted, status_code=202)
|
||||||
async def upload_archive(
|
async def upload_archive(
|
||||||
file: UploadFile = File(...), # noqa: B008
|
file: UploadFile = File(...), # noqa: B008
|
||||||
db: Session = Depends(get_db), # noqa: B008
|
) -> UploadAccepted:
|
||||||
) -> UploadOut:
|
"""Receive a source archive; scan it in the background (phase 49,
|
||||||
"""Upload a source archive and scan it (phase 49, task 02).
|
backgrounded in phase 64 task 03 — owner-locked A1/A2).
|
||||||
|
|
||||||
The scan is **synchronous in the request** (phase locked decisions,
|
The **inline (request) work is exactly three gates** — steps 1–3 —
|
||||||
owner-confirmed) and mirrors the admin sync pipeline:
|
everything else runs in a background task behind
|
||||||
|
``GET /upload/status`` (the phase-32 ``SyncStatus`` pattern), so
|
||||||
|
navigating away mid-scan no longer aborts anything:
|
||||||
|
|
||||||
1. name/format gate — only ``.tar``/``.tar.gz``/``.tgz``/``.zip``
|
1. name/format gate — only ``.tar``/``.tar.gz``/``.tgz``/``.zip``
|
||||||
(422 naming the accepted set) and a safe source name
|
(422 naming the accepted set) and a safe source name
|
||||||
(``archive_source_name`` — its message is the 422 detail);
|
(``archive_source_name`` — its message is the 422 detail);
|
||||||
2. one at a time — 409 ``an upload is already in progress``;
|
2. one at a time — 409 ``an upload is already in progress`` while
|
||||||
|
the flag is held (checked and set with no await in between,
|
||||||
|
BEFORE the receive — see ``_upload_in_progress``);
|
||||||
3. stream the upload in 1 MiB chunks into a dotfile temp with the
|
3. stream the upload in 1 MiB chunks into a dotfile temp with the
|
||||||
``upload_max_mb`` cap — 413 naming the cap, temp deleted;
|
``upload_max_mb`` cap — 413 naming the cap, temp deleted.
|
||||||
|
|
||||||
|
Then the archive is **safely on disk** — 202 + ``UploadAccepted``
|
||||||
|
(the "successfully uploaded" moment the UI toasts on, A2) and
|
||||||
|
:func:`_run_upload` runs the rest on the app's event loop:
|
||||||
|
|
||||||
4. unpack to a temp sibling (traversal/symlink/device/corrupt/
|
4. unpack to a temp sibling (traversal/symlink/device/corrupt/
|
||||||
over-cap all 422 with the task-01 user-safe message, temps
|
over-cap → ``failed`` with the task-01 user-safe message, temps
|
||||||
deleted); a zero-entry archive is 422 ``the archive contains no
|
deleted); a zero-entry archive is ``failed`` ``the archive
|
||||||
files`` — an archive with only non-A9 files is a VALID
|
contains no files`` — an archive with only non-A9 files is a
|
||||||
replacement (the scan indexes nothing, prune removes the
|
VALID replacement (the scan indexes nothing, prune removes the
|
||||||
source's docs);
|
source's docs);
|
||||||
5. atomic swap-in — a same-name re-upload replaces the previous
|
5. atomic swap-in — a same-name re-upload replaces the previous
|
||||||
folder in place; a failure leaves the previous folder/row/KB
|
folder in place; a failure leaves the previous folder/row/KB
|
||||||
untouched (422);
|
untouched;
|
||||||
6. upsert the row by ``path`` (``kind='local'``; an existing row is
|
6. upsert the row by ``path`` (``kind='local'``; an existing row is
|
||||||
left as-is — ``added_at`` preserved — and the unique index is
|
left as-is — ``added_at`` preserved — and the unique index is
|
||||||
the 409 backstop);
|
the backstop: a concurrent insert lands ``failed`` with
|
||||||
7. fail-fast ``check_models`` — 503 with the sanitized
|
``a local source with this path already exists: <path>``);
|
||||||
model-unavailable message; the folder/row are already committed,
|
7. fail-fast ``check_models`` — ``ModelUnavailableError`` →
|
||||||
so the next sync/re-upload retries idempotently;
|
``failed`` with the sanitized message (the phase-49 503 becomes
|
||||||
8. ``import_sources([folder], llm, prune=True)`` + the change-gated
|
a status state, A5); the folder/row are already committed, so
|
||||||
``regenerate_overview``;
|
the next sync/re-upload retries idempotently;
|
||||||
9. one INFO log line (PLAN §9 / AGENTS.md rule 10);
|
8. ``import_sources([folder], llm, prune=True, progress=<hook>)``
|
||||||
10. 200 with the sync-detail count keys (``UploadOut``).
|
+ the change-gated ``regenerate_overview`` — the hook feeds the
|
||||||
|
status ``current_file`` / ``files_done`` / ``files_total``;
|
||||||
|
9. one INFO log line (PLAN §9 / AGENTS.md rule 10 — ``total_ms`` is
|
||||||
|
the background run's duration);
|
||||||
|
10. ``success`` — ``detail`` = the ``UploadOut`` fields.
|
||||||
"""
|
"""
|
||||||
started = time.monotonic()
|
|
||||||
settings = get_settings()
|
|
||||||
total = 0
|
|
||||||
|
|
||||||
# 1. Name/format gate — the accepted formats first (the 422 names
|
# 1. Name/format gate — the accepted formats first (the 422 names
|
||||||
# them), then the task-01 safe-name derivation. A BARE suffix
|
# them), then the task-01 safe-name derivation. A BARE suffix
|
||||||
# ("tar.gz") is an accepted format with no usable stem — it
|
# ("tar.gz") is an accepted format with no usable stem — it
|
||||||
@@ -288,12 +352,15 @@ async def upload_archive(
|
|||||||
raise HTTPException(status_code=422, detail=str(e)) from None
|
raise HTTPException(status_code=422, detail=str(e)) from None
|
||||||
|
|
||||||
# 2. One at a time — the flag is checked and set with no await
|
# 2. One at a time — the flag is checked and set with no await
|
||||||
# between, so the single app loop can never enter twice.
|
# between, BEFORE the streaming receive: the background task
|
||||||
|
# does not exist yet, so the flag (not a task-done check) is the
|
||||||
|
# gate (see ``_upload_in_progress``).
|
||||||
global _upload_in_progress
|
global _upload_in_progress
|
||||||
if _upload_in_progress:
|
if _upload_in_progress:
|
||||||
raise HTTPException(status_code=409, detail="an upload is already in progress")
|
raise HTTPException(status_code=409, detail="an upload is already in progress")
|
||||||
_upload_in_progress = True
|
_upload_in_progress = True
|
||||||
|
|
||||||
|
settings = get_settings()
|
||||||
upload_root = Path(settings.upload_dir).expanduser()
|
upload_root = Path(settings.upload_dir).expanduser()
|
||||||
upload_root.mkdir(parents=True, exist_ok=True)
|
upload_root.mkdir(parents=True, exist_ok=True)
|
||||||
max_bytes = settings.upload_max_mb * 1024 * 1024
|
max_bytes = settings.upload_max_mb * 1024 * 1024
|
||||||
@@ -302,77 +369,192 @@ async def upload_archive(
|
|||||||
try:
|
try:
|
||||||
# 3. Stream with the compressed-size cap — dotfile temps are
|
# 3. Stream with the compressed-size cap — dotfile temps are
|
||||||
# hidden from the upload dir's listing.
|
# hidden from the upload dir's listing.
|
||||||
try:
|
total = 0
|
||||||
with open(temp_upload, "wb") as out:
|
with open(temp_upload, "wb") as out:
|
||||||
while chunk := await file.read(_STREAM_CHUNK):
|
while chunk := await file.read(_STREAM_CHUNK):
|
||||||
total += len(chunk)
|
total += len(chunk)
|
||||||
if total > max_bytes:
|
if total > max_bytes:
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=413,
|
status_code=413,
|
||||||
detail=f"the upload exceeds the {settings.upload_max_mb} MiB limit",
|
detail=f"the upload exceeds the {settings.upload_max_mb} MiB limit",
|
||||||
)
|
)
|
||||||
out.write(chunk)
|
out.write(chunk)
|
||||||
except HTTPException:
|
# The archive is safely on disk — 202 is the "successfully
|
||||||
temp_upload.unlink(missing_ok=True)
|
# uploaded" moment (A2). Steps 4–10 run in the background:
|
||||||
raise
|
asyncio.create_task(
|
||||||
# 4. Unpack to a temp sibling; the compressed bytes are no
|
_run_upload(name, filename, total, upload_root, temp_upload, temp_unpack)
|
||||||
# longer needed once unpacked (phase locked decision: only
|
)
|
||||||
# the unpacked content is kept).
|
except BaseException:
|
||||||
try:
|
# The background task was never created (cap 413, a broken
|
||||||
unpack_archive(temp_upload, temp_unpack, max_bytes)
|
# pipe, cancellation, or create_task itself): release the flag
|
||||||
except ArchiveUploadError as e:
|
# so the next upload is not refused, and make sure no temp
|
||||||
temp_upload.unlink(missing_ok=True)
|
# survives the failed receive.
|
||||||
shutil.rmtree(temp_unpack, ignore_errors=True)
|
temp_upload.unlink(missing_ok=True)
|
||||||
raise HTTPException(status_code=422, detail=str(e)) from None
|
_upload_in_progress = False
|
||||||
|
raise
|
||||||
|
return UploadAccepted(name=name)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/upload/status")
|
||||||
|
def upload_status() -> dict[str, Any]:
|
||||||
|
"""Current upload state (the UI polls this — the phase-32
|
||||||
|
``GET /api/sync/status`` contract, identical key set).
|
||||||
|
|
||||||
|
``started_at`` / ``finished_at`` are ISO-8601 strings or null.
|
||||||
|
``current_file`` (phase 64) is the ``source/relative/path`` the
|
||||||
|
scan is processing right now — null during the unpack/swap/row/
|
||||||
|
model phases and in terminal states; ``files_done`` / ``files_total``
|
||||||
|
carry the hook's position (0/0 idle). The router dependency makes
|
||||||
|
it admin-only like every other route here.
|
||||||
|
"""
|
||||||
|
return {
|
||||||
|
"state": _upload_status.state,
|
||||||
|
"started_at": (
|
||||||
|
_upload_status.started_at.isoformat() if _upload_status.started_at else None
|
||||||
|
),
|
||||||
|
"finished_at": (
|
||||||
|
_upload_status.finished_at.isoformat() if _upload_status.finished_at else None
|
||||||
|
),
|
||||||
|
"detail": _upload_status.detail,
|
||||||
|
"error": _upload_status.error,
|
||||||
|
"current_file": _upload_status.current_file,
|
||||||
|
"files_done": _upload_status.files_done,
|
||||||
|
"files_total": _upload_status.files_total,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
async def _run_upload(
|
||||||
|
name: str,
|
||||||
|
filename: str,
|
||||||
|
total_bytes: int,
|
||||||
|
upload_root: Path,
|
||||||
|
temp_upload: Path,
|
||||||
|
temp_unpack: Path,
|
||||||
|
) -> None:
|
||||||
|
"""The post-202 upload pipeline, one in-process background task
|
||||||
|
(the phase-32 ``_run_sync`` shape — A1).
|
||||||
|
|
||||||
|
Every failure mode (unpack, zero entries, swap, row, models,
|
||||||
|
import, anything else) lands in the ``failed`` state with a
|
||||||
|
sanitized ``error`` string — a background task must die in state,
|
||||||
|
never as an unobserved exception (A5: post-202 failures are status
|
||||||
|
states, never HTTP errors). ``CancelledError`` is deliberately *not*
|
||||||
|
caught: app shutdown cancels the task, and swallowing that would
|
||||||
|
mask a real stop. The ``finally`` cleans both temps (defensive —
|
||||||
|
each step already cleans its own) and clears ``_upload_in_progress``.
|
||||||
|
"""
|
||||||
|
global _upload_in_progress
|
||||||
|
started = time.monotonic()
|
||||||
|
_upload_status.state = "running"
|
||||||
|
_upload_status.started_at = datetime.now(UTC)
|
||||||
|
_upload_status.finished_at = None
|
||||||
|
_upload_status.current_file = None
|
||||||
|
_upload_status.files_done = 0
|
||||||
|
_upload_status.files_total = 0
|
||||||
|
_upload_status.detail = {}
|
||||||
|
_upload_status.error = None
|
||||||
|
try:
|
||||||
|
settings = get_settings()
|
||||||
|
max_bytes = settings.upload_max_mb * 1024 * 1024
|
||||||
|
# Step 4 — unpack to a temp sibling; the compressed bytes are
|
||||||
|
# no longer needed once unpacked (phase 49 locked decision:
|
||||||
|
# only the unpacked content is kept).
|
||||||
|
unpack_archive(temp_upload, temp_unpack, max_bytes)
|
||||||
temp_upload.unlink(missing_ok=True)
|
temp_upload.unlink(missing_ok=True)
|
||||||
if not any(temp_unpack.iterdir()):
|
if not any(temp_unpack.iterdir()):
|
||||||
# Zero entries = a user error. (Only non-A9 files is NOT an
|
# Zero entries = a user error. (Only non-A9 files is NOT an
|
||||||
# error — it still has entries and is a valid replacement.)
|
# error — it still has entries and is a valid replacement.)
|
||||||
shutil.rmtree(temp_unpack, ignore_errors=True)
|
raise ArchiveUploadError("the archive contains no files")
|
||||||
raise HTTPException(status_code=422, detail="the archive contains no files")
|
# Step 5 — swap in — a same-name re-upload replaces the
|
||||||
# 5. Swap in — a same-name re-upload replaces the previous
|
# previous folder atomically; a failure leaves it, the row,
|
||||||
# folder atomically; a failure leaves it, the row, and the
|
# and the KB untouched (the ``failed`` state carries the
|
||||||
# KB untouched.
|
# user-safe message).
|
||||||
final_dir = upload_root / name
|
final_dir = upload_root / name
|
||||||
try:
|
swap_in(temp_unpack, final_dir)
|
||||||
swap_in(temp_unpack, final_dir)
|
# Step 6 — upsert the row by path in a SHORT-LIVED session
|
||||||
except ArchiveUploadError as e:
|
# (open/close around it — the ``effective_sources`` /
|
||||||
shutil.rmtree(temp_unpack, ignore_errors=True)
|
# ``bump_sources_version`` pattern in ``app.api.sync``): the
|
||||||
raise HTTPException(status_code=422, detail=str(e)) from None
|
# background task has no request session to leak locks from
|
||||||
# 6. Upsert the row by path — no duplicates: an existing row is
|
# (the old inline ``db.close()`` discipline, now structural).
|
||||||
# left exactly as it is (``added_at`` preserved); the unique
|
# No duplicates: an existing row is left exactly as it is
|
||||||
# index is the 409 backstop for a concurrent insert the
|
# (``added_at`` preserved); the unique index is the backstop
|
||||||
# pre-check missed.
|
# for a concurrent insert the pre-check missed.
|
||||||
path = str(final_dir)
|
path = str(final_dir)
|
||||||
if db.scalar(select(GitSource).where(GitSource.path == path)) is None:
|
db = SessionLocal()
|
||||||
_commit_new(
|
|
||||||
GitSource(url=path, kind="local", path=path),
|
|
||||||
f"a local source with this path already exists: {path}",
|
|
||||||
db,
|
|
||||||
)
|
|
||||||
# Release the request session NOW — the handler never touches
|
|
||||||
# ``db`` again (the scan below uses its own sessions). If the
|
|
||||||
# session stayed open, its uncommitted transaction (the
|
|
||||||
# ``_commit_new`` refresh SELECT) would hold ``git_sources``
|
|
||||||
# locks for the whole scan, and any concurrent TRUNCATE of the
|
|
||||||
# KB tables (the E2E isolation fixtures) would deadlock against
|
|
||||||
# the scan's own document locks — a cycle Postgres cannot see.
|
|
||||||
# ``get_db``'s teardown close() is idempotent.
|
|
||||||
db.close()
|
|
||||||
# 7. Fail-fast models (phase 41) — 503 with the sanitized
|
|
||||||
# message; nothing else is rolled back (the folder/row are
|
|
||||||
# committed and the next sync/re-upload retries idempotently).
|
|
||||||
llm = LLMClient()
|
|
||||||
try:
|
try:
|
||||||
await check_models(llm)
|
if db.scalar(select(GitSource).where(GitSource.path == path)) is None:
|
||||||
except ModelUnavailableError as e:
|
db.add(GitSource(url=path, kind="local", path=path))
|
||||||
raise HTTPException(status_code=503, detail=_sanitize_error(str(e))) from None
|
try:
|
||||||
# 8. Scan — single source, prune (dropped files leave the KB),
|
db.commit()
|
||||||
# then the change-gated overview refresh (phases 31/32).
|
except IntegrityError:
|
||||||
summary = await import_sources([final_dir], llm, prune=True)
|
db.rollback()
|
||||||
|
raise ValueError(
|
||||||
|
f"a local source with this path already exists: {path}"
|
||||||
|
) from None
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
# Step 7 — fail-fast models (phase 41): ``ModelUnavailableError``
|
||||||
|
# lands in the ``failed`` state sanitized (the phase-49 503
|
||||||
|
# becomes a status state, A5). Nothing is rolled back — the
|
||||||
|
# folder/row are committed and the next sync/re-upload retries
|
||||||
|
# idempotently.
|
||||||
|
llm = LLMClient()
|
||||||
|
await check_models(llm)
|
||||||
|
# Step 8 — scan — single source, prune (dropped files leave
|
||||||
|
# the KB), with the phase-64 progress hook feeding the status,
|
||||||
|
# then the change-gated overview refresh (phases 31/32). The
|
||||||
|
# closure captures the module ``_upload_status`` exactly like
|
||||||
|
# the state assignments above.
|
||||||
|
def _hook(source: str, rel: str, done: int, total: int) -> None:
|
||||||
|
_upload_status.current_file = f"{source}/{rel}"
|
||||||
|
_upload_status.files_done = done
|
||||||
|
_upload_status.files_total = total
|
||||||
|
|
||||||
|
summary = await import_sources([final_dir], llm, prune=True, progress=_hook)
|
||||||
overview = False
|
overview = False
|
||||||
if summary.added + summary.updated > 0:
|
if summary.added + summary.updated > 0:
|
||||||
overview = await regenerate_overview(llm)
|
overview = await regenerate_overview(llm)
|
||||||
|
# Step 9 — per-upload log line (PLAN §9 / AGENTS.md rule 10)
|
||||||
|
# — moved with the scan: ``total_ms`` is the background run's
|
||||||
|
# duration.
|
||||||
|
logger.info(
|
||||||
|
"upload: name=%s file=%s bytes_in=%d files=%d added=%d updated=%d "
|
||||||
|
"unchanged=%d pruned=%d errors=%d overview=%s total_ms=%d",
|
||||||
|
name,
|
||||||
|
filename,
|
||||||
|
total_bytes,
|
||||||
|
summary.files,
|
||||||
|
summary.added,
|
||||||
|
summary.updated,
|
||||||
|
summary.unchanged,
|
||||||
|
summary.pruned,
|
||||||
|
summary.errors,
|
||||||
|
overview,
|
||||||
|
round((time.monotonic() - started) * 1000),
|
||||||
|
)
|
||||||
|
# Step 10 — success: the ``UploadOut`` fields ride in the
|
||||||
|
# status ``detail`` (the UI renders the same result line from
|
||||||
|
# the status that the sync button renders from its own).
|
||||||
|
_upload_status.state = "success"
|
||||||
|
_upload_status.finished_at = datetime.now(UTC)
|
||||||
|
_upload_status.current_file = None # phase 64: keep the final counts
|
||||||
|
_upload_status.detail = {
|
||||||
|
"source": name,
|
||||||
|
"files": summary.files,
|
||||||
|
"added": summary.added,
|
||||||
|
"updated": summary.updated,
|
||||||
|
"unchanged": summary.unchanged,
|
||||||
|
"pruned": summary.pruned,
|
||||||
|
"errors": summary.errors,
|
||||||
|
"chunks": summary.chunks,
|
||||||
|
"overview": overview,
|
||||||
|
}
|
||||||
|
except Exception as e: # noqa: BLE001 — a background task dies in state, see above
|
||||||
|
logger.exception("upload: failed")
|
||||||
|
_upload_status.state = "failed"
|
||||||
|
_upload_status.finished_at = datetime.now(UTC)
|
||||||
|
_upload_status.error = _sanitize_error(str(e))
|
||||||
|
_upload_status.current_file = None # phase 64: keep the final counts
|
||||||
finally:
|
finally:
|
||||||
_upload_in_progress = False
|
_upload_in_progress = False
|
||||||
# No temp may survive any failure path (defensive — each step
|
# No temp may survive any failure path (defensive — each step
|
||||||
@@ -380,35 +562,6 @@ async def upload_archive(
|
|||||||
temp_upload.unlink(missing_ok=True)
|
temp_upload.unlink(missing_ok=True)
|
||||||
shutil.rmtree(temp_unpack, ignore_errors=True)
|
shutil.rmtree(temp_unpack, ignore_errors=True)
|
||||||
|
|
||||||
# 9. Per-upload log line (PLAN §9 / AGENTS.md rule 10).
|
|
||||||
logger.info(
|
|
||||||
"upload: name=%s file=%s bytes_in=%d files=%d added=%d updated=%d "
|
|
||||||
"unchanged=%d pruned=%d errors=%d overview=%s total_ms=%d",
|
|
||||||
name,
|
|
||||||
filename,
|
|
||||||
total,
|
|
||||||
summary.files,
|
|
||||||
summary.added,
|
|
||||||
summary.updated,
|
|
||||||
summary.unchanged,
|
|
||||||
summary.pruned,
|
|
||||||
summary.errors,
|
|
||||||
overview,
|
|
||||||
round((time.monotonic() - started) * 1000),
|
|
||||||
)
|
|
||||||
# 10. Respond 200 with the sync-style counts.
|
|
||||||
return UploadOut(
|
|
||||||
source=name,
|
|
||||||
files=summary.files,
|
|
||||||
added=summary.added,
|
|
||||||
updated=summary.updated,
|
|
||||||
unchanged=summary.unchanged,
|
|
||||||
pruned=summary.pruned,
|
|
||||||
errors=summary.errors,
|
|
||||||
chunks=summary.chunks,
|
|
||||||
overview=overview,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/{source_id}", status_code=204)
|
@router.delete("/{source_id}", status_code=204)
|
||||||
def delete_git_source(
|
def delete_git_source(
|
||||||
|
|||||||
+42
-2
@@ -51,7 +51,12 @@ decisions):
|
|||||||
run aborts in the ``failed`` state before this step.
|
run aborts in the ``failed`` state before this step.
|
||||||
|
|
||||||
Status is in memory: a restart mid-sync loses the running state
|
Status is in memory: a restart mid-sync loses the running state
|
||||||
(accepted — the next click re-syncs idempotently).
|
(accepted — the next click re-syncs idempotently). The status also
|
||||||
|
carries the phase-64 per-file progress — ``current_file`` (the
|
||||||
|
``source/relative/path`` the import is processing right now) plus
|
||||||
|
``files_done`` / ``files_total`` — null/0/0 before the import starts
|
||||||
|
(clone/pull reports no file yet) and in terminal states, which clear
|
||||||
|
``current_file`` but keep the run's final counts.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -107,6 +112,13 @@ class SyncStatus:
|
|||||||
``running``, ``success``, ``failed``. Terminal states carry the run's
|
``running``, ``success``, ``failed``. Terminal states carry the run's
|
||||||
``detail`` (success) or ``error`` (failure) so the UI can render the
|
``detail`` (success) or ``error`` (failure) so the UI can render the
|
||||||
last result after a page reload (task 02's re-attach behavior).
|
last result after a page reload (task 02's re-attach behavior).
|
||||||
|
|
||||||
|
Phase 64 (task 02) progress fields: ``current_file`` is the
|
||||||
|
``source/relative/path`` the import is processing right now (null
|
||||||
|
outside the import phase — clone/pull first, terminal states
|
||||||
|
after); ``files_done`` / ``files_total`` carry the hook's
|
||||||
|
done/total position and survive a terminal state (the run's last
|
||||||
|
position is useful context next to the error).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
state: Literal["idle", "running", "success", "failed"] = "idle"
|
state: Literal["idle", "running", "success", "failed"] = "idle"
|
||||||
@@ -114,6 +126,11 @@ class SyncStatus:
|
|||||||
finished_at: datetime | None = None
|
finished_at: datetime | None = None
|
||||||
detail: dict[str, Any] = field(default_factory=dict)
|
detail: dict[str, Any] = field(default_factory=dict)
|
||||||
error: str | None = None
|
error: str | None = None
|
||||||
|
# Phase 64 (task 02): per-file progress — the file the import is
|
||||||
|
# processing right now and the hook's done/total position.
|
||||||
|
current_file: str | None = None
|
||||||
|
files_done: int = 0
|
||||||
|
files_total: int = 0
|
||||||
|
|
||||||
|
|
||||||
_status = SyncStatus()
|
_status = SyncStatus()
|
||||||
@@ -125,6 +142,10 @@ def sync_status() -> dict[str, Any]:
|
|||||||
"""Current sync state (the UI polls this every 2 s — task 02).
|
"""Current sync state (the UI polls this every 2 s — task 02).
|
||||||
|
|
||||||
``started_at`` / ``finished_at`` are ISO-8601 strings or null.
|
``started_at`` / ``finished_at`` are ISO-8601 strings or null.
|
||||||
|
``current_file`` (phase 64) is the ``source/relative/path`` the
|
||||||
|
import is processing right now — null during the clone/pull phase
|
||||||
|
and in terminal states; ``files_done`` / ``files_total`` carry the
|
||||||
|
hook's position (0/0 idle).
|
||||||
"""
|
"""
|
||||||
return {
|
return {
|
||||||
"state": _status.state,
|
"state": _status.state,
|
||||||
@@ -132,6 +153,9 @@ def sync_status() -> dict[str, Any]:
|
|||||||
"finished_at": _status.finished_at.isoformat() if _status.finished_at else None,
|
"finished_at": _status.finished_at.isoformat() if _status.finished_at else None,
|
||||||
"detail": _status.detail,
|
"detail": _status.detail,
|
||||||
"error": _status.error,
|
"error": _status.error,
|
||||||
|
"current_file": _status.current_file,
|
||||||
|
"files_done": _status.files_done,
|
||||||
|
"files_total": _status.files_total,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -165,6 +189,11 @@ async def _run_sync() -> None:
|
|||||||
_status.finished_at = None
|
_status.finished_at = None
|
||||||
_status.detail = {}
|
_status.detail = {}
|
||||||
_status.error = None
|
_status.error = None
|
||||||
|
# Phase 64 (task 02): the progress fields reset with the run — no
|
||||||
|
# current file until the import starts (the clone/pull phase).
|
||||||
|
_status.current_file = None
|
||||||
|
_status.files_done = 0
|
||||||
|
_status.files_total = 0
|
||||||
try:
|
try:
|
||||||
settings = get_settings()
|
settings = get_settings()
|
||||||
# Step 1 (phase 41): fail fast — verify both models the sync
|
# Step 1 (phase 41): fail fast — verify both models the sync
|
||||||
@@ -208,7 +237,16 @@ async def _run_sync() -> None:
|
|||||||
if not path.is_dir():
|
if not path.is_dir():
|
||||||
raise GitSyncError(f"local source missing: {path}")
|
raise GitSyncError(f"local source missing: {path}")
|
||||||
sources.append(path)
|
sources.append(path)
|
||||||
summary: ImportSummary = await import_sources(sources, llm, prune=True)
|
# Phase 64 (task 02): the per-file progress hook — the status
|
||||||
|
# endpoint reports the file being processed right now. The
|
||||||
|
# closure captures the module ``_status`` exactly like the state
|
||||||
|
# assignments above.
|
||||||
|
def _hook(source: str, rel: str, done: int, total: int) -> None:
|
||||||
|
_status.current_file = f"{source}/{rel}"
|
||||||
|
_status.files_done = done
|
||||||
|
_status.files_total = total
|
||||||
|
|
||||||
|
summary: ImportSummary = await import_sources(sources, llm, prune=True, progress=_hook)
|
||||||
overview = False
|
overview = False
|
||||||
if summary.added + summary.updated > 0:
|
if summary.added + summary.updated > 0:
|
||||||
overview = await regenerate_overview(llm)
|
overview = await regenerate_overview(llm)
|
||||||
@@ -234,6 +272,7 @@ async def _run_sync() -> None:
|
|||||||
db.close()
|
db.close()
|
||||||
_status.state = "success"
|
_status.state = "success"
|
||||||
_status.finished_at = datetime.now(UTC)
|
_status.finished_at = datetime.now(UTC)
|
||||||
|
_status.current_file = None # phase 64: keep the final counts
|
||||||
_status.detail = {
|
_status.detail = {
|
||||||
"files": summary.files,
|
"files": summary.files,
|
||||||
"added": summary.added,
|
"added": summary.added,
|
||||||
@@ -253,3 +292,4 @@ async def _run_sync() -> None:
|
|||||||
_status.state = "failed"
|
_status.state = "failed"
|
||||||
_status.finished_at = datetime.now(UTC)
|
_status.finished_at = datetime.now(UTC)
|
||||||
_status.error = _sanitize_error(str(e))
|
_status.error = _sanitize_error(str(e))
|
||||||
|
_status.current_file = None # phase 64: keep the final counts
|
||||||
|
|||||||
@@ -26,11 +26,15 @@ no longer exist **or no longer match the format filter** — this is how
|
|||||||
previously-imported junk (e.g. dot-dir READMEs) leaves the index. Per-file
|
previously-imported junk (e.g. dot-dir READMEs) leaves the index. Per-file
|
||||||
logging uses the verbs ``added | updated | unchanged | pruned`` plus a
|
logging uses the verbs ``added | updated | unchanged | pruned`` plus a
|
||||||
summary line with per-format counts (PLAN §9).
|
summary line with per-format counts (PLAN §9).
|
||||||
|
|
||||||
|
``import_sources`` accepts an optional per-file ``progress`` callback
|
||||||
|
(phase 64, task 01) reporting the file being processed right now.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import hashlib
|
import hashlib
|
||||||
import logging
|
import logging
|
||||||
|
from collections.abc import Callable
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -148,12 +152,26 @@ async def import_sources(
|
|||||||
prune: bool = False,
|
prune: bool = False,
|
||||||
limit: int | None = None,
|
limit: int | None = None,
|
||||||
session: Session | None = None,
|
session: Session | None = None,
|
||||||
|
progress: Callable[[str, str, int, int], None] | None = None,
|
||||||
) -> ImportSummary:
|
) -> ImportSummary:
|
||||||
"""Import every A9-format file under *sources* (see module docstring).
|
"""Import every A9-format file under *sources* (see module docstring).
|
||||||
|
|
||||||
``session`` may be supplied (tests); a private one is opened and closed
|
``session`` may be supplied (tests); a private one is opened and closed
|
||||||
otherwise. ``limit`` caps the number of files processed (debug only) and
|
otherwise. ``limit`` caps the number of files processed (debug only) and
|
||||||
disables pruning, since an incomplete walk must not drive deletions.
|
disables pruning, since an incomplete walk must not drive deletions.
|
||||||
|
|
||||||
|
``progress`` (phase 64, task 01) is an optional per-file hook called
|
||||||
|
once per importable file, immediately before that file's
|
||||||
|
``_index_file`` — with ``(source, rel_posix_path, done, total)``: the
|
||||||
|
same POSIX *rel* the document rows use, ``done`` = the 1-based index of
|
||||||
|
the current file **across all sources**, and ``total`` = the combined
|
||||||
|
pre-walk count of importable files across all *sources* roots. The
|
||||||
|
pre-walk (same extension/exclusion rules, directory stats only, no file
|
||||||
|
reads) happens **only when *progress* is provided**: callers passing
|
||||||
|
nothing pay no extra walk and behave exactly as before. Under
|
||||||
|
``limit``, the hook still fires per processed file only — ``done`` never
|
||||||
|
exceeds the limit, but ``total`` stays the full pre-walk count (an
|
||||||
|
incomplete walk must not misreport the denominator).
|
||||||
"""
|
"""
|
||||||
if limit is not None and limit <= 0:
|
if limit is not None and limit <= 0:
|
||||||
raise ValueError("limit must be >= 1")
|
raise ValueError("limit must be >= 1")
|
||||||
@@ -163,6 +181,14 @@ async def import_sources(
|
|||||||
session = SessionLocal()
|
session = SessionLocal()
|
||||||
seen: set[tuple[str, str]] = set()
|
seen: set[tuple[str, str]] = set()
|
||||||
source_names: set[str] = set()
|
source_names: set[str] = set()
|
||||||
|
# phase 64 (task 01): the hook's combined denominator, walked with the
|
||||||
|
# exact same rules as the processing loop below (directory stats only,
|
||||||
|
# no file reads). Skipped entirely for ``progress=None`` callers — no
|
||||||
|
# extra pass, byte-identical behaviour and cost.
|
||||||
|
total = 0
|
||||||
|
if progress is not None:
|
||||||
|
for root in sources:
|
||||||
|
total += len(iter_importable_files(root, llm.settings.import_extension_set))
|
||||||
try:
|
try:
|
||||||
for root in sources:
|
for root in sources:
|
||||||
if not root.is_dir():
|
if not root.is_dir():
|
||||||
@@ -180,6 +206,12 @@ async def import_sources(
|
|||||||
summary.files += 1
|
summary.files += 1
|
||||||
ext = path.suffix.lower().lstrip(".") or "unknown"
|
ext = path.suffix.lower().lstrip(".") or "unknown"
|
||||||
summary.formats[ext] = summary.formats.get(ext, 0) + 1
|
summary.formats[ext] = summary.formats.get(ext, 0) + 1
|
||||||
|
if progress is not None:
|
||||||
|
# phase 64: report the file *before* indexing it — a
|
||||||
|
# file that then errors or turns out unchanged was
|
||||||
|
# already the "current file". No try/except around the
|
||||||
|
# call: the hooks in this repo only assign fields.
|
||||||
|
progress(source, rel, summary.files, total)
|
||||||
try:
|
try:
|
||||||
await _index_file(
|
await _index_file(
|
||||||
session, source=source, rel=rel, full_path=path, llm=llm,
|
session, source=source, rel=rel, full_path=path, llm=llm,
|
||||||
|
|||||||
+20
-2
@@ -304,9 +304,12 @@ class GitSourceList(BaseModel):
|
|||||||
|
|
||||||
|
|
||||||
class UploadOut(BaseModel):
|
class UploadOut(BaseModel):
|
||||||
"""``POST /api/git-sources/upload`` response (phase 49, task 02).
|
"""The upload run's result fields (phase 49, task 02; phase 64, task 03).
|
||||||
|
|
||||||
The uploaded source's name (filename minus the archive suffix) plus
|
Phase 64 (task 03): ``POST /api/git-sources/upload`` answers 202 the
|
||||||
|
moment the archive is on disk; these fields become the shape of
|
||||||
|
``GET /api/git-sources/upload/status`` ``detail`` on ``success`` —
|
||||||
|
the uploaded source's name (filename minus the archive suffix) plus
|
||||||
the SAME count keys as the admin sync's success ``detail``
|
the SAME count keys as the admin sync's success ``detail``
|
||||||
(``files``, ``added``, ``updated``, ``unchanged``, ``pruned``,
|
(``files``, ``added``, ``updated``, ``unchanged``, ``pruned``,
|
||||||
``errors``, ``chunks`` — ``app.api.sync._run_sync``) and the
|
``errors``, ``chunks`` — ``app.api.sync._run_sync``) and the
|
||||||
@@ -325,6 +328,21 @@ class UploadOut(BaseModel):
|
|||||||
overview: bool
|
overview: bool
|
||||||
|
|
||||||
|
|
||||||
|
class UploadAccepted(BaseModel):
|
||||||
|
"""``POST /api/git-sources/upload`` 202 response (phase 64, task 03).
|
||||||
|
|
||||||
|
The archive is **safely on disk** — this is the "successfully
|
||||||
|
uploaded" moment the Sources page toasts on (owner-locked A2). The
|
||||||
|
scan itself (unpack → swap → row upsert → model check → import →
|
||||||
|
overview) runs in a background task behind
|
||||||
|
``GET /api/git-sources/upload/status``, whose ``success`` ``detail``
|
||||||
|
carries the :class:`UploadOut` fields.
|
||||||
|
"""
|
||||||
|
|
||||||
|
detail: str = "upload received"
|
||||||
|
name: str
|
||||||
|
|
||||||
|
|
||||||
class ToolCall(BaseModel):
|
class ToolCall(BaseModel):
|
||||||
"""One agent tool-call record (the phase-37 ``tools`` record shape).
|
"""One agent tool-call record (the phase-37 ``tools`` record shape).
|
||||||
|
|
||||||
|
|||||||
+275
-41
@@ -34,20 +34,38 @@
|
|||||||
* shape-aware like the tuning forms) and keeps the input — the
|
* shape-aware like the tuning forms) and keeps the input — the
|
||||||
* instruction survives. 409/422 details are fixed generic strings
|
* instruction survives. 409/422 details are fixed generic strings
|
||||||
* (credential safety — the URL is never echoed).
|
* (credential safety — the URL is never echoed).
|
||||||
* • upload — #archive-upload-form submit (phase 49) → POST
|
* • upload — #archive-upload-form submit (phase 49, reworked to the
|
||||||
|
* phase-64 202 contract in task 05 — the phase-49 synchronous
|
||||||
|
* 200 paragraph is superseded): POST
|
||||||
* /api/git-sources/upload with a FormData file (NO manual
|
* /api/git-sources/upload with a FormData file (NO manual
|
||||||
* Content-Type — the browser sets the multipart boundary). The
|
* Content-Type — the browser sets the multipart boundary). The
|
||||||
* SAME §7.4 never-stale lifecycle: the button disables +
|
* §7.4 never-stale lifecycle keeps its shape — the button
|
||||||
* relabels "Uploading…" while the request is out and is restored
|
* disables + relabels "Uploading…" while the request is out —
|
||||||
* on success AND failure. 200 clears the file input, shows the
|
* but the transfer is now short: the 202 arrives the moment the
|
||||||
* sync-style count line ("2 added · 1 pruned" — fmtUploadResult,
|
* archive is safely on disk (A1). 202 → the page-local
|
||||||
* sources.js's fmtSyncResult convention) in the role=status
|
* "Successfully uploaded — <file>" toast fires (showUploadToast,
|
||||||
* result line, announces "Archive uploaded: …" and reloads the
|
* the phase-55 share-toast pattern; A2: safe to navigate away),
|
||||||
* list (the new/updated row lands with the Local badge; a
|
* the file input clears, and the button hands over to the scan —
|
||||||
* re-upload simply refreshes the row — no duplicate). Non-2xx
|
* the processing state ("Processing…", disabled, title cleared)
|
||||||
* inlines the server detail (422 format/name/traversal, 413
|
* plus startUploadPolling(): a 2 s poll of
|
||||||
* size, 409 busy — the messages are already user-safe) and KEEPS
|
* GET /api/git-sources/upload/status renders the live
|
||||||
* the file selection — the fix is one re-pick, not a re-type.
|
* "Processing… <file> (n/m)" label (A4 — bare during unpack; the
|
||||||
|
* full path rides the button title) and settles it: success →
|
||||||
|
* the sync-style count line (fmtUploadResult, the role=status
|
||||||
|
* result line) + the "Archive uploaded: …" announce +
|
||||||
|
* loadSources (the new/updated row lands with the Local badge;
|
||||||
|
* a re-upload refreshes the row — no duplicate; NO second toast
|
||||||
|
* — A2); failure → the sanitized server error in the role=alert
|
||||||
|
* banner + loadSources, the file selection KEPT for a one-click
|
||||||
|
* re-upload. 409 (an upload is already in progress) raises NO
|
||||||
|
* error banner — it re-attaches to the in-flight run (processing
|
||||||
|
* state + poll, never stale). Other non-2xx (422 format/name,
|
||||||
|
* 413 cap, 5xx) keep the phase-49 error banner + the kept file
|
||||||
|
* selection. The submit finally restores the button ONLY when no
|
||||||
|
* poll is active (§7.4). Boot re-attach (initUploadStatus, admin
|
||||||
|
* branch): a running scan re-enters the processing state + poll
|
||||||
|
* (a reload mid-scan re-attaches — no second upload), a terminal
|
||||||
|
* run re-renders its result line / error banner.
|
||||||
* • remove — a row's Remove button asks window.confirm first
|
* • remove — a row's Remove button asks window.confirm first
|
||||||
* (removal prunes the documents only on the NEXT sync — the
|
* (removal prunes the documents only on the NEXT sync — the
|
||||||
* confirm says so). Cancel → nothing; ok → the row button
|
* confirm says so). Cancel → nothing; ok → the row button
|
||||||
@@ -64,8 +82,8 @@
|
|||||||
* removing a source does NOT clone, import, or prune — the sync
|
* removing a source does NOT clone, import, or prune — the sync
|
||||||
* service (server-side) performs that; the page's hint box says so.
|
* service (server-side) performs that; the page's hint box says so.
|
||||||
* The phase-49 upload is the exception: it unpacks and scans the
|
* The phase-49 upload is the exception: it unpacks and scans the
|
||||||
* single source in place, and its response counts render as the
|
* single source in place (the phase-64 background task — 202 +
|
||||||
* result line.
|
* status endpoint), and its counts render as the result line.
|
||||||
*
|
*
|
||||||
* The shared header module loads through this script's own relative
|
* The shared header module loads through this script's own relative
|
||||||
* import ("./header.js") — a hoisted import evaluated before this body
|
* import ("./header.js") — a hoisted import evaluated before this body
|
||||||
@@ -354,22 +372,42 @@ wireAddForm({
|
|||||||
idleLabel: "Add source",
|
idleLabel: "Add source",
|
||||||
});
|
});
|
||||||
|
|
||||||
/* ---------- upload (POST /api/git-sources/upload) — phase 49 -------
|
/* ---------- upload (POST /api/git-sources/upload) — phase 64 (task 05) -------
|
||||||
* The archive upload form: the file input's selection is posted as
|
* The archive upload form follows the phase-64 202 contract (A1):
|
||||||
* FormData (the browser sets the multipart boundary — no manual
|
* the file input's selection is posted as FormData (the browser sets
|
||||||
* Content-Type). §7.4 never-stale: "Uploading…" while in flight,
|
* the multipart boundary — no manual Content-Type), and the 202
|
||||||
* restored in the finally block on success AND failure. 200 → the
|
* answers the moment the archive is safely on disk — the "Uploading…"
|
||||||
* input clears, the sync-style counts land in the role=status result
|
* label covers only that short receive. Then the button HANDS OVER to
|
||||||
* line, the announcer confirms, and loadSources() re-renders the row
|
* the scan: 202 → the page-local "Successfully uploaded — <file>"
|
||||||
* (Local badge; a re-upload refreshes the existing row — no
|
* toast (showUploadToast — A2, safe to navigate away), the file input
|
||||||
* duplicate). Non-2xx → the server detail inline (role=alert; 422
|
* clears, and the processing state ("Processing…", disabled, title
|
||||||
* format/name/traversal, 413 size, 409 busy — user-safe as-is) with
|
* cleared) + startUploadPolling() own it — a 2 s poll of
|
||||||
* the file selection KEPT; network failure → the fixed line. */
|
* GET /api/git-sources/upload/status renders the live "Processing…
|
||||||
|
* <file> (n/m)" label (A4 — bare during unpack; the full path rides
|
||||||
|
* the button title) and settles it: success → the sync-style count
|
||||||
|
* line (fmtUploadResult) in the role=status result line + the
|
||||||
|
* "Archive uploaded: …" announce + loadSources (NO second toast — it
|
||||||
|
* already fired at the 202, A2); failure → the sanitized server
|
||||||
|
* error in the role=alert banner + loadSources, the file selection
|
||||||
|
* KEPT for a one-click re-upload. 409 (an upload is already in
|
||||||
|
* progress) raises NO error banner — it re-attaches to the in-flight
|
||||||
|
* run (processing state + poll, never stale); the phase-49 "server
|
||||||
|
* detail inline for 409" branch is superseded. Other non-2xx (422
|
||||||
|
* format/name, 413 cap, 5xx) keep the phase-49 error banner + the
|
||||||
|
* kept file selection; a network failure keeps the fixed line. The
|
||||||
|
* submit finally restores the button ONLY when no poll is active
|
||||||
|
* (PLAN §7.4 — while startUploadPolling owns the button it stays
|
||||||
|
* disabled / "Processing…"). Boot re-attach (initUploadStatus, the
|
||||||
|
* admin branch): a running scan re-enters the processing state + poll
|
||||||
|
* (no second upload, no error); a terminal run re-renders its result
|
||||||
|
* line (success) or error banner (failed); idle does nothing.
|
||||||
|
* (The phase-49 synchronous 200 paragraph is superseded by phase 64.) */
|
||||||
|
|
||||||
/* The success line's text — the sync-result shape (sources.js's
|
/* The success line's text — the sync-result shape (sources.js's
|
||||||
fmtSyncResult convention): "N added" always leads, then updated /
|
fmtSyncResult convention): "N added" always leads, then updated /
|
||||||
unchanged / pruned — zero parts omitted (unchanged is shown
|
unchanged / pruned — zero parts omitted (unchanged is shown
|
||||||
when nothing was added or updated). */
|
when nothing was added or updated). Reads exactly the keys the
|
||||||
|
upload status's detail carries (task 03's UploadOut-shaped dict). */
|
||||||
function fmtUploadResult(detail) {
|
function fmtUploadResult(detail) {
|
||||||
const d = detail || {};
|
const d = detail || {};
|
||||||
const added = d.added || 0;
|
const added = d.added || 0;
|
||||||
@@ -383,6 +421,183 @@ function fmtUploadResult(detail) {
|
|||||||
return parts.join(" · ");
|
return parts.join(" · ");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Upload-success toast (phase 64 task 05, A2 — owner-locked): the
|
||||||
|
* "successfully uploaded" confirmation, the phase-55 share-toast
|
||||||
|
* pattern (frontend/assets/app.js) made page-local. A SINGLE node —
|
||||||
|
* lazy-created on the first 202 and reused (toasts never stack): a
|
||||||
|
* new toast replaces a pending one (clear the prior dismiss timer,
|
||||||
|
* re-run the entry). role="status" aria-live="polite" — on THIS page
|
||||||
|
* the toast IS the a11y announcer for the 202 (there is no other
|
||||||
|
* live-region line for it). SUCCESS-ONLY (A2): failures are the
|
||||||
|
* #archive-upload-error banner, never a toast. The .toast CSS ships
|
||||||
|
* as-is (styles.css, phase 55). */
|
||||||
|
const UPLOAD_TOAST_MS = 5000; // ~5 s auto-dismiss (A2)
|
||||||
|
let uploadToastEl = null; // the single toast node — lazy-created, reused
|
||||||
|
let uploadToastTimer = 0; // the pending auto-dismiss (replaced by a new toast)
|
||||||
|
|
||||||
|
function showUploadToast(message) {
|
||||||
|
if (!uploadToastEl) {
|
||||||
|
uploadToastEl = document.createElement("div");
|
||||||
|
uploadToastEl.className = "toast";
|
||||||
|
uploadToastEl.setAttribute("role", "status");
|
||||||
|
uploadToastEl.setAttribute("aria-live", "polite");
|
||||||
|
document.body.appendChild(uploadToastEl);
|
||||||
|
}
|
||||||
|
uploadToastEl.textContent = message; // XSS-safe text assignment
|
||||||
|
// Re-trigger the entry even when a toast is already up (a second
|
||||||
|
// upload accepted while the first toast is showing): clear the
|
||||||
|
// pending dismiss, drop the visible state, force a reflow
|
||||||
|
// (restarts the CSS transition), then show again.
|
||||||
|
clearTimeout(uploadToastTimer);
|
||||||
|
uploadToastEl.classList.remove("is-visible");
|
||||||
|
void uploadToastEl.offsetWidth; // force reflow — the entry transition restarts
|
||||||
|
uploadToastEl.classList.add("is-visible");
|
||||||
|
uploadToastTimer = setTimeout(() => {
|
||||||
|
uploadToastEl.classList.remove("is-visible"); // auto-dismiss ~5 s
|
||||||
|
}, UPLOAD_TOAST_MS);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The scan poll (phase 64 task 05): a 2 s cadence — the SYNC_POLL_MS
|
||||||
|
* house value. Single timer, one loop at a time (the guard makes a
|
||||||
|
* double-start a no-op, and the submit finally reads this same
|
||||||
|
* variable to know whether the poll OWNS the button). Each tick
|
||||||
|
* fetches GET /api/git-sources/upload/status: running → the live
|
||||||
|
* "Processing… <file> (n/m)" label (A4 — bare "Processing…" during
|
||||||
|
* the unpack phase, before any file is indexed; the full untruncated
|
||||||
|
* path rides the button title) + reschedule; success → stop + the
|
||||||
|
* result line + the announcement + the row reload (NO toast — it
|
||||||
|
* fired at the 202, A2); failed → stop + the sanitized server error
|
||||||
|
* banner + the row reload (a post-swap failure keeps the row — the
|
||||||
|
* list state may have changed), the file selection kept for a
|
||||||
|
* one-click re-upload; idle → stop + the button restored
|
||||||
|
* (defensive — a started run never returns to idle). A network blip
|
||||||
|
* retries next tick. */
|
||||||
|
const UPLOAD_POLL_MS = 2000; // the SYNC_POLL_MS house value
|
||||||
|
let uploadPollTimer = null; // null = no poll active (the finally's guard)
|
||||||
|
|
||||||
|
function stopUploadPolling() {
|
||||||
|
if (uploadPollTimer !== null) {
|
||||||
|
clearTimeout(uploadPollTimer);
|
||||||
|
uploadPollTimer = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The button's processing entry (the 202 + the 409 re-attach): from
|
||||||
|
* here the poll OWNS it — disabled, "Processing…", title cleared (a
|
||||||
|
* live file lands on it at the first tick). */
|
||||||
|
function enterUploadProcessingState() {
|
||||||
|
uploadBtn.disabled = true;
|
||||||
|
uploadBtn.textContent = "Processing…";
|
||||||
|
uploadBtn.title = "";
|
||||||
|
}
|
||||||
|
|
||||||
|
/* The idle restore (the poll's terminal branches + the submit
|
||||||
|
* finally, which calls this ONLY when no poll is active — PLAN §7.4). */
|
||||||
|
function restoreUploadButton() {
|
||||||
|
uploadBtn.disabled = false; // never stale — success OR failure
|
||||||
|
uploadBtn.textContent = "Upload & scan";
|
||||||
|
uploadBtn.removeAttribute("title");
|
||||||
|
}
|
||||||
|
|
||||||
|
function startUploadPolling() {
|
||||||
|
if (uploadPollTimer !== null) return; // one poll loop at a time
|
||||||
|
const tick = async () => {
|
||||||
|
let status = null;
|
||||||
|
try {
|
||||||
|
const r = await fetch("/api/git-sources/upload/status");
|
||||||
|
if (r.ok) status = await r.json();
|
||||||
|
} catch { /* network blip — retry next tick */ }
|
||||||
|
if (!status) {
|
||||||
|
uploadPollTimer = setTimeout(tick, UPLOAD_POLL_MS);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// running: the live file label (A4 — bare "Processing…" during
|
||||||
|
// the unpack phase, before any file is indexed).
|
||||||
|
if (status.state === "running") {
|
||||||
|
uploadBtn.textContent =
|
||||||
|
"Processing…" +
|
||||||
|
(status.current_file ? ` ${status.current_file}` : "") +
|
||||||
|
(status.files_total > 0 ? ` (${status.files_done}/${status.files_total})` : "");
|
||||||
|
uploadBtn.title = status.current_file || ""; // full path on hover
|
||||||
|
uploadPollTimer = setTimeout(tick, UPLOAD_POLL_MS);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
stopUploadPolling();
|
||||||
|
if (status.state === "success") {
|
||||||
|
// The scan finished: the result line (the existing helper reads
|
||||||
|
// exactly these keys), the announcement, the row lands. NO toast
|
||||||
|
// here — it already fired at the 202 (A2).
|
||||||
|
const detail = status.detail || {};
|
||||||
|
if (uploadResult) {
|
||||||
|
uploadResult.textContent = fmtUploadResult(detail);
|
||||||
|
uploadResult.hidden = false;
|
||||||
|
}
|
||||||
|
announce(`Archive uploaded: ${detail.source}.`);
|
||||||
|
uploadFileInput.value = "";
|
||||||
|
restoreUploadButton();
|
||||||
|
loadSources(); // the row lands / refreshes
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (status.state === "failed") {
|
||||||
|
// Sanitized server-side (task 03's _sanitize_error): the banner
|
||||||
|
// is the failure UI (A2), the selection stays KEPT for a
|
||||||
|
// one-click re-upload, and the list reloads (a post-swap failure
|
||||||
|
// keeps the row — the list state may have changed).
|
||||||
|
if (uploadError) {
|
||||||
|
uploadError.textContent = status.error || "The upload scan failed.";
|
||||||
|
uploadError.hidden = false;
|
||||||
|
}
|
||||||
|
restoreUploadButton();
|
||||||
|
loadSources(); // the list state may have changed
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// idle: defensive — a started run never returns to idle; just
|
||||||
|
// settle the button (the poll stopped above).
|
||||||
|
restoreUploadButton();
|
||||||
|
};
|
||||||
|
uploadPollTimer = setTimeout(tick, UPLOAD_POLL_MS);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Boot re-attach (phase 64 task 05, the admin branch only): fetch the
|
||||||
|
* upload status ONCE — a running scan re-enters the processing state
|
||||||
|
* + the poll (a reload mid-scan re-attaches instead of dead-ending —
|
||||||
|
* no second upload, no error); a finished run re-renders its result
|
||||||
|
* line ONLY (no announce, no toast — the toast fired at the 202, A2);
|
||||||
|
* a failed run re-renders its error banner; idle does nothing (and a
|
||||||
|
* blip is a no-op — the page boots honest either way). */
|
||||||
|
async function initUploadStatus() {
|
||||||
|
if (!uploadBtn) return;
|
||||||
|
let status;
|
||||||
|
try {
|
||||||
|
const r = await fetch("/api/git-sources/upload/status");
|
||||||
|
if (!r.ok) return;
|
||||||
|
status = await r.json();
|
||||||
|
} catch {
|
||||||
|
/* network blip — boot without the re-attach */
|
||||||
|
}
|
||||||
|
if (!status) return;
|
||||||
|
if (status.state === "running") {
|
||||||
|
enterUploadProcessingState();
|
||||||
|
startUploadPolling(); // the first tick carries the live file
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (status.state === "success") {
|
||||||
|
// The last run's result line only — no announce, no toast (A2).
|
||||||
|
if (uploadResult) {
|
||||||
|
uploadResult.textContent = fmtUploadResult(status.detail);
|
||||||
|
uploadResult.hidden = false;
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (status.state === "failed") {
|
||||||
|
if (uploadError) {
|
||||||
|
uploadError.textContent = status.error || "The upload scan failed.";
|
||||||
|
uploadError.hidden = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// idle: nothing to re-attach.
|
||||||
|
}
|
||||||
|
|
||||||
if (uploadFormEl && uploadFileInput && uploadBtn) {
|
if (uploadFormEl && uploadFileInput && uploadBtn) {
|
||||||
uploadFormEl.addEventListener("submit", async (e) => {
|
uploadFormEl.addEventListener("submit", async (e) => {
|
||||||
e.preventDefault();
|
e.preventDefault();
|
||||||
@@ -399,7 +614,7 @@ if (uploadFormEl && uploadFileInput && uploadBtn) {
|
|||||||
if (uploadError) uploadError.hidden = true;
|
if (uploadError) uploadError.hidden = true;
|
||||||
if (uploadResult) uploadResult.hidden = true; // a new attempt starts clean
|
if (uploadResult) uploadResult.hidden = true; // a new attempt starts clean
|
||||||
uploadBtn.disabled = true; // §7.4: one upload per click
|
uploadBtn.disabled = true; // §7.4: one upload per click
|
||||||
uploadBtn.textContent = "Uploading…";
|
uploadBtn.textContent = "Uploading…"; // the transfer is now short — the 202
|
||||||
try {
|
try {
|
||||||
// Multipart from the form itself (the file input's name is
|
// Multipart from the form itself (the file input's name is
|
||||||
// "file") — the browser sets the boundary; NO manual
|
// "file") — the browser sets the boundary; NO manual
|
||||||
@@ -408,23 +623,35 @@ if (uploadFormEl && uploadFileInput && uploadBtn) {
|
|||||||
method: "POST",
|
method: "POST",
|
||||||
body: new FormData(uploadFormEl),
|
body: new FormData(uploadFormEl),
|
||||||
});
|
});
|
||||||
if (r.ok) {
|
if (r.status === 202) {
|
||||||
let data = {};
|
// The archive is safely on disk (A1) — the "successfully
|
||||||
|
// uploaded" moment: the toast fires NOW (A2), the file input
|
||||||
|
// clears, and the scan's poll takes over the button. The 202
|
||||||
|
// body (UploadAccepted) carries the safe source name; a body
|
||||||
|
// parse failure degrades to the picked file's name.
|
||||||
|
let name = file.name;
|
||||||
try {
|
try {
|
||||||
data = await r.json();
|
const data = await r.json();
|
||||||
|
if (data && typeof data.name === "string" && data.name) name = data.name;
|
||||||
} catch {
|
} catch {
|
||||||
/* the body is advisory — the counts line degrades gracefully */
|
/* body parse failure — the picked file's name degrades fine */
|
||||||
}
|
}
|
||||||
uploadFileInput.value = ""; // 200: the archive is unpacked + scanned
|
showUploadToast(`Successfully uploaded — ${name}`);
|
||||||
if (uploadResult) {
|
uploadFileInput.value = ""; // 202: the archive is on the server
|
||||||
uploadResult.textContent = fmtUploadResult(data);
|
enterUploadProcessingState();
|
||||||
uploadResult.hidden = false;
|
startUploadPolling();
|
||||||
}
|
|
||||||
announce(`Archive uploaded: ${data.source || file.name}.`);
|
|
||||||
await loadSources(); // the new/updated row lands (Local badge)
|
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
// 422 (format/name/traversal), 413 (size), 409 (busy): the server
|
// 409 (an upload is already in progress): NO error banner —
|
||||||
|
// re-attach to the in-flight run (never stale): the processing
|
||||||
|
// state + the poll track it to the terminal. The phase-49
|
||||||
|
// "server detail inline" branch does not apply to 409 anymore.
|
||||||
|
if (r.status === 409) {
|
||||||
|
enterUploadProcessingState();
|
||||||
|
startUploadPolling();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// Other non-2xx (422 format/name, 413 cap, 5xx): the server
|
||||||
// detail inline, the file selection KEPT — the fix is one
|
// detail inline, the file selection KEPT — the fix is one
|
||||||
// re-pick, not a re-type.
|
// re-pick, not a re-type.
|
||||||
if (uploadError) {
|
if (uploadError) {
|
||||||
@@ -437,8 +664,11 @@ if (uploadFormEl && uploadFileInput && uploadBtn) {
|
|||||||
uploadError.hidden = false;
|
uploadError.hidden = false;
|
||||||
}
|
}
|
||||||
} finally {
|
} finally {
|
||||||
uploadBtn.disabled = false; // never stale — success OR failure
|
// Never stale (PLAN §7.4) — but ONLY when no poll owns the
|
||||||
uploadBtn.textContent = "Upload & scan";
|
// button: while startUploadPolling tracks the scan (202 / 409)
|
||||||
|
// it stays disabled / "Processing…", so a finally restore here
|
||||||
|
// would race the poll. No poll → the button is ours to restore.
|
||||||
|
if (uploadPollTimer === null) restoreUploadButton();
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -474,4 +704,8 @@ if (retryBtn) retryBtn.addEventListener("click", () => loadSources());
|
|||||||
if (gateEl) gateEl.hidden = true;
|
if (gateEl) gateEl.hidden = true;
|
||||||
if (contentEl) contentEl.hidden = false;
|
if (contentEl) contentEl.hidden = false;
|
||||||
await loadSources();
|
await loadSources();
|
||||||
|
// Phase 64 (task 05): re-attach a running scan (a reload mid-scan
|
||||||
|
// resumes the Processing state) or re-render a terminal run's
|
||||||
|
// result line / error banner.
|
||||||
|
await initUploadStatus();
|
||||||
})();
|
})();
|
||||||
|
|||||||
+159
-28
@@ -14,6 +14,13 @@
|
|||||||
* page "new chat" means going to the chat, fresh — the module clears
|
* page "new chat" means going to the chat, fresh — the module clears
|
||||||
* the phase-14 conversation key and navigates to "/").
|
* the phase-14 conversation key and navigates to "/").
|
||||||
*
|
*
|
||||||
|
* Phase 64 (task 04): the sync button is also the live progress face
|
||||||
|
* of the BACKGROUND archive scan (task 03) — while an upload import
|
||||||
|
* is in flight the button animates with the upload's current file
|
||||||
|
* ("Importing <file>"), settles + refreshes the catalog when it
|
||||||
|
* finishes, and re-attaches to it on page load. See the sync-button
|
||||||
|
* block below for the full two-job contract.
|
||||||
|
*
|
||||||
* Phase 26: the table's path links open the document in the
|
* Phase 26: the table's path links open the document in the
|
||||||
* almost-fullscreen modal overlay (assets/document-modal.js) on the
|
* almost-fullscreen modal overlay (assets/document-modal.js) on the
|
||||||
* SAME page — no new tab, no navigation. The link keeps its
|
* SAME page — no new tab, no navigation. The link keeps its
|
||||||
@@ -36,6 +43,33 @@ import { fetchIsAdmin, initSharedHeader } from "./header.js";
|
|||||||
* opens an error modal (same as the former header.js module — recreated
|
* opens an error modal (same as the former header.js module — recreated
|
||||||
* here since the navbar button is gone).
|
* here since the navbar button is gone).
|
||||||
*
|
*
|
||||||
|
* Phase 64 (task 04): the button reports the FILE being processed, not
|
||||||
|
* just "Syncing…" — TWO jobs drive it. Each poll tick fetches BOTH
|
||||||
|
* status endpoints — GET /api/sync/status +
|
||||||
|
* GET /api/git-sources/upload/status — and applies this decision
|
||||||
|
* tree, in order (startSyncPolling):
|
||||||
|
* 1. sync running → "Syncing… <file> (n/m)" — bare "Syncing…" until
|
||||||
|
* the import's first file (clone/pull, A4);
|
||||||
|
* 2. upload running → "Importing <file> (n/m)" — the background
|
||||||
|
* archive scan (the "clicked upload, then opened
|
||||||
|
* sources" contract, A3);
|
||||||
|
* 3. sync success → the phase-32 settle (counts + catalog refresh);
|
||||||
|
* 4. sync failed → the phase-32 failure (banner + modal);
|
||||||
|
* 5. upload success → settle "Sync sources" + catalog refresh
|
||||||
|
* (loadDocs — the new documents must appear); the
|
||||||
|
* upload's counts live on the Sources page, never
|
||||||
|
* in #sync-result (A3);
|
||||||
|
* 6. upload failed → settle "Sync sources" — the failure is the
|
||||||
|
* Sources page's error banner, never this page's (A3);
|
||||||
|
* 7. both idle → retry-ready idle.
|
||||||
|
* The live label is the status endpoint's full source/relative/path
|
||||||
|
* (A4): CSS ellipsizes #sync-label; the full untruncated path also
|
||||||
|
* rides the button title (hover) and #sync-result (the aria-live
|
||||||
|
* announcer — screen readers hear it). The load-time re-attach
|
||||||
|
* (initSyncButton) re-enters a RUNNING upload the same way; a terminal
|
||||||
|
* upload is a no-op there (the boot-time loadDocs() already shows the
|
||||||
|
* current catalog).
|
||||||
|
*
|
||||||
* Elements: #sync-btn (the button), #sync-label (the text),
|
* Elements: #sync-btn (the button), #sync-label (the text),
|
||||||
* #sync-icon (the spinner icon), #sync-result (aria-live result
|
* #sync-icon (the spinner icon), #sync-result (aria-live result
|
||||||
* line), #sync-error-banner / #sync-error-text (error banner).
|
* line), #sync-error-banner / #sync-error-text (error banner).
|
||||||
@@ -70,6 +104,22 @@ function fmtSyncTime(iso) {
|
|||||||
return `${pad(d.getHours())}:${pad(d.getMinutes())}`;
|
return `${pad(d.getHours())}:${pad(d.getMinutes())}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Phase 64 (task 04): the live-file label. `kind` picks the prefix —
|
||||||
|
* "sync" → "Syncing…", "upload" → "Importing" (the background scan's
|
||||||
|
* word, A3). The current file — the status endpoint's full
|
||||||
|
* source/relative/path (A4) — is appended while one is being processed;
|
||||||
|
* the BARE prefix shows during the clone/pull (sync) or unpack (upload)
|
||||||
|
* phase, before any file is indexed. The counts appear only once the
|
||||||
|
* import has started (total > 0). CSS ellipsizes the button label; the
|
||||||
|
* same untruncated text goes to the button title + #sync-result (the
|
||||||
|
* aria-live announcer). */
|
||||||
|
function fmtSyncLabel(kind, currentFile, done, total) {
|
||||||
|
const prefix = kind === "upload" ? "Importing" : "Syncing…";
|
||||||
|
let label = currentFile ? `${prefix} ${currentFile}` : prefix;
|
||||||
|
if (total > 0) label += ` (${done}/${total})`;
|
||||||
|
return label;
|
||||||
|
}
|
||||||
|
|
||||||
function fmtSyncResult(detail) {
|
function fmtSyncResult(detail) {
|
||||||
const d = detail || {};
|
const d = detail || {};
|
||||||
const added = d.added || 0;
|
const added = d.added || 0;
|
||||||
@@ -88,15 +138,25 @@ function sanitizeSyncError(message) {
|
|||||||
return text.length > 200 ? `${text.slice(0, 200)}…` : text;
|
return text.length > 200 ? `${text.slice(0, 200)}…` : text;
|
||||||
}
|
}
|
||||||
|
|
||||||
function enterSyncRunningState() {
|
/* The single running-state entry point (phase 64 task 04: the label
|
||||||
|
* carries the live file — `kind` "sync" | "upload", the status
|
||||||
|
* endpoint's current_file + done/total). Same mechanics as before
|
||||||
|
* (disabled, aria-busy, spinning icon, no is-error) plus: the full
|
||||||
|
* untruncated path on the button title (removed when null — no file
|
||||||
|
* yet) and in #sync-result (the aria-live announcer reads the full
|
||||||
|
* live path; CSS ellipsizes the button's label span only). */
|
||||||
|
function enterSyncRunningState(kind, currentFile, done, total) {
|
||||||
if (!syncBtn) return;
|
if (!syncBtn) return;
|
||||||
syncBtn.disabled = true;
|
syncBtn.disabled = true;
|
||||||
syncBtn.setAttribute("aria-busy", "true");
|
syncBtn.setAttribute("aria-busy", "true");
|
||||||
syncBtn.removeAttribute("title");
|
if (currentFile) syncBtn.title = currentFile;
|
||||||
|
else syncBtn.removeAttribute("title");
|
||||||
syncBtn.setAttribute("aria-label", "Sync sources");
|
syncBtn.setAttribute("aria-label", "Sync sources");
|
||||||
syncBtn.classList.remove("is-error");
|
syncBtn.classList.remove("is-error");
|
||||||
if (syncIcon) syncIcon.classList.add("is-spinning");
|
if (syncIcon) syncIcon.classList.add("is-spinning");
|
||||||
if (syncLabel) syncLabel.textContent = "Syncing…";
|
const label = fmtSyncLabel(kind, currentFile, done, total);
|
||||||
|
if (syncLabel) syncLabel.textContent = label;
|
||||||
|
if (syncResult) syncResult.textContent = label;
|
||||||
}
|
}
|
||||||
|
|
||||||
function settleSyncButton(label) {
|
function settleSyncButton(label) {
|
||||||
@@ -193,15 +253,22 @@ function applySyncIdle(status) {
|
|||||||
emitSyncStatus(status || { state: "idle" });
|
emitSyncStatus(status || { state: "idle" });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* The 2 s poll (phase 64 task 04): each tick fetches BOTH jobs — the
|
||||||
|
* sync AND the background upload scan — and applies the two-job
|
||||||
|
* decision tree in order (see the section header). The 403 on the SYNC
|
||||||
|
* fetch hides the button (the whoami backstop); a 403 on the UPLOAD
|
||||||
|
* fetch is simply "no upload" (never a hide), and a network blip on
|
||||||
|
* either fetch retries next tick. */
|
||||||
function startSyncPolling() {
|
function startSyncPolling() {
|
||||||
if (syncPollTimer !== null) return;
|
if (syncPollTimer !== null) return;
|
||||||
const tick = async () => {
|
const tick = async () => {
|
||||||
let status = null;
|
let syncStatus = null;
|
||||||
|
let uploadStatus = null;
|
||||||
let notAdmin = false;
|
let notAdmin = false;
|
||||||
try {
|
try {
|
||||||
const r = await fetch("/api/sync/status");
|
const r = await fetch("/api/sync/status");
|
||||||
if (r.status === 403) notAdmin = true;
|
if (r.status === 403) notAdmin = true;
|
||||||
else if (r.ok) status = await r.json();
|
else if (r.ok) syncStatus = await r.json();
|
||||||
} catch { /* network blip — retry next tick */ }
|
} catch { /* network blip — retry next tick */ }
|
||||||
if (notAdmin) {
|
if (notAdmin) {
|
||||||
stopSyncPolling();
|
stopSyncPolling();
|
||||||
@@ -209,28 +276,66 @@ function startSyncPolling() {
|
|||||||
applySyncIdle();
|
applySyncIdle();
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (!status) {
|
// The SECOND job: the background upload scan (admin-only surface).
|
||||||
|
try {
|
||||||
|
const ur = await fetch("/api/git-sources/upload/status");
|
||||||
|
if (ur.ok) uploadStatus = await ur.json();
|
||||||
|
} catch { /* network blip — retry next tick */ }
|
||||||
|
if (!syncStatus) {
|
||||||
syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
|
syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (status.state === "success") {
|
// 1. sync running: the live sync file (bare "Syncing…" until the
|
||||||
stopSyncPolling();
|
// import's first file — A4).
|
||||||
applySyncSuccess(status);
|
if (syncStatus.state === "running") {
|
||||||
|
enterSyncRunningState(
|
||||||
|
"sync", syncStatus.current_file, syncStatus.files_done, syncStatus.files_total
|
||||||
|
);
|
||||||
|
syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (status.state === "failed") {
|
// 2. upload running: the same animation, the upload's file (A3).
|
||||||
stopSyncPolling();
|
if (uploadStatus && uploadStatus.state === "running") {
|
||||||
applySyncFailure(status);
|
enterSyncRunningState(
|
||||||
|
"upload", uploadStatus.current_file, uploadStatus.files_done, uploadStatus.files_total
|
||||||
|
);
|
||||||
|
syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (status.state === "idle") {
|
if (syncStatus.state === "success") {
|
||||||
stopSyncPolling();
|
stopSyncPolling();
|
||||||
applySyncIdle(status);
|
applySyncSuccess(syncStatus);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
// Still running: keep button state honest and re-schedule.
|
if (syncStatus.state === "failed") {
|
||||||
enterSyncRunningState();
|
stopSyncPolling();
|
||||||
syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
|
applySyncFailure(syncStatus);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// 5. upload success: settle + catalog refresh (A3 — the upload's
|
||||||
|
// counts live on the Sources page; #sync-result stays empty).
|
||||||
|
if (uploadStatus && uploadStatus.state === "success") {
|
||||||
|
stopSyncPolling();
|
||||||
|
settleSyncButton("Sync sources");
|
||||||
|
if (syncResult) syncResult.textContent = "";
|
||||||
|
hideSyncError();
|
||||||
|
emitSyncStatus({ state: "idle" });
|
||||||
|
loadDocs();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// 6. upload failed: settle only — the failure is the Sources page's
|
||||||
|
// error banner, never this page's (A3).
|
||||||
|
if (uploadStatus && uploadStatus.state === "failed") {
|
||||||
|
stopSyncPolling();
|
||||||
|
settleSyncButton("Sync sources");
|
||||||
|
if (syncResult) syncResult.textContent = "";
|
||||||
|
hideSyncError();
|
||||||
|
emitSyncStatus({ state: "idle" });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// 7. both idle: settle retry-ready.
|
||||||
|
stopSyncPolling();
|
||||||
|
applySyncIdle(syncStatus);
|
||||||
};
|
};
|
||||||
syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
|
syncPollTimer = setTimeout(tick, SYNC_POLL_MS);
|
||||||
}
|
}
|
||||||
@@ -253,8 +358,10 @@ async function startSync() {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (r.status === 202 || r.status === 409) {
|
if (r.status === 202 || r.status === 409) {
|
||||||
enterSyncRunningState();
|
// Phase 64: the run is just starting (model check / clone-pull) —
|
||||||
if (syncResult) syncResult.textContent = "";
|
// bare "Syncing…" until the first polled file (A4); entering the
|
||||||
|
// running state also clears #sync-result with the same label.
|
||||||
|
enterSyncRunningState("sync", null, 0, 0);
|
||||||
hideSyncError();
|
hideSyncError();
|
||||||
if (lastSyncState !== "running") emitSyncStatus({ state: "running" });
|
if (lastSyncState !== "running") emitSyncStatus({ state: "running" });
|
||||||
startSyncPolling();
|
startSyncPolling();
|
||||||
@@ -268,8 +375,12 @@ async function startSync() {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Load-time re-attach (ADMIN ONLY): a running run re-enters running state,
|
/* Load-time re-attach (ADMIN ONLY): a running run re-enters running
|
||||||
* a terminal run renders its last result. */
|
* state, a terminal run renders its last result. Phase 64 (A3): with
|
||||||
|
* the sync IDLE, an in-flight background upload scan adopts the button
|
||||||
|
* the same way — the "user clicked upload, then opened sources" case;
|
||||||
|
* a terminal upload is a no-op (the boot-time loadDocs() already shows
|
||||||
|
* the current catalog). */
|
||||||
async function initSyncButton() {
|
async function initSyncButton() {
|
||||||
if (!syncBtn) return;
|
if (!syncBtn) return;
|
||||||
if (!(await fetchIsAdmin())) return;
|
if (!(await fetchIsAdmin())) return;
|
||||||
@@ -281,16 +392,36 @@ async function initSyncButton() {
|
|||||||
status = await r.json();
|
status = await r.json();
|
||||||
} catch { return; }
|
} catch { return; }
|
||||||
if (status.state === "running") {
|
if (status.state === "running") {
|
||||||
enterSyncRunningState();
|
enterSyncRunningState(
|
||||||
|
"sync", status.current_file, status.files_done, status.files_total
|
||||||
|
);
|
||||||
emitSyncStatus(status);
|
emitSyncStatus(status);
|
||||||
startSyncPolling();
|
startSyncPolling();
|
||||||
} else if (status.state === "success") {
|
return;
|
||||||
applySyncSuccess(status);
|
|
||||||
} else if (status.state === "failed") {
|
|
||||||
applySyncFailure(status);
|
|
||||||
} else {
|
|
||||||
applySyncIdle(status);
|
|
||||||
}
|
}
|
||||||
|
if (status.state === "success") {
|
||||||
|
applySyncSuccess(status);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (status.state === "failed") {
|
||||||
|
applySyncFailure(status);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// Sync idle: check the SECOND job — an in-flight upload scan re-attaches.
|
||||||
|
let upload;
|
||||||
|
try {
|
||||||
|
const ur = await fetch("/api/git-sources/upload/status");
|
||||||
|
if (ur.ok) upload = await ur.json();
|
||||||
|
} catch { /* network blip — the idle settle below is still honest */ }
|
||||||
|
if (upload && upload.state === "running") {
|
||||||
|
enterSyncRunningState(
|
||||||
|
"upload", upload.current_file, upload.files_done, upload.files_total
|
||||||
|
);
|
||||||
|
emitSyncStatus({ state: "running" });
|
||||||
|
startSyncPolling();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
applySyncIdle(status);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (syncBtn) {
|
if (syncBtn) {
|
||||||
|
|||||||
@@ -1496,7 +1496,18 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
|
|||||||
@media (prefers-reduced-motion: reduce) {
|
@media (prefers-reduced-motion: reduce) {
|
||||||
.sync-btn .sync-icon.is-spinning { animation: none; }
|
.sync-btn .sync-icon.is-spinning { animation: none; }
|
||||||
}
|
}
|
||||||
.sync-label { display: inline; }
|
/* Phase 64: the live-file label ("Syncing… <file> (n/m)" / "Importing
|
||||||
|
<file> (n/m)") — a long source/relative/path ellipsizes inside the
|
||||||
|
pill; the full path lives in the button title + #sync-result (the
|
||||||
|
aria-live announcer). */
|
||||||
|
.sync-label {
|
||||||
|
display: inline-block;
|
||||||
|
max-width: min(16rem, 40vw);
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
white-space: nowrap;
|
||||||
|
vertical-align: bottom;
|
||||||
|
}
|
||||||
/* Last-result announcer — soft ink, small mono */
|
/* Last-result announcer — soft ink, small mono */
|
||||||
.sync-result {
|
.sync-result {
|
||||||
display: block;
|
display: block;
|
||||||
|
|||||||
+21
-10
@@ -179,16 +179,27 @@
|
|||||||
<p class="git-source-error" id="git-source-error" role="alert" hidden></p>
|
<p class="git-source-error" id="git-source-error" role="alert" hidden></p>
|
||||||
</form>
|
</form>
|
||||||
|
|
||||||
<!-- Phase 49 (owner permission 2026-08-28): the archive upload form
|
<!-- Phase 49 (owner permission 2026-08-28): the archive upload
|
||||||
replaces the phase-38 local-directory form — an uploaded
|
form replaces the phase-38 local-directory form — an
|
||||||
.tar/.tar.gz/.tgz/.zip is unpacked under BOR_UPLOAD_DIR and
|
uploaded .tar/.tar.gz/.tgz/.zip is unpacked under
|
||||||
scanned immediately; the same filename replaces the source in
|
BOR_UPLOAD_DIR and scanned; the same filename replaces the
|
||||||
place (no new folder, no duplicate row). The file control is
|
source in place (no new folder, no duplicate row). The file
|
||||||
labeled (visible <label for=…> — WCAG input-label rule); the
|
control is labeled (visible <label for=…> — WCAG
|
||||||
button runs the §7.4 never-stale lifecycle ("Uploading…"
|
input-label rule); the button runs the §7.4 never-stale
|
||||||
while the POST is out, restored on success AND failure);
|
lifecycle ("Uploading…" while the POST is out). Phase 64
|
||||||
non-2xx shows the server detail inline (role=alert), 200
|
(task 05) reworks the rest to the 202 contract (the
|
||||||
shows the sync-style counts (role=status). -->
|
phase-49 synchronous 200 paragraph is superseded): the 202
|
||||||
|
arrives the moment the archive is safely on disk (A1) — a
|
||||||
|
JS-created "Successfully uploaded — <file>" toast fires
|
||||||
|
then (A2 — the phase-55 .toast node, no markup here; safe
|
||||||
|
to navigate away) and the button settles into the live
|
||||||
|
"Processing… <file> (n/m)" label (A4 — the full path rides
|
||||||
|
the button title) driven by the 2 s poll of
|
||||||
|
GET /api/git-sources/upload/status, until the success line
|
||||||
|
(role=status) or the sanitized error banner (role=alert)
|
||||||
|
lands; 409 re-attaches to the in-flight run — no error
|
||||||
|
banner; the other non-2xx still show the server detail
|
||||||
|
inline. -->
|
||||||
<form id="archive-upload-form">
|
<form id="archive-upload-form">
|
||||||
<label for="archive-upload-file">Upload a source archive (.tar, .tar.gz, .tgz, .zip)</label>
|
<label for="archive-upload-file">Upload a source archive (.tar, .tar.gz, .tgz, .zip)</label>
|
||||||
<input id="archive-upload-file" name="file" type="file"
|
<input id="archive-upload-file" name="file" type="file"
|
||||||
|
|||||||
@@ -113,8 +113,14 @@
|
|||||||
to pull the latest and re-import.
|
to pull the latest and re-import.
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
<!-- #sync-result is the aria-live announcer for the last sync
|
<!-- #sync-result is the aria-live announcer: the last sync
|
||||||
result ("N added · …") — renders off the "bor:sync-status" event. -->
|
result ("N added · …") when a sync settles, and — phase 64 —
|
||||||
|
the LIVE file label while either job runs ("Syncing… <file>
|
||||||
|
(n/m)" / "Importing <file> (n/m)"), UNTRUNCATED (the button's
|
||||||
|
label span ellipsizes; screen readers hear the full
|
||||||
|
source/relative path, which also rides the button title).
|
||||||
|
After an upload settles it stays empty — the upload's counts
|
||||||
|
live on the Sources page (A3). -->
|
||||||
<span class="sync-result" id="sync-result" role="status" aria-live="polite"></span>
|
<span class="sync-result" id="sync-result" role="status" aria-live="polite"></span>
|
||||||
<!-- Sync failure banner — role="alert" so a failed sync is announced. -->
|
<!-- Sync failure banner — role="alert" so a failed sync is announced. -->
|
||||||
<div class="kb-banner is-error" id="sync-error-banner" role="alert" hidden>
|
<div class="kb-banner is-error" id="sync-error-banner" role="alert" hidden>
|
||||||
|
|||||||
@@ -0,0 +1,93 @@
|
|||||||
|
"""Phase 64 E2E helper — a delay-injecting reverse proxy in front of the
|
||||||
|
mock LLM.
|
||||||
|
|
||||||
|
The mock LLM (``mock_llm.py``) answers instantly: a real archive scan of
|
||||||
|
dozens of files finishes in well under a second and never outlives the
|
||||||
|
UI's 2 s status poll — the phase-64 live file labels ("Processing…
|
||||||
|
<file>", "Importing <file>", "Syncing… <file>"), the at-202 toast →
|
||||||
|
navigate-away contract, and the mid-scan reload re-attach would all be
|
||||||
|
races against the mock. This proxy sits between the app under test and
|
||||||
|
the mock LLM and sleeps ``SLOW_LLM_DELAY_S`` seconds (default 0.15)
|
||||||
|
before forwarding each request, so a scan's duration is deterministic
|
||||||
|
(≈ the run's number of LLM requests × the delay — for an N-file
|
||||||
|
archive/sync that is N + 3: the ``check_models`` embed + chat probe,
|
||||||
|
one embed per file, and the change-gated overview chat). Both the
|
||||||
|
tests' ~100 ms status polling (the deterministic layer) and the UI's
|
||||||
|
2 s poll (the UI layer) then observe the running state, the current
|
||||||
|
file, and the counts reliably.
|
||||||
|
|
||||||
|
Everything else is byte-transparent: method, path, query, headers, and
|
||||||
|
body are forwarded verbatim; the upstream response's status and body
|
||||||
|
come back as-is (``content-encoding`` / ``content-length`` are dropped
|
||||||
|
— httpx has already decoded the body and starlette recomputes the
|
||||||
|
length). The mock's responses are all finite (its SSE streams end with
|
||||||
|
``[DONE]``), so the proxy reads each body to completion before
|
||||||
|
answering.
|
||||||
|
|
||||||
|
Run it the conftest way (a suite's module ``app_server`` fixture spawns
|
||||||
|
it as a subprocess):
|
||||||
|
|
||||||
|
uv run python -m uvicorn tests.e2e.slow_llm:app --port 8902
|
||||||
|
|
||||||
|
with ``E2E_MOCK_PORT`` (upstream, default 8901) and ``SLOW_LLM_DELAY_S``
|
||||||
|
(delay, default 0.15) in its environment.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from fastapi import FastAPI, Request, Response
|
||||||
|
|
||||||
|
#: The mock LLM this proxy forwards to (the conftest's MOCK_PORT).
|
||||||
|
MOCK_PORT = int(os.environ.get("E2E_MOCK_PORT", "8901"))
|
||||||
|
UPSTREAM = f"http://127.0.0.1:{MOCK_PORT}"
|
||||||
|
|
||||||
|
#: Per-request delay in seconds — each suite's proxy fixture picks its
|
||||||
|
#: own (passed through the subprocess env).
|
||||||
|
DELAY_S = float(os.environ.get("SLOW_LLM_DELAY_S", "0.15"))
|
||||||
|
|
||||||
|
app = FastAPI()
|
||||||
|
_client: httpx.AsyncClient | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def _get_client() -> httpx.AsyncClient:
|
||||||
|
global _client
|
||||||
|
if _client is None:
|
||||||
|
_client = httpx.AsyncClient(base_url=UPSTREAM, timeout=60.0)
|
||||||
|
return _client
|
||||||
|
|
||||||
|
|
||||||
|
@app.on_event("shutdown")
|
||||||
|
async def _close_client() -> None:
|
||||||
|
global _client
|
||||||
|
if _client is not None:
|
||||||
|
await _client.aclose()
|
||||||
|
_client = None
|
||||||
|
|
||||||
|
|
||||||
|
@app.api_route("/{path:path}", methods=["GET", "POST", "PUT", "DELETE", "PATCH"])
|
||||||
|
async def proxy(path: str, request: Request) -> Response:
|
||||||
|
"""Sleep ``DELAY_S``, then forward the request to the mock LLM."""
|
||||||
|
await asyncio.sleep(DELAY_S)
|
||||||
|
body = await request.body()
|
||||||
|
headers = {
|
||||||
|
k: v for k, v in request.headers.items() if k.lower() not in ("host", "content-length")
|
||||||
|
}
|
||||||
|
upstream = await _get_client().request(
|
||||||
|
request.method,
|
||||||
|
f"/{path}",
|
||||||
|
content=body,
|
||||||
|
headers=headers,
|
||||||
|
params=dict(request.query_params),
|
||||||
|
)
|
||||||
|
resp_headers = {
|
||||||
|
k: v
|
||||||
|
for k, v in upstream.headers.items()
|
||||||
|
if k.lower()
|
||||||
|
not in ("content-length", "content-encoding", "transfer-encoding", "connection")
|
||||||
|
}
|
||||||
|
return Response(
|
||||||
|
content=upstream.content, status_code=upstream.status_code, headers=resp_headers
|
||||||
|
)
|
||||||
@@ -11,10 +11,26 @@ form is gone, replaced by this form): an uploaded ``.tar``/``.tar.gz``/
|
|||||||
``.tgz``/``.zip`` is safely unpacked under ``BOR_UPLOAD_DIR/<name>/``
|
``.tgz``/``.zip`` is safely unpacked under ``BOR_UPLOAD_DIR/<name>/``
|
||||||
(name = filename minus the archive suffix), the ``git_sources`` row is
|
(name = filename minus the archive suffix), the ``git_sources`` row is
|
||||||
upserted (``kind='local'``, no duplicates), and the source is **scanned
|
upserted (``kind='local'``, no duplicates), and the source is **scanned
|
||||||
synchronously in the request** (single-source ``import_sources`` with
|
in a background task** (phase 64, task 03 — owner-locked A1: the POST
|
||||||
``prune=True`` + the change-gated overview refresh) — the real pipeline,
|
answers **202 the moment the archive is safely on disk** — the
|
||||||
against the deterministic mock LLM (no real models, no network beyond
|
"Successfully uploaded — <source>" toast fires then and the user may
|
||||||
the app itself).
|
navigate away — while unpack → swap → row upsert → model check →
|
||||||
|
single-source ``import_sources`` with ``prune=True`` + the change-gated
|
||||||
|
overview refresh run server-side behind
|
||||||
|
``GET /api/git-sources/upload/status``, the phase-32 ``SyncStatus``
|
||||||
|
pattern with the phase-64 ``current_file``/counts). The result line and
|
||||||
|
the row land from the status ``success`` (same ``UploadOut`` counts,
|
||||||
|
uncompressed in shape) — the real pipeline, against the deterministic
|
||||||
|
mock LLM (no real models, no network beyond the app itself).
|
||||||
|
|
||||||
|
**Timing fixture (phase 64):** the mock LLM answers instantly, so this
|
||||||
|
module's app boots behind ``tests/e2e/slow_llm.py`` — a delay-injecting
|
||||||
|
reverse proxy in front of it (``SLOW_DELAY_S`` per request). A 2-file
|
||||||
|
scan is 5 LLM requests ≈ 5 × 0.6 s ≈ 3 s: long enough to outlive the
|
||||||
|
UI's 2 s status poll, so the button's literal
|
||||||
|
"Uploading… → Processing… → restored" lifecycle is observable
|
||||||
|
(the "Processing…" tick even carries the live file, A4) instead of
|
||||||
|
racing the mock.
|
||||||
|
|
||||||
The archives are **built in-test** with Python's ``tarfile`` over
|
The archives are **built in-test** with Python's ``tarfile`` over
|
||||||
``tmp_path`` fixture files carrying markdown sentinels (``ALPHA-…`` /
|
``tmp_path`` fixture files carrying markdown sentinels (``ALPHA-…`` /
|
||||||
@@ -38,24 +54,33 @@ Contract under test:
|
|||||||
the upload form is in its place with the labeled file input (accept
|
the upload form is in its place with the labeled file input (accept
|
||||||
= the four archive extensions), the "Upload & scan" button, and the
|
= the four archive extensions), the "Upload & scan" button, and the
|
||||||
hint explains unpack/scan + in-place replace;
|
hint explains unpack/scan + in-place replace;
|
||||||
* **upload → scan → list** (§7.4 never-stale): the button shows
|
* **upload → 202 + toast → background scan → list** (§7.4 never-stale,
|
||||||
"Uploading…" while the POST is in flight (the request is held in the
|
phase-64 A1/A2): the button shows "Uploading…" while the POST is in
|
||||||
browser via ``page.route`` so the in-flight state is deterministic),
|
flight (the request is held in the browser via ``page.route`` so the
|
||||||
then restores; the result line shows the added count; the list gains
|
in-flight state is deterministic); at the 202 the "Successfully
|
||||||
exactly one row for ``e2e-upload`` with the **Local** badge;
|
uploaded — <source>" toast fires (``.toast.is-visible``,
|
||||||
``GET /api/docs`` lists both sentinel files under source
|
``role="status"``) WHILE the scan is still running, and the button
|
||||||
``e2e-upload``; the RAG catalog (``/sources.html``) shows them;
|
hands over to the scan — "Processing…" (the live-file tick carries
|
||||||
|
the current file, A4) — then restores when the status ``success``
|
||||||
|
lands: the result line shows the added count; the list gains exactly
|
||||||
|
one row for ``e2e-upload`` with the **Local** badge; ``GET
|
||||||
|
/api/docs`` lists both sentinel files under source ``e2e-upload``;
|
||||||
|
the RAG catalog (``/sources.html``) shows them;
|
||||||
* **re-upload, same filename** → in-place replace: the result line
|
* **re-upload, same filename** → in-place replace: the result line
|
||||||
shows the prune, the list still has exactly ONE ``e2e-upload`` row
|
shows the prune, the SECOND RUN'S STATUS ``detail`` carries the
|
||||||
(no duplicate), the KB shows the changed ``alpha`` + the new
|
prune/refresh counts, the list still has exactly ONE ``e2e-upload``
|
||||||
|
row (no duplicate), the KB shows the changed ``alpha`` + the new
|
||||||
``gamma`` and NOT the dropped ``beta``, and the on-disk folder holds
|
``gamma`` and NOT the dropped ``beta``, and the on-disk folder holds
|
||||||
only the new archive's files;
|
only the new archive's files;
|
||||||
* **bad file** → inline 422 (role=alert) naming the accepted formats,
|
* **bad file** → inline 422 (role=alert) naming the accepted formats
|
||||||
button restored, the file selection kept, the list unchanged, and a
|
(UNCHANGED — the name/format/cap gates are inline, pre-202, exactly
|
||||||
subsequent good upload still works (the form is not wedged);
|
as before), button restored, the file selection kept, the list
|
||||||
|
unchanged, and a subsequent good upload still works (the form is not
|
||||||
|
wedged);
|
||||||
* **anonymous** → the sign-in gate (``#git-sources-gate``) shows, the
|
* **anonymous** → the sign-in gate (``#git-sources-gate``) shows, the
|
||||||
manager (and thus the upload form) stays hidden, and
|
manager (and thus the upload form) stays hidden, and
|
||||||
``POST /api/git-sources/upload`` is 403.
|
``POST /api/git-sources/upload`` is 403 — as is the phase-64
|
||||||
|
``GET /api/git-sources/upload/status`` (same wall).
|
||||||
|
|
||||||
Test → story mapping (Playwright Mapping Rule):
|
Test → story mapping (Playwright Mapping Rule):
|
||||||
1. ``test_form_swapped``
|
1. ``test_form_swapped``
|
||||||
@@ -86,6 +111,7 @@ from e2e.auth_helpers import login
|
|||||||
from e2e.conftest import (
|
from e2e.conftest import (
|
||||||
ADMIN_PASSWORD,
|
ADMIN_PASSWORD,
|
||||||
APP_PORT,
|
APP_PORT,
|
||||||
|
MOCK_PORT,
|
||||||
SESSION_SECRET,
|
SESSION_SECRET,
|
||||||
USE_REAL_LLM,
|
USE_REAL_LLM,
|
||||||
_wait_http,
|
_wait_http,
|
||||||
@@ -94,6 +120,17 @@ from e2e.conftest import (
|
|||||||
REPO = Path(__file__).resolve().parents[2]
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||||
|
|
||||||
|
#: The slow-LLM proxy's port (the conftest's mock LLM stays on MOCK_PORT).
|
||||||
|
SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT", "8902"))
|
||||||
|
SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}"
|
||||||
|
|
||||||
|
#: Per-LLM-request delay on the proxy — a 2-file scan is 5 LLM requests
|
||||||
|
#: (the check_models embed + chat probe, one embed per file, the
|
||||||
|
#: change-gated overview chat) ≈ 5 × 0.6 s ≈ 3 s: the scan outlives the
|
||||||
|
#: UI's 2 s status poll, so the button's "Uploading… → Processing… →
|
||||||
|
#: restored" lifecycle (with the live-file tick, A4) is observable.
|
||||||
|
SLOW_DELAY_S = "0.6"
|
||||||
|
|
||||||
GIT_SOURCES_URL = "/git-sources.html"
|
GIT_SOURCES_URL = "/git-sources.html"
|
||||||
SOURCES_URL = "/sources.html"
|
SOURCES_URL = "/sources.html"
|
||||||
|
|
||||||
@@ -184,17 +221,47 @@ def tarball_v2(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
|||||||
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V2_FILES)
|
return _build_targz(root / f"{SOURCE_NAME}.tar.gz", V2_FILES)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def slow_llm(mock_llm: int) -> Iterator[int]:
|
||||||
|
"""The delay-injecting reverse proxy in front of the mock LLM
|
||||||
|
(tests/e2e/slow_llm.py) — this suite's timing fixture: the phase-64
|
||||||
|
button lifecycle ("Uploading… → Processing… → restored") needs the
|
||||||
|
2-file scan to outlive the UI's 2 s status poll (see
|
||||||
|
``SLOW_DELAY_S``)."""
|
||||||
|
env = dict(os.environ)
|
||||||
|
env.pop("DEBUGPY", None)
|
||||||
|
env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S
|
||||||
|
env["E2E_MOCK_PORT"] = str(MOCK_PORT)
|
||||||
|
proc = subprocess.Popen(
|
||||||
|
[sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app",
|
||||||
|
"--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"],
|
||||||
|
cwd=REPO,
|
||||||
|
env=env,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
_wait_http(f"{SLOW_URL}/v1/models")
|
||||||
|
yield SLOW_PORT
|
||||||
|
finally:
|
||||||
|
proc.terminate()
|
||||||
|
try:
|
||||||
|
proc.wait(timeout=10)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
proc.kill()
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="module")
|
@pytest.fixture(scope="module")
|
||||||
def app_server(
|
def app_server(
|
||||||
mock_llm: int,
|
mock_llm: int,
|
||||||
|
slow_llm: int,
|
||||||
upload_dir: Path,
|
upload_dir: Path,
|
||||||
tmp_path_factory: pytest.TempPathFactory,
|
tmp_path_factory: pytest.TempPathFactory,
|
||||||
) -> Iterator[str]:
|
) -> Iterator[str]:
|
||||||
"""The real app under test — per-module env: uploads unpack into a
|
"""The real app under test — per-module env: the LLM base URL is the
|
||||||
scratch dir and the env git list is forced empty (the dev ``.env``'s
|
SLOW PROXY in front of the mock (the timing fixture), uploads unpack
|
||||||
``BOR_GIT_SOURCES`` must not render as env rows on the initially
|
into a scratch dir and the env git list is forced empty (the dev
|
||||||
empty table). No sync is triggered here — the upload's own scan is
|
``.env``'s ``BOR_GIT_SOURCES`` must not render as env rows on the
|
||||||
the pipeline under test."""
|
initially empty table). No sync is triggered here — the upload's own
|
||||||
|
scan is the pipeline under test."""
|
||||||
env = dict(os.environ)
|
env = dict(os.environ)
|
||||||
env.pop("DEBUGPY", None)
|
env.pop("DEBUGPY", None)
|
||||||
env["BOR_ENVIRONMENT"] = "e2e"
|
env["BOR_ENVIRONMENT"] = "e2e"
|
||||||
@@ -202,7 +269,7 @@ def app_server(
|
|||||||
env["BOR_LLM_BASE_URL"] = (
|
env["BOR_LLM_BASE_URL"] = (
|
||||||
"https://aipi.reeseapps.com/v1"
|
"https://aipi.reeseapps.com/v1"
|
||||||
if USE_REAL_LLM
|
if USE_REAL_LLM
|
||||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
else f"{SLOW_URL}/v1"
|
||||||
)
|
)
|
||||||
# Mock-calibrated threshold (conftest pattern) — no chat turn is
|
# Mock-calibrated threshold (conftest pattern) — no chat turn is
|
||||||
# ever sent in this suite, but the app boots with the same env shape.
|
# ever sent in this suite, but the app boots with the same env shape.
|
||||||
@@ -286,9 +353,10 @@ def _docs(page: Page, app_url: str) -> list[tuple[str, str]]:
|
|||||||
|
|
||||||
def _upload_via_page(page: Page, archive: Path) -> str:
|
def _upload_via_page(page: Page, archive: Path) -> str:
|
||||||
"""Pick the archive, submit the form, and wait for the result line
|
"""Pick the archive, submit the form, and wait for the result line
|
||||||
(the 200 path) — returns its text. The failing path is asserted
|
(the phase-64 202 path: toast at the 202, then the button's status
|
||||||
explicitly by the bad-file test, so any non-result outcome here is
|
polling renders the line from the run's ``success``) — returns its
|
||||||
a test error."""
|
text. The failing path is asserted explicitly by the bad-file test,
|
||||||
|
so any non-result outcome here is a test error."""
|
||||||
page.set_input_files("#archive-upload-file", str(archive))
|
page.set_input_files("#archive-upload-file", str(archive))
|
||||||
page.click("#archive-upload-btn")
|
page.click("#archive-upload-btn")
|
||||||
result = page.locator("#archive-upload-result")
|
result = page.locator("#archive-upload-result")
|
||||||
@@ -298,6 +366,24 @@ def _upload_via_page(page: Page, archive: Path) -> str:
|
|||||||
return text
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_upload_running(page: Page, app_url: str, timeout_s: float = 15.0) -> dict[str, Any]:
|
||||||
|
"""Poll (cookie-authenticated) the upload status endpoint until the
|
||||||
|
run is ``running`` — the phase-64 single source of truth for the
|
||||||
|
background scan (A1)."""
|
||||||
|
deadline = time.monotonic() + timeout_s
|
||||||
|
body: dict[str, Any] = {}
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
r = page.request.get(f"{app_url}/api/git-sources/upload/status")
|
||||||
|
assert r.status == 200, r.text
|
||||||
|
body = r.json()
|
||||||
|
if body["state"] == "running":
|
||||||
|
return body
|
||||||
|
if body["state"] in ("success", "failed"):
|
||||||
|
raise AssertionError(f"the scan settled too fast to observe: {body}")
|
||||||
|
time.sleep(0.1)
|
||||||
|
raise AssertionError(f"the scan never entered running: {body}")
|
||||||
|
|
||||||
|
|
||||||
def _hold_upload_request(page: Page, hold_s: float) -> None:
|
def _hold_upload_request(page: Page, hold_s: float) -> None:
|
||||||
"""Intercept the upload POST and hold the REQUEST in the browser for
|
"""Intercept the upload POST and hold the REQUEST in the browser for
|
||||||
``hold_s`` seconds before letting it reach the server. While it is
|
``hold_s`` seconds before letting it reach the server. While it is
|
||||||
@@ -365,11 +451,15 @@ def test_upload_scans_and_lists(
|
|||||||
page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path
|
page: Page, app_url: str, db_ready: None, tarball_v1: Path, upload_dir: Path
|
||||||
) -> None:
|
) -> None:
|
||||||
"""One real upload through the page: while the POST is in flight the
|
"""One real upload through the page: while the POST is in flight the
|
||||||
button is disabled and reads "Uploading…"; on the 200 it restores,
|
button is disabled and reads "Uploading…"; at the 202 the
|
||||||
the result line shows the added count (2), the file input clears,
|
"Successfully uploaded — <source>" toast fires (A2) WHILE the
|
||||||
the list gains exactly ONE row for ``e2e-upload`` with the Local
|
background scan is still running and the button hands over to it —
|
||||||
badge, ``/api/docs`` lists both sentinel files under the source, and
|
"Processing…" (the 2 s poll tick carries the live file, A4); when
|
||||||
the RAG catalog shows them where the admin expects them."""
|
the status ``success`` lands the button restores, the result line
|
||||||
|
shows the added count (2), the file input clears, the list gains
|
||||||
|
exactly ONE row for ``e2e-upload`` with the Local badge,
|
||||||
|
``/api/docs`` lists both sentinel files under the source, and the
|
||||||
|
RAG catalog shows them where the admin expects them."""
|
||||||
page.set_default_timeout(30_000)
|
page.set_default_timeout(30_000)
|
||||||
_admin_git_sources_page(page, app_url)
|
_admin_git_sources_page(page, app_url)
|
||||||
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
|
||||||
@@ -378,7 +468,7 @@ def test_upload_scans_and_lists(
|
|||||||
result = page.locator("#archive-upload-result")
|
result = page.locator("#archive-upload-result")
|
||||||
|
|
||||||
# Hold the upload request in the browser: the in-flight state below
|
# Hold the upload request in the browser: the in-flight state below
|
||||||
# cannot race the (fast) mock-LLM scan while it is held.
|
# cannot race the receive while it is held.
|
||||||
_hold_upload_request(page, hold_s=0.8)
|
_hold_upload_request(page, hold_s=0.8)
|
||||||
page.set_input_files("#archive-upload-file", str(tarball_v1))
|
page.set_input_files("#archive-upload-file", str(tarball_v1))
|
||||||
btn.click()
|
btn.click()
|
||||||
@@ -388,12 +478,35 @@ def test_upload_scans_and_lists(
|
|||||||
expect(btn).to_have_text("Uploading…")
|
expect(btn).to_have_text("Uploading…")
|
||||||
expect(result).to_be_hidden()
|
expect(result).to_be_hidden()
|
||||||
|
|
||||||
# The request goes out, the server unpacks + scans (mock LLM) and
|
# The request goes out; the server stores the archive and answers
|
||||||
# answers 200 → the result line shows the added count.
|
# 202 the moment it is safely on disk (A1) → the toast fires NOW
|
||||||
|
# (A2) — while the scan is still running — and the button hands
|
||||||
|
# over to the scan (bare "Processing…" — A4: no file yet during the
|
||||||
|
# unpack phase).
|
||||||
|
toast = page.locator(".toast")
|
||||||
|
expect(toast).to_have_count(1, timeout=UPLOAD_TIMEOUT_MS)
|
||||||
|
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
|
||||||
|
assert toast.get_attribute("role") == "status"
|
||||||
|
expect(toast).to_have_text(f"Successfully uploaded — {SOURCE_NAME}")
|
||||||
|
expect(result).to_be_hidden()
|
||||||
|
expect(btn).to_be_disabled()
|
||||||
|
expect(btn).to_have_text("Processing…", timeout=5_000)
|
||||||
|
|
||||||
|
# The scan is running server-side (the status endpoint is the
|
||||||
|
# single source of truth, A1) — the run the UI's poll tracks.
|
||||||
|
_wait_upload_running(page, app_url)
|
||||||
|
|
||||||
|
# …and the button's 2 s poll tick renders the live file label
|
||||||
|
# ("Processing… <file> (n/m)", A4).
|
||||||
|
expect(btn).to_have_text(
|
||||||
|
re.compile(rf"Processing… {re.escape(SOURCE_NAME)}/.+\.md"),
|
||||||
|
timeout=UPLOAD_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
|
||||||
|
# The status success lands → the result line (the same UploadOut
|
||||||
|
# counts) + the never-stale restore (input cleared).
|
||||||
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
||||||
expect(result).to_have_text("2 added")
|
expect(result).to_have_text("2 added")
|
||||||
|
|
||||||
# Never stale: the button restored on success and the input cleared.
|
|
||||||
expect(btn).to_be_enabled()
|
expect(btn).to_be_enabled()
|
||||||
expect(btn).to_have_text("Upload & scan")
|
expect(btn).to_have_text("Upload & scan")
|
||||||
expect(page.locator("#archive-upload-file")).to_have_value("")
|
expect(page.locator("#archive-upload-file")).to_have_value("")
|
||||||
@@ -434,7 +547,9 @@ def test_reupload_replaces_in_place(
|
|||||||
upload_dir: Path,
|
upload_dir: Path,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""v1 then v2 under the SAME filename (``e2e-upload.tar.gz``): the
|
"""v1 then v2 under the SAME filename (``e2e-upload.tar.gz``): the
|
||||||
result line shows the prune, the list still has exactly ONE
|
result line shows the prune, the SECOND RUN'S STATUS ``detail``
|
||||||
|
carries the prune/refresh counts (phase 64 — the line is rendered
|
||||||
|
from the status success), the list still has exactly ONE
|
||||||
``e2e-upload`` row (the row count for that source is invariant — no
|
``e2e-upload`` row (the row count for that source is invariant — no
|
||||||
duplicate), the KB shows the changed ``alpha`` + the new ``gamma``
|
duplicate), the KB shows the changed ``alpha`` + the new ``gamma``
|
||||||
and NOT the dropped ``beta``, and the on-disk folder holds only the
|
and NOT the dropped ``beta``, and the on-disk folder holds only the
|
||||||
@@ -442,7 +557,7 @@ def test_reupload_replaces_in_place(
|
|||||||
page.set_default_timeout(30_000)
|
page.set_default_timeout(30_000)
|
||||||
_admin_git_sources_page(page, app_url)
|
_admin_git_sources_page(page, app_url)
|
||||||
|
|
||||||
# Baseline: v1 through the page (200 → "2 added", one row).
|
# Baseline: v1 through the page (202 → "2 added", one row).
|
||||||
assert _upload_via_page(page, tarball_v1) == "2 added"
|
assert _upload_via_page(page, tarball_v1) == "2 added"
|
||||||
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
||||||
|
|
||||||
@@ -452,6 +567,19 @@ def test_reupload_replaces_in_place(
|
|||||||
result = page.locator("#archive-upload-result")
|
result = page.locator("#archive-upload-result")
|
||||||
expect(result).to_have_text(re.compile(r"\d+ pruned"))
|
expect(result).to_have_text(re.compile(r"\d+ pruned"))
|
||||||
|
|
||||||
|
# The SECOND RUN's status ``detail`` shows the prune/refresh counts
|
||||||
|
# (phase 64: the result line is rendered from this success).
|
||||||
|
r = page.request.get(f"{app_url}/api/git-sources/upload/status")
|
||||||
|
assert r.status == 200, r.text
|
||||||
|
status = r.json()
|
||||||
|
assert status["state"] == "success", status
|
||||||
|
detail = status["detail"]
|
||||||
|
assert detail["source"] == SOURCE_NAME
|
||||||
|
assert detail["files"] == 2
|
||||||
|
assert detail["added"] == 1 # gamma — new in v2
|
||||||
|
assert detail["updated"] == 1 # alpha — changed in v2
|
||||||
|
assert detail["pruned"] == 1 # beta — dropped in v2
|
||||||
|
|
||||||
# No duplicate: exactly ONE row for that source (and one row total).
|
# No duplicate: exactly ONE row for that source (and one row total).
|
||||||
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
expect(page.locator("#git-sources-tbody tr", has_text=SOURCE_NAME)).to_have_count(1)
|
||||||
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
|
||||||
@@ -552,6 +680,8 @@ def test_anonymous_gate(page: Page, app_url: str, db_ready: None) -> None:
|
|||||||
|
|
||||||
# The upload route 403s anonymous callers (require_admin runs before
|
# The upload route 403s anonymous callers (require_admin runs before
|
||||||
# the multipart body is parsed — the body is a stand-in, the
|
# the multipart body is parsed — the body is a stand-in, the
|
||||||
# test_local_directory_sources.py pattern for this route).
|
# test_local_directory_sources.py pattern for this route)…
|
||||||
r = page.request.post(f"{app_url}/api/git-sources/upload", data={"file": ""})
|
r = page.request.post(f"{app_url}/api/git-sources/upload", data={"file": ""})
|
||||||
assert r.status == 403
|
assert r.status == 403
|
||||||
|
# …and so does the phase-64 upload STATUS endpoint (same wall).
|
||||||
|
assert page.request.get(f"{app_url}/api/git-sources/upload/status").status == 403
|
||||||
|
|||||||
@@ -0,0 +1,758 @@
|
|||||||
|
"""Phase 64 story E2E (Playwright): real-time progress for sync + upload.
|
||||||
|
|
||||||
|
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||||
|
|
||||||
|
uv run pytest tests/e2e/test_sync_upload_progress.py -v --no-cov
|
||||||
|
|
||||||
|
The story gate for the phase's executable proof (owner-locked A1–A5):
|
||||||
|
both long-running KB jobs report **which file is being processed right
|
||||||
|
now** — not just "Syncing…"/"Uploading…" — and the archive upload is
|
||||||
|
fully **backgrounded**: ``POST /api/git-sources/upload`` answers 202 the
|
||||||
|
moment the archive is on disk (the "Successfully uploaded — <file>"
|
||||||
|
toast fires — the user may navigate away), the unpack/scan continues
|
||||||
|
server-side behind ``GET /api/git-sources/upload/status`` (the phase-32
|
||||||
|
``SyncStatus`` pattern), and the RAG-page sync button
|
||||||
|
(``/sources.html``) animates with the upload's current file while that
|
||||||
|
scan runs.
|
||||||
|
|
||||||
|
**Timing fixture (the phase's fixture note):** the mock LLM indexes
|
||||||
|
fast — the in-progress state is real but brief (a 25-file scan against
|
||||||
|
it takes ≈0.4 s, well under the UI's 2 s status poll). This module's
|
||||||
|
app therefore boots behind ``tests/e2e/slow_llm.py`` — a delay-injecting
|
||||||
|
reverse proxy in front of the mock LLM (``SLOW_DELAY_S`` per request →
|
||||||
|
a 25-file scan is 28 LLM requests ≈ 4.2 s), so the scan outlives the
|
||||||
|
2 s poll and the live-file label is asserted at BOTH layers the task
|
||||||
|
pins:
|
||||||
|
|
||||||
|
* **deterministic** — the status endpoints (``page.request`` / the
|
||||||
|
concurrent recorder, ~100 ms cadence): ``state == "running"`` with a
|
||||||
|
non-null ``current_file`` (``source/relative/path``) observed at some
|
||||||
|
tick, the counts advancing, and the file-less ticks (unpack/row/
|
||||||
|
model probe — A4) preceding the first file tick;
|
||||||
|
* **UI** — polling the button labels for the ``Importing`` /
|
||||||
|
``Syncing…`` / ``Processing…`` prefix plus a file path (generous
|
||||||
|
timeout), which the pages' own 2 s poll ticks render.
|
||||||
|
|
||||||
|
The upload archive is built in-test with Python's ``tarfile`` from
|
||||||
|
**25 small ``.md`` files** (``e2e-prog.tar.gz`` → source ``e2e-prog``);
|
||||||
|
the sync subject is a host temp dir (``sync-corpus/``, 25 small
|
||||||
|
``.md`` files under ``notes/``) registered as a ``kind=local`` row —
|
||||||
|
the ``test_sync_button.py`` / ``test_local_directory_sources.py``
|
||||||
|
fixture styles. Per-module app env (the conftest pattern):
|
||||||
|
``BOR_UPLOAD_DIR`` scratch, ``BOR_GIT_SOURCES`` forced empty (the sync
|
||||||
|
sources are this suite's own local row), ``BOR_LLM_BASE_URL`` the slow
|
||||||
|
proxy.
|
||||||
|
|
||||||
|
Contract under test:
|
||||||
|
|
||||||
|
* **toast → navigate away (A2 + A3)**: on ``/git-sources.html`` the
|
||||||
|
"Successfully uploaded — <source>" toast (``.toast.is-visible``,
|
||||||
|
``role="status"``) appears while the scan is still running;
|
||||||
|
navigating to ``/sources.html`` shows the sync button animating
|
||||||
|
(spinner + ``aria-busy``) with the ``Importing <file>`` label; on
|
||||||
|
completion the button settles to "Sync sources" (no error UI,
|
||||||
|
``#sync-result`` stays empty — the upload's counts never render
|
||||||
|
there, A3) and the catalog shows the uploaded documents (the
|
||||||
|
phase-63 listing, untouched);
|
||||||
|
* **upload progress (A4)**: during the scan the status endpoint
|
||||||
|
reports a non-null ``current_file`` (``source/relative/path`` shape,
|
||||||
|
full denominator, advancing counts) at running ticks, the upload
|
||||||
|
button shows "Processing… <file>" (bare "Processing…" during unpack)
|
||||||
|
before the result line lands, and the toast fired earlier in the run
|
||||||
|
— the result line itself comes from the status ``success``;
|
||||||
|
* **sync live file (A4)**: a multi-file local source; clicking
|
||||||
|
**Sync sources** on ``/sources.html`` shows "Syncing…" (bare, the
|
||||||
|
pre-64 click state) then "Syncing… <file> (n/m)" (both layers), then
|
||||||
|
the pre-64 success settle — "Synced HH:MM" + the counts result line —
|
||||||
|
preserved, plus the file in the label;
|
||||||
|
* **reload re-attach (A1 + A2)**: starting an upload and reloading
|
||||||
|
``/git-sources.html`` mid-scan leaves the button in the Processing
|
||||||
|
state (disabled) with no error banner and NO second upload (the
|
||||||
|
status endpoint's single run is still the one from before the
|
||||||
|
reload — pinned on its ``started_at``); it then settles with the
|
||||||
|
result line and the list shows exactly one row for the archive.
|
||||||
|
|
||||||
|
Test → story mapping (Playwright Mapping Rule):
|
||||||
|
1. ``test_upload_toast_then_navigate_away``
|
||||||
|
2. ``test_upload_progress_shows_current_file``
|
||||||
|
3. ``test_sync_live_file_label``
|
||||||
|
4. ``test_upload_reattach_after_reload``
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tarfile
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import pytest
|
||||||
|
from playwright.sync_api import Page, expect
|
||||||
|
from sqlalchemy import text
|
||||||
|
|
||||||
|
from app.db import SessionLocal
|
||||||
|
from app.models import GitSource
|
||||||
|
from e2e.auth_helpers import login
|
||||||
|
from e2e.conftest import (
|
||||||
|
ADMIN_PASSWORD,
|
||||||
|
APP_PORT,
|
||||||
|
MOCK_PORT,
|
||||||
|
SESSION_SECRET,
|
||||||
|
USE_REAL_LLM,
|
||||||
|
_wait_http,
|
||||||
|
)
|
||||||
|
|
||||||
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
|
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||||
|
|
||||||
|
GIT_SOURCES_URL = "/git-sources.html"
|
||||||
|
SOURCES_URL = "/sources.html"
|
||||||
|
|
||||||
|
#: The slow-LLM proxy's port (the conftest's mock LLM stays on MOCK_PORT).
|
||||||
|
SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT", "8902"))
|
||||||
|
SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}"
|
||||||
|
|
||||||
|
#: Per-LLM-request delay on the proxy — the scan's duration becomes
|
||||||
|
#: deterministic: an N-file archive scan issues N + 3 LLM requests
|
||||||
|
#: (the check_models embed + chat probe, one embed per file, the
|
||||||
|
#: change-gated overview chat), so a 25-file upload takes ≈ 28 × 0.15 s
|
||||||
|
#: ≈ 4.2 s — long enough to outlive the UI's 2 s status poll (see the
|
||||||
|
#: module docstring's timing-fixture note).
|
||||||
|
SLOW_DELAY_S = "0.15"
|
||||||
|
|
||||||
|
#: The uploaded archive: 25 small docs under ``docs/`` (the phase's
|
||||||
|
#: fixture note — 20+ files so the scan outlives the 2 s poll).
|
||||||
|
UPLOAD_NAME = "e2e-prog"
|
||||||
|
UPLOAD_ARCHIVE = f"{UPLOAD_NAME}.tar.gz"
|
||||||
|
N_FILES = 25
|
||||||
|
UPLOAD_FILES: dict[str, str] = {
|
||||||
|
f"docs/{i:02d}.md": f"# Doc {i:02d}\n\nDeterministic upload content {i:02d}.\n"
|
||||||
|
for i in range(N_FILES)
|
||||||
|
}
|
||||||
|
|
||||||
|
#: The local sync source (host temp dir — the app runs on the same
|
||||||
|
#: machine): 25 small docs under ``notes/`` (the current_file's
|
||||||
|
#: source/relative/path shape has a directory level).
|
||||||
|
SYNC_SOURCE_DIR = "sync-corpus"
|
||||||
|
SYNC_FILES: dict[str, str] = {
|
||||||
|
f"notes/{i:02d}.md": f"# Sync doc {i:02d}\n\nDeterministic sync content {i:02d}.\n"
|
||||||
|
for i in range(N_FILES)
|
||||||
|
}
|
||||||
|
|
||||||
|
#: "Synced HH:MM" — the local-time last-result label (sources.js's
|
||||||
|
#: fmtSyncTime), any hour/minute (test_sync_button.py's pattern).
|
||||||
|
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}")
|
||||||
|
|
||||||
|
#: Generous settle budget: a 25-file scan against the slowed LLM is
|
||||||
|
#: ≈4.2 s; the UI's 2 s poll settles at most one tick after the
|
||||||
|
#: terminal state lands.
|
||||||
|
SETTLE_TIMEOUT_MS = 45_000
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Fixtures
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _build_targz(path: Path, files: dict[str, str]) -> Path:
|
||||||
|
"""A deterministic ``.tar.gz`` (mtime 0) over the given files."""
|
||||||
|
with tarfile.open(path, "w:gz") as tf:
|
||||||
|
for rel, content in files.items():
|
||||||
|
data = content.encode("utf-8")
|
||||||
|
info = tarfile.TarInfo(rel)
|
||||||
|
info.size = len(data)
|
||||||
|
info.mtime = 0
|
||||||
|
tf.addfile(info, io.BytesIO(data))
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def slow_llm(mock_llm: int) -> Iterator[int]:
|
||||||
|
"""The delay-injecting reverse proxy in front of the mock LLM
|
||||||
|
(tests/e2e/slow_llm.py) — this suite's timing fixture: the live-file
|
||||||
|
contract needs the scan to outlive the UI's 2 s poll (see
|
||||||
|
``SLOW_DELAY_S``)."""
|
||||||
|
env = dict(os.environ)
|
||||||
|
env.pop("DEBUGPY", None)
|
||||||
|
env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S
|
||||||
|
env["E2E_MOCK_PORT"] = str(MOCK_PORT)
|
||||||
|
proc = subprocess.Popen(
|
||||||
|
[sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app",
|
||||||
|
"--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"],
|
||||||
|
cwd=REPO,
|
||||||
|
env=env,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
_wait_http(f"{SLOW_URL}/v1/models")
|
||||||
|
yield SLOW_PORT
|
||||||
|
finally:
|
||||||
|
proc.terminate()
|
||||||
|
try:
|
||||||
|
proc.wait(timeout=10)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
proc.kill()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def upload_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||||
|
"""The app's ``BOR_UPLOAD_DIR`` for this suite — a scratch dir the
|
||||||
|
host-side assertions inspect (the app server runs on the same
|
||||||
|
machine)."""
|
||||||
|
return tmp_path_factory.mktemp("bor_uploads") / "uploads"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def upload_archive(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||||
|
"""The 25-file upload archive (``e2e-prog.tar.gz``)."""
|
||||||
|
root = tmp_path_factory.mktemp("bor_archive")
|
||||||
|
return _build_targz(root / UPLOAD_ARCHIVE, UPLOAD_FILES)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def sync_local_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||||
|
"""The host temp dir the sync test registers as a ``kind=local``
|
||||||
|
source — ``sync-corpus/notes/NN.md`` (25 files)."""
|
||||||
|
root = tmp_path_factory.mktemp("bor_sync_local") / SYNC_SOURCE_DIR
|
||||||
|
for rel, content in SYNC_FILES.items():
|
||||||
|
path = root / rel
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(content, encoding="utf-8")
|
||||||
|
return root
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def app_server(
|
||||||
|
mock_llm: int,
|
||||||
|
slow_llm: int,
|
||||||
|
upload_dir: Path,
|
||||||
|
tmp_path_factory: pytest.TempPathFactory,
|
||||||
|
) -> Iterator[str]:
|
||||||
|
"""The real app under test — per-module env: the LLM base URL is the
|
||||||
|
SLOW PROXY in front of the mock (the timing fixture), uploads unpack
|
||||||
|
into a scratch dir, and the env git list is forced empty (the sync
|
||||||
|
sources are this suite's own ``kind=local`` row, seeded per test)."""
|
||||||
|
env = dict(os.environ)
|
||||||
|
env.pop("DEBUGPY", None)
|
||||||
|
env["BOR_ENVIRONMENT"] = "e2e"
|
||||||
|
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||||
|
env["BOR_LLM_BASE_URL"] = (
|
||||||
|
"https://aipi.reeseapps.com/v1"
|
||||||
|
if USE_REAL_LLM
|
||||||
|
else f"{SLOW_URL}/v1"
|
||||||
|
)
|
||||||
|
# Mock-calibrated threshold (conftest pattern) — no chat turn is
|
||||||
|
# ever sent in this suite, but the app boots with the same env shape.
|
||||||
|
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||||
|
env.setdefault(
|
||||||
|
"BOR_DATABASE_URL",
|
||||||
|
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||||
|
)
|
||||||
|
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||||
|
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||||
|
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||||
|
env["BOR_GIT_SOURCES"] = ""
|
||||||
|
env["BOR_UPLOAD_DIR"] = str(upload_dir)
|
||||||
|
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
|
||||||
|
proc = subprocess.Popen(
|
||||||
|
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||||
|
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||||
|
cwd=REPO,
|
||||||
|
env=env,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
_wait_http(f"{APP_URL}/api/health")
|
||||||
|
yield APP_URL
|
||||||
|
finally:
|
||||||
|
proc.terminate()
|
||||||
|
try:
|
||||||
|
proc.wait(timeout=10)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
proc.kill()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def app_url(app_server: str) -> str:
|
||||||
|
return app_server
|
||||||
|
|
||||||
|
|
||||||
|
def _truncate_all() -> None:
|
||||||
|
"""Fresh registry + KB per test (the E2E isolation pattern): the
|
||||||
|
upload's/sync's counts and every ``/api/docs`` assertion must be
|
||||||
|
this test's own doing. The E2E suites share one Postgres, and a
|
||||||
|
leftover git_sources row or document would corrupt the row-count
|
||||||
|
and doc-list assertions (and a leftover row would be a SECOND sync
|
||||||
|
source, skewing the per-file counts)."""
|
||||||
|
with SessionLocal() as db:
|
||||||
|
db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview, git_sources"))
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_no_running_jobs(app_url: str) -> None:
|
||||||
|
"""No background job may leak across tests (the run states live in
|
||||||
|
the app's memory, and a still-running scan would keep importing
|
||||||
|
into the NEXT test's truncated KB): wait for both status endpoints
|
||||||
|
to be non-running BEFORE the truncate. Own admin session (the
|
||||||
|
endpoints are admin-only) — usually a no-op: the tests settle only
|
||||||
|
after their run's terminal state."""
|
||||||
|
with httpx.Client(timeout=5.0) as client:
|
||||||
|
client.post(f"{app_url}/api/login", json={"password": ADMIN_PASSWORD})
|
||||||
|
for path in ("/api/sync/status", "/api/git-sources/upload/status"):
|
||||||
|
body: dict[str, Any] = {}
|
||||||
|
deadline = time.monotonic() + 90
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
r = client.get(f"{app_url}{path}")
|
||||||
|
if r.status_code == 200:
|
||||||
|
body = r.json()
|
||||||
|
if body["state"] != "running":
|
||||||
|
break
|
||||||
|
time.sleep(0.2)
|
||||||
|
assert body["state"] != "running", (
|
||||||
|
f"a background job was still running at test boundary: {path} {body}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
|
||||||
|
_wait_no_running_jobs(app_url)
|
||||||
|
_truncate_all()
|
||||||
|
yield
|
||||||
|
_truncate_all()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_local_source(path: str) -> None:
|
||||||
|
"""Register a ``kind=local`` source row directly (deterministic —
|
||||||
|
the phase-49 page form is upload-only; a plain directory is an
|
||||||
|
API/DB-only operation, the test_local_directory_sources.py note)."""
|
||||||
|
with SessionLocal() as db:
|
||||||
|
db.add(GitSource(url=path, kind="local", path=path))
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _admin_git_sources_page(page: Page, app_url: str) -> None:
|
||||||
|
"""Real form login landing on the git sources page (admin settled:
|
||||||
|
Sign out visible, the manager revealed by the page module)."""
|
||||||
|
login(page, app_url, next=GIT_SOURCES_URL)
|
||||||
|
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
|
||||||
|
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
||||||
|
expect(page.locator("#git-sources-gate")).to_be_hidden()
|
||||||
|
expect(page.locator("#git-sources-content")).to_be_visible()
|
||||||
|
|
||||||
|
|
||||||
|
def _status(page: Page, app_url: str, path: str) -> dict[str, Any]:
|
||||||
|
"""One (cookie-authenticated) status-endpoint fetch."""
|
||||||
|
r = page.request.get(f"{app_url}{path}")
|
||||||
|
assert r.status == 200, r.text
|
||||||
|
return r.json()
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_running_started_at(
|
||||||
|
page: Page, app_url: str, path: str, timeout_s: float = 15.0
|
||||||
|
) -> str:
|
||||||
|
"""Poll the status endpoint until the run is ``running``; return its
|
||||||
|
``started_at`` (the run's identity — a second run would reset it)."""
|
||||||
|
deadline = time.monotonic() + timeout_s
|
||||||
|
body: dict[str, Any] = {}
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
body = _status(page, app_url, path)
|
||||||
|
if body["state"] == "running":
|
||||||
|
assert body["started_at"] is not None
|
||||||
|
return str(body["started_at"])
|
||||||
|
if body["state"] in ("success", "failed"):
|
||||||
|
raise AssertionError(f"the run settled too fast to observe: {body}")
|
||||||
|
time.sleep(0.1)
|
||||||
|
raise AssertionError(f"the run never entered running: {body}")
|
||||||
|
|
||||||
|
|
||||||
|
class _TickRecorder:
|
||||||
|
"""The deterministic layer, concurrent with the UI assertions.
|
||||||
|
|
||||||
|
A daemon thread that tight-polls (~100 ms cadence) the status
|
||||||
|
endpoint with its OWN admin session (``httpx`` — the browser page
|
||||||
|
drives itself in the meantime; the Playwright sync API is not
|
||||||
|
thread-safe, so the thread never touches it), recording every tick
|
||||||
|
from the first poll: the idle prelude, the running ticks (file-less
|
||||||
|
unpack/row/probe phase, then the per-file ticks), and the terminal
|
||||||
|
body. The thread only READS the same endpoint the UI's 2 s poll
|
||||||
|
reads — it starts no jobs and cannot skew the run."""
|
||||||
|
|
||||||
|
def __init__(self, app_url: str, path: str) -> None:
|
||||||
|
self._url = f"{app_url}{path}"
|
||||||
|
self._login_url = f"{app_url}/api/login"
|
||||||
|
self._ticks: list[dict[str, Any]] = []
|
||||||
|
self._terminal: dict[str, Any] | None = None
|
||||||
|
self._stop = threading.Event()
|
||||||
|
self._thread: threading.Thread | None = None
|
||||||
|
|
||||||
|
def start(self) -> None:
|
||||||
|
def run() -> None:
|
||||||
|
with httpx.Client(timeout=5.0) as client:
|
||||||
|
# Its own admin session — the recorder's reads must not
|
||||||
|
# depend on (or disturb) the browser context's cookie.
|
||||||
|
client.post(self._login_url, json={"password": ADMIN_PASSWORD})
|
||||||
|
saw_running = False
|
||||||
|
while not self._stop.is_set():
|
||||||
|
try:
|
||||||
|
r = client.get(self._url)
|
||||||
|
if r.status_code == 200:
|
||||||
|
body = r.json()
|
||||||
|
self._ticks.append(body)
|
||||||
|
# The run status is in the app's memory and
|
||||||
|
# SURVIVES across this module's tests: a
|
||||||
|
# residual terminal state (a previous test's
|
||||||
|
# run) must not be mistaken for this test's
|
||||||
|
# own terminal — accept it only AFTER this
|
||||||
|
# run's "running" has been observed.
|
||||||
|
if body["state"] == "running":
|
||||||
|
saw_running = True
|
||||||
|
elif body["state"] in ("success", "failed") and saw_running:
|
||||||
|
self._terminal = body
|
||||||
|
return
|
||||||
|
except Exception: # noqa: BLE001 — blip: retry next tick
|
||||||
|
pass
|
||||||
|
self._stop.wait(0.1)
|
||||||
|
|
||||||
|
self._thread = threading.Thread(target=run, daemon=True)
|
||||||
|
self._thread.start()
|
||||||
|
|
||||||
|
def stop(self, timeout_s: float = 60.0) -> dict[str, Any]:
|
||||||
|
"""Join until a terminal tick is recorded (or fail the test)."""
|
||||||
|
deadline = time.monotonic() + timeout_s
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
if self._terminal is not None:
|
||||||
|
break
|
||||||
|
time.sleep(0.05)
|
||||||
|
self._stop.set()
|
||||||
|
if self._thread is not None:
|
||||||
|
self._thread.join(timeout=5)
|
||||||
|
assert self._terminal is not None, (
|
||||||
|
"the recorder saw no terminal state "
|
||||||
|
f"(last ticks: {self._ticks[-3:] if self._ticks else 'none'})"
|
||||||
|
)
|
||||||
|
return self._terminal
|
||||||
|
|
||||||
|
@property
|
||||||
|
def running_ticks(self) -> list[dict[str, Any]]:
|
||||||
|
return [t for t in self._ticks if t["state"] == "running"]
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_live_file_ticks(
|
||||||
|
ticks: list[dict[str, Any]], source: str, n_files: int
|
||||||
|
) -> None:
|
||||||
|
"""A4 against the recorded running ticks (the deterministic layer):
|
||||||
|
|
||||||
|
* the file-less ticks (unpack/row/model probe — before any file is
|
||||||
|
indexed) come FIRST (the label is the bare prefix then);
|
||||||
|
* SOME tick reports a non-null ``current_file`` in the
|
||||||
|
``source/relative/path`` shape;
|
||||||
|
* ``files_total`` is the full pre-walk count from the first file
|
||||||
|
tick, and ``files_done`` advances monotonically to it.
|
||||||
|
"""
|
||||||
|
with_file = [t for t in ticks if t["current_file"]]
|
||||||
|
assert with_file, f"no running tick carried a current_file: {ticks[:6]}"
|
||||||
|
first_with = with_file[0]
|
||||||
|
assert first_with["current_file"].startswith(f"{source}/"), first_with
|
||||||
|
assert re.fullmatch(
|
||||||
|
rf"{re.escape(source)}/.+\.(md|markdown)", first_with["current_file"]
|
||||||
|
), first_with
|
||||||
|
assert first_with["files_total"] == n_files, first_with
|
||||||
|
assert 1 <= first_with["files_done"] <= n_files, first_with
|
||||||
|
dones = [t["files_done"] for t in with_file]
|
||||||
|
assert dones == sorted(dones), f"files_done not monotonic: {dones}"
|
||||||
|
# The last file tick is (n-1, n): the hook fires before the final
|
||||||
|
# file's index, so the final count itself can land in the terminal
|
||||||
|
# body instead of a recorded tick (100 ms cadence vs ≈160 ms file).
|
||||||
|
assert max(dones) >= n_files - 1, f"files_done never advanced: {dones}"
|
||||||
|
pre = [t for t in ticks if t["current_file"] is None]
|
||||||
|
assert pre, f"no file-less running tick (the unpack phase): {ticks[:6]}"
|
||||||
|
assert ticks.index(pre[0]) < ticks.index(first_with), (
|
||||||
|
"a file tick preceded the file-less unpack ticks"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1. Toast on 202 → navigate away → sync button animates with the
|
||||||
|
# upload's current file → settle + catalog refresh (A2 + A3)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_upload_toast_then_navigate_away(
|
||||||
|
page: Page, app_url: str, db_ready: None, upload_archive: Path
|
||||||
|
) -> None:
|
||||||
|
"""On ``/git-sources.html``: pick the multi-file archive, submit →
|
||||||
|
the "Successfully uploaded — <source>" toast appears WHILE the scan
|
||||||
|
is still running; immediately navigate to ``/sources.html`` → the
|
||||||
|
sync button is present, animating (icon ``is-spinning``,
|
||||||
|
``aria-busy``) with the ``Importing`` label; wait for the settle →
|
||||||
|
button idle ("Sync sources"), no error UI, and the catalog table
|
||||||
|
shows the uploaded documents (the phase-63 listing, untouched)."""
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
_admin_git_sources_page(page, app_url)
|
||||||
|
# (A previous test's terminal run may re-render its result line at
|
||||||
|
# boot — the task-05 re-attach contract; the submit below clears
|
||||||
|
# it, and that is the "clean start" asserted after the click.)
|
||||||
|
|
||||||
|
page.set_input_files("#archive-upload-file", str(upload_archive))
|
||||||
|
page.click("#archive-upload-btn")
|
||||||
|
|
||||||
|
# The 202 moment (A2): a single .toast node, visible, role=status,
|
||||||
|
# naming the safe source name…
|
||||||
|
toast = page.locator(".toast")
|
||||||
|
expect(toast).to_have_count(1, timeout=SETTLE_TIMEOUT_MS)
|
||||||
|
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
|
||||||
|
assert toast.get_attribute("role") == "status"
|
||||||
|
expect(toast).to_have_text(f"Successfully uploaded — {UPLOAD_NAME}")
|
||||||
|
# …and the scan is still running — the result line is not up yet
|
||||||
|
# (the toast precedes the scan's completion, A2).
|
||||||
|
expect(page.locator("#archive-upload-result")).to_be_hidden()
|
||||||
|
started_at = _wait_running_started_at(page, app_url, "/api/git-sources/upload/status")
|
||||||
|
assert started_at is not None
|
||||||
|
|
||||||
|
# Navigate away immediately (A1: the scan no longer dies with the
|
||||||
|
# page).
|
||||||
|
page.goto(app_url + SOURCES_URL)
|
||||||
|
|
||||||
|
# The RAG-page sync button re-attaches to the in-flight upload scan
|
||||||
|
# (A3): present, animating (spinner + aria-busy), disabled, with
|
||||||
|
# the live "Importing <file>" label — no error UI on this page
|
||||||
|
# (the upload's failure UI lives on the Sources page, A3).
|
||||||
|
btn = page.locator("#sync-btn")
|
||||||
|
expect(btn).to_be_visible(timeout=30_000)
|
||||||
|
expect(btn).to_be_disabled()
|
||||||
|
expect(btn).to_have_attribute("aria-busy", "true")
|
||||||
|
expect(btn.locator(".sync-icon")).to_have_class(re.compile(r"\bis-spinning\b"))
|
||||||
|
expect(page.locator("#sync-label")).to_have_text(
|
||||||
|
re.compile(rf"Importing {re.escape(UPLOAD_NAME)}/"), timeout=SETTLE_TIMEOUT_MS
|
||||||
|
)
|
||||||
|
expect(page.locator("#sync-error-banner")).to_be_hidden()
|
||||||
|
|
||||||
|
# Settle: the button returns to idle, the sync-result line never
|
||||||
|
# rendered the upload's counts (A3), and the catalog refreshes with
|
||||||
|
# the uploaded documents — the phase-63 listing, untouched.
|
||||||
|
expect(page.locator("#sync-label")).to_have_text("Sync sources", timeout=SETTLE_TIMEOUT_MS)
|
||||||
|
expect(btn).to_be_enabled()
|
||||||
|
expect(btn).not_to_have_attribute("aria-busy")
|
||||||
|
expect(btn.locator(".sync-icon")).not_to_have_class(re.compile(r"\bis-spinning\b"))
|
||||||
|
expect(page.locator("#sync-result")).to_have_text("")
|
||||||
|
expect(page.locator("#docs-tbody tr")).to_have_count(N_FILES, timeout=30_000)
|
||||||
|
expect(page.locator("#docs-tbody tr", has_text="docs/00.md")).to_have_count(1)
|
||||||
|
expect(page.locator("#docs-tbody tr", has_text="docs/24.md")).to_have_count(1)
|
||||||
|
expect(page.locator("#docs-tbody tr", has_text=UPLOAD_NAME)).to_have_count(N_FILES)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 2. Upload progress: live current file at BOTH layers; the toast fired
|
||||||
|
# earlier than the result (A4)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_upload_progress_shows_current_file(
|
||||||
|
page: Page, app_url: str, db_ready: None, upload_archive: Path
|
||||||
|
) -> None:
|
||||||
|
"""During the scan: the status endpoint reports a non-null
|
||||||
|
``current_file`` (``source/relative/path`` shape) at some running
|
||||||
|
tick; the upload button label shows "Processing…" with a file path
|
||||||
|
(UI layer) BEFORE the result line lands; the toast fired earlier in
|
||||||
|
the run (not after the result). The result line + the settle come
|
||||||
|
from the status ``success``."""
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
_admin_git_sources_page(page, app_url)
|
||||||
|
|
||||||
|
recorder = _TickRecorder(app_url, "/api/git-sources/upload/status")
|
||||||
|
recorder.start()
|
||||||
|
page.set_input_files("#archive-upload-file", str(upload_archive))
|
||||||
|
page.click("#archive-upload-btn")
|
||||||
|
# A new attempt starts clean (the submit handler hides the result
|
||||||
|
# line — any previous run's re-rendered line is gone by now).
|
||||||
|
expect(page.locator("#archive-upload-result")).to_be_hidden()
|
||||||
|
|
||||||
|
# The toast fires at the 202 — earlier in the run, NOT after the
|
||||||
|
# result (the result line is still down when the toast is up).
|
||||||
|
toast = page.locator(".toast")
|
||||||
|
expect(toast).to_have_count(1, timeout=SETTLE_TIMEOUT_MS)
|
||||||
|
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
|
||||||
|
expect(toast).to_have_text(f"Successfully uploaded — {UPLOAD_NAME}")
|
||||||
|
expect(page.locator("#archive-upload-result")).to_be_hidden()
|
||||||
|
|
||||||
|
# UI layer: the button hands over to the scan — bare "Processing…"
|
||||||
|
# at the 202 (A4: no file yet during the unpack phase)…
|
||||||
|
btn = page.locator("#archive-upload-btn")
|
||||||
|
expect(btn).to_be_disabled()
|
||||||
|
expect(btn).to_have_text("Processing…", timeout=5_000)
|
||||||
|
# …then the live file label ("Processing… <file> (n/m)") at the
|
||||||
|
# page's next 2 s poll tick, still before the result line lands.
|
||||||
|
expect(btn).to_have_text(
|
||||||
|
re.compile(rf"Processing… {re.escape(UPLOAD_NAME)}/.+\.md \(\d+/{N_FILES}\)"),
|
||||||
|
timeout=SETTLE_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
expect(page.locator("#archive-upload-result")).to_be_hidden()
|
||||||
|
|
||||||
|
# Deterministic layer: the recorder's full tick series (the same
|
||||||
|
# endpoint the UI's 2 s poll reads) — file-less unpack ticks first,
|
||||||
|
# then the per-file ticks with the full denominator.
|
||||||
|
terminal = recorder.stop()
|
||||||
|
assert terminal["state"] == "success", terminal
|
||||||
|
_assert_live_file_ticks(recorder.running_ticks, UPLOAD_NAME, N_FILES)
|
||||||
|
# Phase 64: current_file is null in terminal states (the final
|
||||||
|
# counts survive).
|
||||||
|
assert terminal["current_file"] is None
|
||||||
|
assert terminal["files_done"] == N_FILES
|
||||||
|
assert terminal["files_total"] == N_FILES
|
||||||
|
# The result line is rendered from the status success (the
|
||||||
|
# UploadOut-shaped detail, counts unchanged in shape).
|
||||||
|
assert terminal["detail"]["files"] == N_FILES
|
||||||
|
assert terminal["detail"]["added"] == N_FILES
|
||||||
|
assert terminal["detail"]["source"] == UPLOAD_NAME
|
||||||
|
|
||||||
|
# Settle: the result line lands, the button restores, the input
|
||||||
|
# cleared, and the list has exactly one row for the archive.
|
||||||
|
result = page.locator("#archive-upload-result")
|
||||||
|
expect(result).to_have_text(f"{N_FILES} added", timeout=30_000)
|
||||||
|
expect(btn).to_be_enabled()
|
||||||
|
expect(btn).to_have_text("Upload & scan")
|
||||||
|
expect(page.locator("#archive-upload-file")).to_have_value("")
|
||||||
|
expect(page.locator("#git-sources-tbody tr", has_text=UPLOAD_NAME)).to_have_count(1)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 3. Sync live file label: both layers, then the preserved pre-64
|
||||||
|
# success settle (A4)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_sync_live_file_label(
|
||||||
|
page: Page, app_url: str, db_ready: None, sync_local_dir: Path
|
||||||
|
) -> None:
|
||||||
|
"""A multi-file local source (the ``test_sync_button.py`` fixture
|
||||||
|
style): on ``/sources.html`` click **Sync sources** → the label
|
||||||
|
shows "Syncing…" (bare, the pre-64 click state) then "Syncing…
|
||||||
|
<file> (n/m)" while running (endpoint layer: ``current_file``
|
||||||
|
non-null at running ticks; UI layer: the label poll), then the
|
||||||
|
success settle with the counts result line — the pre-phase-64 sync
|
||||||
|
UX is preserved, plus the file."""
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
_seed_local_source(str(sync_local_dir))
|
||||||
|
login(page, app_url) # lands on /sources.html (the button's home)
|
||||||
|
|
||||||
|
btn = page.locator("#sync-btn")
|
||||||
|
expect(btn).to_be_visible(timeout=30_000)
|
||||||
|
expect(page.locator("#sync-label")).to_have_text("Sync sources")
|
||||||
|
|
||||||
|
recorder = _TickRecorder(app_url, "/api/sync/status")
|
||||||
|
recorder.start()
|
||||||
|
btn.click()
|
||||||
|
|
||||||
|
# The click's immediate state is the pre-64 one (A4 — the bare
|
||||||
|
# prefix until the import's first file): disabled, aria-busy,
|
||||||
|
# spinning icon, no error…
|
||||||
|
expect(btn).to_be_disabled()
|
||||||
|
expect(btn).to_have_attribute("aria-busy", "true")
|
||||||
|
expect(btn.locator(".sync-icon")).to_have_class(re.compile(r"\bis-spinning\b"))
|
||||||
|
expect(page.locator("#sync-label")).to_have_text("Syncing…")
|
||||||
|
expect(page.locator("#sync-error-banner")).to_be_hidden()
|
||||||
|
|
||||||
|
# UI layer: the label gains the live file at the page's 2 s poll
|
||||||
|
# tick ("Syncing… <source/relative/path> (n/m)").
|
||||||
|
expect(page.locator("#sync-label")).to_have_text(
|
||||||
|
re.compile(rf"Syncing… {re.escape(SYNC_SOURCE_DIR)}/.+\.md \(\d+/{N_FILES}\)"),
|
||||||
|
timeout=SETTLE_TIMEOUT_MS,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Deterministic layer: the recorder's full tick series — file-less
|
||||||
|
# model-check ticks first, then the per-file ticks (full
|
||||||
|
# denominator, advancing counts).
|
||||||
|
terminal = recorder.stop()
|
||||||
|
assert terminal["state"] == "success", terminal
|
||||||
|
_assert_live_file_ticks(recorder.running_ticks, SYNC_SOURCE_DIR, N_FILES)
|
||||||
|
assert terminal["current_file"] is None
|
||||||
|
assert terminal["files_done"] == N_FILES
|
||||||
|
assert terminal["files_total"] == N_FILES
|
||||||
|
assert terminal["detail"]["added"] == N_FILES
|
||||||
|
|
||||||
|
# The pre-64 success settle, preserved: "Synced HH:MM" + the counts
|
||||||
|
# result line, button re-enabled, no error — and the catalog lists
|
||||||
|
# the imported docs.
|
||||||
|
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
|
||||||
|
expect(btn).to_be_enabled()
|
||||||
|
expect(btn).not_to_have_attribute("aria-busy")
|
||||||
|
expect(page.locator("#sync-result")).to_have_text(f"{N_FILES} added")
|
||||||
|
expect(page.locator("#docs-tbody tr")).to_have_count(N_FILES, timeout=30_000)
|
||||||
|
expect(page.locator("#docs-tbody tr", has_text="notes/00.md")).to_have_count(1)
|
||||||
|
expect(page.locator("#docs-tbody tr", has_text=SYNC_SOURCE_DIR)).to_have_count(N_FILES)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 4. Reload mid-scan → re-attach: no error, no second upload (A1 + A2)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_upload_reattach_after_reload(
|
||||||
|
page: Page, app_url: str, db_ready: None, upload_archive: Path
|
||||||
|
) -> None:
|
||||||
|
"""Start the upload and, DURING the scan, reload
|
||||||
|
``/git-sources.html`` → the button is in the Processing state
|
||||||
|
(disabled) with no error banner and NO second upload (the status
|
||||||
|
endpoint's single run is still the one from before the reload —
|
||||||
|
pinned on its ``started_at``); it then settles with the result line
|
||||||
|
and the list shows exactly one row for the archive (in-place
|
||||||
|
identity preserved)."""
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
_admin_git_sources_page(page, app_url)
|
||||||
|
|
||||||
|
page.set_input_files("#archive-upload-file", str(upload_archive))
|
||||||
|
page.click("#archive-upload-btn")
|
||||||
|
|
||||||
|
# The 202 toast (the run is in flight)…
|
||||||
|
toast = page.locator(".toast")
|
||||||
|
expect(toast).to_have_count(1, timeout=SETTLE_TIMEOUT_MS)
|
||||||
|
expect(toast).to_have_class(re.compile(r"\bis-visible\b"))
|
||||||
|
# …and the run's identity: the status's started_at (a second run
|
||||||
|
# would reset it — the single-run claim is pinned on it).
|
||||||
|
started_at = _wait_running_started_at(page, app_url, "/api/git-sources/upload/status")
|
||||||
|
|
||||||
|
# Reload mid-scan — the page must re-attach, not dead-end.
|
||||||
|
page.reload()
|
||||||
|
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
|
||||||
|
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
||||||
|
expect(page.locator("#git-sources-content")).to_be_visible()
|
||||||
|
|
||||||
|
# The boot re-attach (task 05): the button is in the Processing
|
||||||
|
# state (disabled, "Processing…") with no error banner and no
|
||||||
|
# result line yet…
|
||||||
|
btn = page.locator("#archive-upload-btn")
|
||||||
|
expect(btn).to_be_disabled(timeout=15_000)
|
||||||
|
expect(btn).to_have_text(re.compile(r"Processing…"), timeout=15_000)
|
||||||
|
expect(page.locator("#archive-upload-error")).to_be_hidden()
|
||||||
|
expect(page.locator("#archive-upload-result")).to_be_hidden()
|
||||||
|
|
||||||
|
# …and it settles with the result line from the status success…
|
||||||
|
expect(page.locator("#archive-upload-result")).to_have_text(
|
||||||
|
f"{N_FILES} added", timeout=SETTLE_TIMEOUT_MS
|
||||||
|
)
|
||||||
|
expect(btn).to_be_enabled()
|
||||||
|
expect(btn).to_have_text("Upload & scan")
|
||||||
|
# …and the list shows exactly one row for the archive.
|
||||||
|
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
|
||||||
|
row = page.locator("#git-sources-tbody tr", has_text=UPLOAD_NAME)
|
||||||
|
expect(row).to_have_count(1)
|
||||||
|
expect(row.locator("span.git-source-kind")).to_have_text("Local")
|
||||||
|
|
||||||
|
# No second upload: the terminal status is the SAME run — its
|
||||||
|
# started_at is the one from before the reload.
|
||||||
|
terminal = _status(page, app_url, "/api/git-sources/upload/status")
|
||||||
|
assert terminal["state"] == "success", terminal
|
||||||
|
assert str(terminal["started_at"]) == started_at, (
|
||||||
|
f"the run's identity changed (a second upload ran): {terminal['started_at']}"
|
||||||
|
)
|
||||||
|
assert terminal["current_file"] is None
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -63,7 +63,7 @@ from __future__ import annotations
|
|||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
import time
|
import time
|
||||||
from collections.abc import Iterator
|
from collections.abc import Callable, Iterator
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
@@ -235,6 +235,9 @@ class FakeImportSources:
|
|||||||
self.sources: list[list[Path]] = []
|
self.sources: list[list[Path]] = []
|
||||||
self.llms: list[LLMClient] = []
|
self.llms: list[LLMClient] = []
|
||||||
self.prune_flags: list[bool] = []
|
self.prune_flags: list[bool] = []
|
||||||
|
# Phase 64 (task 02): the progress hook the runner passes (a live
|
||||||
|
# closure while wired, None if the wiring regresses).
|
||||||
|
self.progress_hooks: list[object] = []
|
||||||
|
|
||||||
async def __call__(
|
async def __call__(
|
||||||
self,
|
self,
|
||||||
@@ -244,10 +247,12 @@ class FakeImportSources:
|
|||||||
prune: bool = False,
|
prune: bool = False,
|
||||||
limit: int | None = None,
|
limit: int | None = None,
|
||||||
session: Session | None = None,
|
session: Session | None = None,
|
||||||
|
progress: Callable[[str, str, int, int], None] | None = None,
|
||||||
) -> ImportSummary:
|
) -> ImportSummary:
|
||||||
self.sources.append(list(sources))
|
self.sources.append(list(sources))
|
||||||
self.llms.append(llm)
|
self.llms.append(llm)
|
||||||
self.prune_flags.append(prune)
|
self.prune_flags.append(prune)
|
||||||
|
self.progress_hooks.append(progress)
|
||||||
if self.delay:
|
if self.delay:
|
||||||
await asyncio.sleep(self.delay)
|
await asyncio.sleep(self.delay)
|
||||||
return self.summary
|
return self.summary
|
||||||
@@ -324,6 +329,10 @@ def test_admin_sync_success_reports_full_detail(
|
|||||||
"finished_at": None,
|
"finished_at": None,
|
||||||
"detail": {},
|
"detail": {},
|
||||||
"error": None,
|
"error": None,
|
||||||
|
# Phase 64 (task 02): the per-file progress keys — null/0/0 idle.
|
||||||
|
"current_file": None,
|
||||||
|
"files_done": 0,
|
||||||
|
"files_total": 0,
|
||||||
}
|
}
|
||||||
|
|
||||||
r = sync_client.post("/api/sync")
|
r = sync_client.post("/api/sync")
|
||||||
@@ -353,6 +362,10 @@ def test_admin_sync_success_reports_full_detail(
|
|||||||
assert fake_import.prune_flags == [True]
|
assert fake_import.prune_flags == [True]
|
||||||
assert len(fake_import.llms) == 1
|
assert len(fake_import.llms) == 1
|
||||||
assert isinstance(fake_import.llms[0], LLMClient)
|
assert isinstance(fake_import.llms[0], LLMClient)
|
||||||
|
# Phase 64 (task 02): the runner wires the per-file progress hook
|
||||||
|
# (the live closure the status endpoint reads while the import runs).
|
||||||
|
assert len(fake_import.progress_hooks) == 1
|
||||||
|
assert callable(fake_import.progress_hooks[0])
|
||||||
# Overview: refreshed (added + updated > 0) with the same client.
|
# Overview: refreshed (added + updated > 0) with the same client.
|
||||||
assert fake_overview.llms == [fake_import.llms[0]]
|
assert fake_overview.llms == [fake_import.llms[0]]
|
||||||
# Phase 41: the probe ran first and got the very client the import
|
# Phase 41: the probe ran first and got the very client the import
|
||||||
@@ -733,6 +746,7 @@ def test_import_error_is_reported_with_credentials_masked(
|
|||||||
prune: bool = False,
|
prune: bool = False,
|
||||||
limit: int | None = None,
|
limit: int | None = None,
|
||||||
session: Session | None = None,
|
session: Session | None = None,
|
||||||
|
progress: Callable[[str, str, int, int], None] | None = None,
|
||||||
) -> ImportSummary:
|
) -> ImportSummary:
|
||||||
raise EmbeddingError(
|
raise EmbeddingError(
|
||||||
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 "
|
"embeddings request to https://user:secret@aipi.reeseapps.com/v1 "
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user