Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 129_sync_chunk_progress
Build and Push Containers / build-and-push-app (push) Successful in 8m23s
Details
Build and Push Containers / build-and-push-db (push) Successful in 40s
Details

Browse Source 
All green. Final report:

**Phase 129 final verification pass — 100% green** (previous work intact; this pass re-verified everything and fixed two pre-existing README inconsistencies)

- Verified on-spec: `LLMClient.embed` per-batch `on_progress` hook; importer `on_chunk_progress` plumbing (text/image/PDF branches; summary embed excluded); `SyncStatus.current_chunk/chunks_total` reset+clear contract; UI `.sync-label-chunk` pinned part with untruncated `title`/`#sync-result`
- `uv run pytest --cov=app -q`: exit 0, ~3041 passed, 0 failed, `app/` coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` (isolated, DB up): **2 passed** (endpoint chunk climb with true `chunk_document` total + UI `· chunk n/m` part)
- `uv run ruff check .`: all checks passed · `uv run pyright`: 0 errors, 0 warnings
- Completion criteria: all 5 ✅ (incl. no behavior change in completed phases — full suite green, CLI/chat no-op paths byte-identical)
- Docs: README.md fixed 2 pre-existing inconsistencies (upload section claimed "unpacks + scans immediately" — phase 90 made it unpack+register only, scan deferred to Sync; troubleshooting claimed "120s client timeout" — actual guard is the 300 s pre-token `TURN_TIMEOUT_MS`). Phase-129 label behavior already documented. AGENTS.md re-verified consistent — no changes
- Removed stray untracked `NOTES.md` (8-byte artifact); no live-infra changes; no commits — all changes left in working tree for the harness
- Next pending phase: `130_chat_answer_clusters`
This commit is contained in:
ducoterra
2026-09-28 03:38:14 -04:00
parent 1139f007cf
commit 2d0f608246
53 changed files with 3590 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/129_sync_chunk_progress/00_phase.md → .agents/phases/complete/129_sync_chunk_progress/00_phase.md
View File
.agents/phases/todo/129_sync_chunk_progress/01_llm_embed_hook.md → .agents/phases/complete/129_sync_chunk_progress/01_llm_embed_hook.md
View File
.agents/phases/todo/129_sync_chunk_progress/02_importer_chunk_progress.md → .agents/phases/complete/129_sync_chunk_progress/02_importer_chunk_progress.md
View File
.agents/phases/todo/129_sync_chunk_progress/03_sync_status_chunk_fields.md → .agents/phases/complete/129_sync_chunk_progress/03_sync_status_chunk_fields.md
View File
.agents/phases/todo/129_sync_chunk_progress/04_ui_chunk_part.md → .agents/phases/complete/129_sync_chunk_progress/04_ui_chunk_part.md
View File
.agents/phases/todo/129_sync_chunk_progress/05_e2e_chunk_progress.md → .agents/phases/complete/129_sync_chunk_progress/05_e2e_chunk_progress.md
View File
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a1.err
View File
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a1.md
+12
View File
@@ -0,0 +1,12 @@
All green. Final report:
**Phase 129 final verification pass — complete**
- Verified implementation on-spec: `LLMClient.embed` per-batch `on_progress` hook, `import_sources(on_chunk_progress=…)` plumbed through text/image/PDF branches (summary-chunk embed excluded), `SyncStatus.current_chunk/chunks_total` with reset/clear contract, UI `.sync-label-chunk` third pinned part
- `uv run pytest --cov=app --cov-report=term-missing`: **3041 passed, 0 failed**, `app/` coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` (isolated, DB up): **2 passed** — endpoint chunk-position climb (total = true `chunk_document` count) + label `· chunk n/m` part with untruncated `title`/`#sync-result`
- `uv run ruff check .`: all checks passed · `uv run pyright`: 0 errors, 0 warnings
- Completion criteria: all 5 ✅ (incl. no behavior change in completed phases — full existing suite green, CLI/chat paths byte-identical no-op)
- Docs: README.md — fixed 2 pre-existing inconsistencies: (1) archive-upload section claimed "unpacks + scans immediately" but phase 90 made uploads unpack+register only (scan deferred to the Sync button); (2) troubleshooting claimed a "120s client timeout" but the real guard is the 300 s pre-token `TURN_TIMEOUT_MS`. Phase-129 label behavior already documented ("Sync from the UI"). AGENTS.md — verified consistent (user stories, no-CDN test, gpgsign, quick-ref commands all checked) — no changes
- Removed stray untracked `NOTES.md` (8-byte `# Notes` artifact); no live-infra changes; nothing committed (left in working tree for the harness)
- Next pending phase: `130_chat_answer_clusters`
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a1.validate
+114
View File
@@ -0,0 +1,114 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 97%]
........................................................................ [ 99%]
................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 156 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5152 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a2.err
View File
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a2.md
+12
View File
@@ -0,0 +1,12 @@
All green. Final report:
**Phase 129 final verification pass — 100% green** (previous work intact; this pass re-verified everything and fixed two pre-existing README inconsistencies)
- Verified on-spec: `LLMClient.embed` per-batch `on_progress` hook; importer `on_chunk_progress` plumbing (text/image/PDF branches; summary embed excluded); `SyncStatus.current_chunk/chunks_total` reset+clear contract; UI `.sync-label-chunk` pinned part with untruncated `title`/`#sync-result`
- `uv run pytest --cov=app -q`: exit 0, ~3041 passed, 0 failed, `app/` coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` (isolated, DB up): **2 passed** (endpoint chunk climb with true `chunk_document` total + UI `· chunk n/m` part)
- `uv run ruff check .`: all checks passed · `uv run pyright`: 0 errors, 0 warnings
- Completion criteria: all 5 ✅ (incl. no behavior change in completed phases — full suite green, CLI/chat no-op paths byte-identical)
- Docs: README.md fixed 2 pre-existing inconsistencies (upload section claimed "unpacks + scans immediately" — phase 90 made it unpack+register only, scan deferred to Sync; troubleshooting claimed "120s client timeout" — actual guard is the 300 s pre-token `TURN_TIMEOUT_MS`). Phase-129 label behavior already documented. AGENTS.md re-verified consistent — no changes
- Removed stray untracked `NOTES.md` (8-byte artifact); no live-infra changes; no commits — all changes left in working tree for the harness
- Next pending phase: `130_chat_answer_clusters`
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a2.validate
+114
View File
@@ -0,0 +1,114 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 97%]
........................................................................ [ 99%]
................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 156 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5152 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__01_llm_embed_hook.a1.err
View File
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__01_llm_embed_hook.a1.md
+17
View File
@@ -0,0 +1,17 @@
Task 01 complete. All gates green.
**Implemented**
- `app/rag/llm.py`: `LLMClient.embed` gained keyword-only `on_progress: Callable[[int, int], None] | None = None`; fires `on_progress(len(out) + 1, len(texts))` before each top-level batch request (before the try, so a failing batch still announces its start). Docstring documents the contract, the halving-backstop's non-sub-reporting (approximate position, rare by construction), and the `None` zero-cost no-op.
- `tests/unit/test_llm_client.py`: 6 new tests — hook sequence asserted against `_batch_texts`' actual partition (count-cap and token-budget splits, first-indices strictly increasing, total always `len(texts)`), failing-batch announcement, halving does not sub-report, `None`/default byte-identical (same calls + vectors + counter), empty `texts` → no calls. Test fake `_FakeEmbeddingsService` gained a `fail_from` knob (test-only).
- No existing `embed`/`embed_one` call sites touched (`_batch_texts`/`_embed_batch`/`embed_batches`/`embed_one` internals untouched).
**Results**
- `uv run pytest` → 3030/3030 passed, exit 0 (100% green; no F/E in progress output)
- `uv run pytest --cov=app --cov-report=term-missing` → exit 0, `app/` TOTAL **99%** (>90%); `app/rag/llm.py` 99% (only miss is pre-existing unreachable line 509)
- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors, 0 warnings
**Docs**: no doc impact — internal optional hook only; no env vars, API, CLI, or setup changes (the sync endpoint/UI changes land in tasks 03/04).
**Decisions**: `Callable` imported from `collections.abc` (matches `folder_summaries.py` hook precedent).
**Next pending task**: `.agents/phases/todo/129_sync_chunk_progress/02_importer_chunk_progress.md`
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__01_llm_embed_hook.a1.validate
+114
View File
@@ -0,0 +1,114 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 99%]
...... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5135 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__02_importer_chunk_progress.a1.err
View File
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__02_importer_chunk_progress.a1.md
+16
View File
@@ -0,0 +1,16 @@
Task 02 complete. Report:
**Implemented**
- `app/rag/importer.py`: `import_sources(on_chunk_progress=…)` keyword-only hook (documented beside the `progress` paragraph) → `_index_file(on_chunk=…)` → all three branches (text, image, PDF) → `_replace_content_chunks(on_chunk=…)` → `llm.embed(…, on_progress=on_chunk)`; `Embedder` protocol extended with the task-01 `on_progress` kwarg
- Re-chunk retry naturally reports against the new chunk list (documented); `_store_summary`'s summary-chunk embed never receives the hook; CLI (`scripts/import_docs.py`) passes nothing — byte-identical
- Test fakes made protocol-conform (`tests/fakes.py` single-batch announce + 5 importer-facing fakes forward the kwarg); 4 new unit tests in `tests/unit/test_importer.py` (multi-batch per-file sequences, single-batch + summary exclusion, image `1/1`, PDF content chunks)
**Verification**
- `uv run pytest --cov=app --cov-report=term-missing` → 3034 passed, `app/` 99% (>90%)
- `uv run ruff check . && uv run pyright` → clean (0 errors)
**Docs:** no doc impact (internal plumbing; CLI/API/env/UI unchanged — those land in tasks 03/04).
**Notable decision:** the PDF's embedded-image docs and the phase-125 post-index enrichment pass do NOT receive the hook — they index separate documents, and their `1/1` positions would be A6 "noise after the real progress ends"; the task's work items scope the hook to the walked file's three branch content embeds.
**Next pending task:** `03_sync_status_chunk_fields.md` (SyncStatus fields, closure, clearing, endpoint).
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__02_importer_chunk_progress.a1.validate
+114
View File
@@ -0,0 +1,114 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 97%]
........................................................................ [ 99%]
.......... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5135 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__03_sync_status_chunk_fields.a1.err
View File
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__03_sync_status_chunk_fields.a1.md
+16
View File
@@ -0,0 +1,16 @@
All checks pass. The task is complete — here's the report:
**Implemented (task 03 — `SyncStatus` chunk fields):**
- `app/api/sync.py`: `SyncStatus` gains `current_chunk`/`chunks_total` (defaults 0/0); module + dataclass + endpoint docstrings document the clearing contract
- `_run_sync`: run-start reset, `_chunk_hook` closure passed as `on_chunk_progress=` to `import_sources`, cleared at every import-phase-ending point (overview transition, both summaries transitions, both terminals) while file counts keep-final
- `GET /api/sync/status` exposes both fields after `files_*`
- Tests: `tests/unit/test_sync_button.py` (idle dict pin, mid-run pair test 1/38→13/38 read between ticks, phase-transition + terminal clearing test with gated overview/summaries, failed-terminal convention split, prelude 0/0 pins; `_GatedImport` extended with chunk pairs); `tests/integration/test_sync_api.py` (idle/phase-machine pins + fake signature updates)
**Verification (all green):**
- `uv run pytest -q` (unit+integration): 100% pass
- `uv run pytest --cov=app --cov-report=term-missing`: TOTAL 99% (>90%); `app/api/sync.py` 100%
- `uv run ruff check .`: clean; `uv run pyright`: 0 errors, 0 warnings
**Docs:** no doc impact — README/AGENTS.md don't enumerate the status JSON fields (the user-visible label change lands in task 04).
**Next pending task:** `04_ui_chunk_part.md` (the label's `.sync-label-chunk` part + unit pins).
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__03_sync_status_chunk_fields.a1.validate
+114
View File
@@ -0,0 +1,114 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 97%]
........................................................................ [ 99%]
............ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 156 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5152 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a1.err
View File
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a1.md
+20
View File
@@ -0,0 +1,20 @@
Everything is green and the working tree is in the expected state (no git add/commit; tasks 01–03's changes remain for the harness's phase commit).
**Task 04 complete — the label's chunk part: `Syncing… <file> (2/3) · chunk 14/38`**
- `index.html`: third part `<span class="sync-label-chunk" hidden></span>` after the count span in `#sync-label`; announcer comment updated
- `styles.css`: `.sync-label-chunk` pinned-part rule (flex 0 0 auto / nowrap / --mono) right after `.sync-label-count`; container + mobile comments updated
- `sources.js`: null-guarded `syncLabelChunk` ref; `setSyncLabel(mainText, countText = "", chunkText = "")` (same present/absent contract; full = main + count + chunk rides title + `#sync-result`, A4); `fmtSyncPhaseLabel` import fall-through emits `· chunk n/m` only while `chunks_total > 0`, overview/summaries/bare builders pass none; `enterSyncRunningState` destructures `chunk = ""` default
- Unit pins: 4 new tests (markup, CSS, node harness of the real JS across all 8 label states incl. exact title `Syncing… src/a/b.md (2/3) · chunk 14/38`, only-fall-through-reads-chunk-fields) + updated phase-128 writer/markup pins in both unit files
- Fixed completed-phase regressions the new part caused: phase-128 E2E import snapshot/title regex and phase-90 E2E label/title regexes now accept the chunk part (Playwright `to_have_text` is full-match — verified empirically)
**Results**
- `uv run pytest --cov=app --cov-report=term-missing` → 3040 passed, app/ 99% (>90%)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
- E2E in isolation: `test_sync_label_mobile.py` 4/4, `test_sync_upload_progress.py` 4/4, `test_sync_summary_visibility.py` 3/3
- Live probes: status endpoint reported `58/61` mid-import (file's true chunk count) and 0/0 terminal; real browser rendered the chunk part in 132 frames with the untruncated title
- Docs: README sync States bullet documents the `· chunk n/m` label + title/announcer; no AGENTS.md impact
- Notable: the pre-first-batch window is milliseconds, so running-label E2E assertions had to be chunk-aware rather than relying on a 0/0 frame
Next pending task: `05_e2e_chunk_progress.md` (dedicated Playwright suite, run in isolation).
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a1.validate
+114
View File
@@ -0,0 +1,114 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 97%]
........................................................................ [ 99%]
................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 156 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5152 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a2.err
View File
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a2.md
+10
View File
@@ -0,0 +1,10 @@
All green — the prior rejection was a missing final report, not a failure; the working tree was left intact and now re-verified.
**Report (task 04 — the label's chunk part `· chunk n/m`)**
- Implemented: `#sync-label` third pinned part (hidden `span.sync-label-chunk` in `index.html`; `.sync-label-chunk` mono/pinned CSS after `.sync-label-count`); `sources.js` — null-guarded ref, `setSyncLabel(mainText, countText = "", chunkText = "")` with the untruncated main+count+chunk riding `title` + `#sync-result` (A4), `fmtSyncPhaseLabel` import fall-through emits `· chunk n/m` only while `chunks_total > 0` (overview/summaries/prelude/upload/202-click/idle all pass none), `enterSyncRunningState` destructures `chunk = ""`
- Unit pins: 4 new tests in `tests/unit/test_frontend_sync_upload.py` (markup, CSS, node harness of the real JS across all label states incl. exact title `Syncing… src/a/b.md (2/3) · chunk 14/38`, fall-through-only-reads-chunk-fields) + updated phase-128 pins in `test_sync_button.py`; chunk-aware regexes in phase-128 and phase-90 E2E (Playwright `to_have_text` is full-match)
- Results: `uv run pytest --cov=app --cov-report=term-missing` → **3040 passed**, app/ **99%** (>90%); `uv run ruff check .` → clean; `uv run pyright` → 0 errors
- E2E (isolation): `test_sync_label_mobile.py` 4/4 (re-verified this session); `test_sync_upload_progress.py` 4/4 and `test_sync_summary_visibility.py` 3/4→3/3 passed on identical code last session
- Docs: README sync States bullet documents the chunk part + title/announcer; no AGENTS.md impact
- Decisions: chunk part renders only post-first-batch (the 0/0 window is milliseconds); live browser probe last session confirmed 132 rendered chunk frames + 0/0 terminal
- Next pending task: `05_e2e_chunk_progress.md`
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a2.validate
+114
View File
@@ -0,0 +1,114 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 97%]
........................................................................ [ 99%]
................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 156 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5152 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a1.err
View File
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a1.md
+15
View File
@@ -0,0 +1,15 @@
All gates are green. The task is complete — final report:
**Phase 129, task 05 complete** — `tests/e2e/test_sync_chunk_progress.py` (new, mock-only, house pattern):
- **Test 1 (endpoint, slow leg):** ~100 ms tight-poll recorder proves `chunks_total` == the deterministic `chunk_document` count (61 — the file's TRUE chunk count, computed in-test with the importer's exact args; sizing knobs leak-guard-pinned to code defaults) while `current_chunk` climbs strictly 1 → 60 through the 30 token-budgeted batches (every value a `_batch_texts` boundary; last tick = final batch's first index); the tiny second file restarts to `(1, 1)` at `files_done/total == 2/2`; overview/summaries ticks + the terminal are 0/0 while file counts keep-final (2/2)
- **Test 2 (UI, slow leg):** the page's 2 s poll renders `Syncing… chunkprog/big.md` + `(1/2)` + visible `· chunk n/61`; title + `#sync-result` carry the untruncated concatenation (atomic in-browser check); after the file's span the chunk part is hidden + empty while the label keeps rendering in sync; settle = `Synced HH:MM`, parts hidden, `2 added`
- Fixture: `chunkprog/` local source, ~63.5 KB `big.md` (61 chunks) + one-chunk `small.md`; `SLOW_DELAY_S=1.5` so each embed batch outlives the 100 ms cadence ~15× and the import span (~45 s) the 2 s UI poll ~20×
**Results (all green):**
- `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` → 2 passed (126 s, DB up)
- `uv run pytest --cov=app --cov-report=term-missing` → 3040 passed, `app/` coverage **99%**
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
**Docs:** no impact (test-only task; README/AGENTS.md reference E2E files only as examples).
**Notable:** Playwright's `wait_for_function` takes `arg=` keyword-only in this version; ports 8165/8906 (unused by other suites).
**Next pending task:** phase 129 is complete (task 05 was its last); next phase is `130_chat_answer_clusters` in `.agents/phases/todo/`.
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a1.validate
+114
View File
@@ -0,0 +1,114 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 97%]
........................................................................ [ 99%]
................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 156 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5152 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a2.err
View File
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a2.md
+9
View File
@@ -0,0 +1,9 @@
Documentation check complete: README.md's sync paragraph matches the implemented behavior (label parts, batch-granularity chunk position, untruncated title/announcer) and neither file documents the status fields individually, so nothing is stale; AGENTS.md needs no changes (all referenced commands/files verified present).
**Final report — Phase 129, task 05 (E2E chunk progress):**
- **Implemented:** `tests/e2e/test_sync_chunk_progress.py` (new, mock-only, house pattern): test 1 — ~100 ms recorder on `GET /api/sync/status` proves `chunks_total` == the deterministic `chunk_document` count (61 — the file's TRUE chunk count, computed in-test with the importer's exact args, sizing knobs leak-guard-pinned to code defaults) while `current_chunk` climbs strictly 1 → 60 through the 30 token-budgeted batches (every value a `_batch_texts` boundary; last tick = final batch's first index); the tiny second file restarts to `(1, 1)` at `files_done/total == 2/2`; overview/summaries ticks + terminal are 0/0 while file counts keep-final (2/2). Test 2 — the page's 2 s poll renders `Syncing… chunkprog/big.md` + `(1/2)` + visible `· chunk n/61`; title + `#sync-result` carry the untruncated concatenation (atomic in-browser check); after the file's span the chunk part is hidden + empty while the label keeps rendering; settle = `Synced HH:MM`, parts hidden, `2 added`. Fixture: `chunkprog/` local source, ~63.5 KB `big.md` (61 chunks) + one-chunk `small.md`; `SLOW_DELAY_S=1.5` (each embed batch outlives the 100 ms cadence ~15×, the ~45 s import span the 2 s UI poll ~20×).
- **This attempt:** prior rejection was process-only ("no clean final report"); code was already in the working tree with the two fixes intact (Playwright `arg=` keyword-only; ruff line split). No code changes were needed; all gates re-verified fresh.
- **Results (all green):** `uv run ruff check .` → clean · `uv run pyright` → 0 errors · `uv run pytest --cov=app --cov-report=term-missing` → 3040 passed, `app/` coverage **99%** · `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` → 2 passed (126 s, DB up)
- **Docs:** README.md verified consistent (phase-129 label paragraph matches code); AGENTS.md unchanged, references verified — no doc impact this task.
- **Next pending:** phase 129 is complete (task 05 was its last); next is `.agents/phases/todo/130_chat_answer_clusters`.
.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a2.validate
+114
View File
@@ -0,0 +1,114 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 56%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 97%]
........................................................................ [ 99%]
................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 262 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 156 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 132 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 374 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 571 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 81 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 400 0 100%
--------------------------------------------------
TOTAL 5152 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
README.md
+14 -5
View File
@@ -255,7 +255,10 @@ BOR_SOURCES_DIR=~/bor-sources # default; each repo lands in <dir>/<repo-name>/
Upload a `.tar`, `.tar.gz`, `.tgz`, or `.zip` archive to make it a source.
The form on the Git sources page — and the `POST /api/git-sources/upload`
route behind it — accepts archives and unpacks + scans them immediately.
route behind it — accepts archives and unpacks + registers them (no scan —
the upload indexes nothing; the **Sync sources** button on the RAG page
performs the scan, which leaves room to edit the new source's ignore list
first).
- Re-uploading the same filename replaces the source **in place** (no
second folder, no duplicate row).
@@ -293,8 +296,13 @@ whole git-source refresh in one click:
any missing rows (`only_missing`).
- States: the button shows **Syncing…** with a spinner while polling
`GET /api/sync/status` every 2 s. No client-side timeout — a clone +
embed can legitimately take minutes.
`GET /api/sync/status` every 2 s. During the import phase the label
carries the live file, its `(n/m)` file position, and — while a file's
embed batch is in flight — the pinned `· chunk n/m` embed position
(batch granularity: the first chunk of the in-flight batch over the
file's total content chunks); the full untruncated label also rides
the button title and the aria-live result line. No client-side
timeout — a clone + embed can legitimately take minutes.
- One sync at a time: a second trigger while a run is in flight gets a
`409`.
@@ -869,8 +877,9 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
```
- **KB offline banner in the chat** — Postgres isn't running:
`podman compose up -d db`.
- **Stuck "Thinking…"** — the LLM is slow or down; a 120s client timeout
turns it into an error banner automatically.
- **Stuck "Thinking…"** — the LLM is slow or down; the client's 300 s
pre-token guard (`TURN_TIMEOUT_MS`) turns a hung stream into the error
state automatically.
---
app/api/sync.py
+96 -6
View File
@@ -112,6 +112,21 @@ Terminal states (success AND failed) clear ``phase`` +
``current_summary`` but keep the run's final ``summaries_done`` /
``summaries_total`` (the phase-64 keep-final-counts convention).
Phase 129 (task 03) adds the in-flight file's EMBED-chunk position —
``current_chunk`` (the 1-based first chunk of the in-flight
token-budgeted batch) + ``chunks_total`` (the file's total content
chunks), batch granularity via the :meth:`app.rag.llm.LLMClient.embed`
``on_progress`` hook (tasks 01/02). They are 0/0 before the first
batch (the prelude), during the ``"overview"`` / ``"summaries"``
phases, and in terminal states — the import phase clearing them at
EVERY point it ends (the ``phase = "overview"`` and
``phase = "summaries"`` transitions and both terminals, the same spots
``current_file`` clears). Unlike the phase-64 FILE counts, the chunk
position is a SUB-position of a file that's done — the keep-final-
counts convention does NOT extend to it: the file counts survive the
terminal for the result line's context, the chunk position clears
with the file.
The ``failed`` state's ``error`` string is masked by the shared
sanitizer — the ``user:pass@`` masker now lives in :mod:`app.core.errors`
(imported here under the private name ``_sanitize_error``).
@@ -178,6 +193,19 @@ class SyncStatus:
Terminal states clear ``phase`` + ``current_summary`` but keep the
run's final summary counts (the phase-64 keep-final-counts
convention).
Phase 129 (task 03) chunk-position fields: ``current_chunk`` is
the 1-based first chunk of the in-flight token-budgeted batch and
``chunks_total`` the in-flight file's total content chunks (batch
granularity, the ``LLMClient.embed`` hook — the module docstring's
contract). They are 0/0 before the first batch, during the
``"overview"`` / ``"summaries"`` phases, and in terminal states —
the import phase clears them at every point it ends (the
overview/summaries transitions + both terminals, where
``current_file`` also clears). The phase-64 keep-final-counts
convention covers the FILE counts only — the chunk position is a
sub-position of a file that's done, so it clears in the terminal
while ``files_done`` / ``files_total`` keep their final values.
"""
state: Literal["idle", "running", "success", "failed"] = "idle"
@@ -197,6 +225,13 @@ class SyncStatus:
current_summary: str | None = None
summaries_done: int = 0
summaries_total: int = 0
# Phase 129 (task 03): the current file's EMBED chunk position —
# the first chunk of the in-flight token-budgeted batch and the
# file's total content chunks (the LLMClient.embed hook, batch
# granularity). 0/0 before the first batch, during the
# prelude/overview/summaries phases, and in terminal states.
current_chunk: int = 0
chunks_total: int = 0
_status = SyncStatus()
@@ -211,11 +246,17 @@ def sync_status() -> dict[str, Any]:
``current_file`` (phase 64) is the ``source/relative/path`` the
import is processing right now — null during the clone/pull phase
and in terminal states; ``files_done`` / ``files_total`` carry the
hook's position (0/0 idle). ``phase`` (phase 98) is the pipeline
stage — null in the prelude and terminal states; while in
``"summaries"``, ``current_summary`` + ``summaries_done`` /
``summaries_total`` carry the generator's position (the terminal
keeps the final summary counts).
hook's position (0/0 idle) and keep their final values in the
terminal. ``current_chunk`` / ``chunks_total`` (phase 129) are the
in-flight file's embed-chunk position (batch granularity — the
first chunk of the in-flight batch over the file's total content
chunks): 0/0 before the first batch, in the prelude/overview/
summaries phases, and in terminal states (they clear at every
point the import phase ends; the FILE counts do not). ``phase``
(phase 98) is the pipeline stage — null in the prelude and
terminal states; while in ``"summaries"``, ``current_summary`` +
``summaries_done`` / ``summaries_total`` carry the generator's
position (the terminal keeps the final summary counts).
"""
return {
"state": _status.state,
@@ -226,6 +267,12 @@ def sync_status() -> dict[str, Any]:
"current_file": _status.current_file,
"files_done": _status.files_done,
"files_total": _status.files_total,
# Phase 129 (task 03): the in-flight file's embed-chunk
# position — the dataclass defaults make idle 0/0, and the
# runner clears it at every point the import phase ends
# (overview/summaries transitions + both terminals).
"current_chunk": _status.current_chunk,
"chunks_total": _status.chunks_total,
# Phase 98 (task 01): the phase machine — null/0/0 idle (the
# dataclass defaults) and in terminal states (which clear
# phase + current_summary but keep the final summary counts).
@@ -278,6 +325,10 @@ async def _run_sync() -> None:
_status.current_summary = None
_status.summaries_done = 0
_status.summaries_total = 0
# Phase 129 (task 03): the chunk-position fields reset with the
# run — 0/0 until the import's first embed batch reports.
_status.current_chunk = 0
_status.chunks_total = 0
try:
settings = get_settings()
# Step 1 (phase 41): fail fast — verify both models the sync
@@ -359,18 +410,35 @@ async def _run_sync() -> None:
_status.files_done = done
_status.files_total = total
# Phase 129 (task 03): the in-flight file's embed-chunk
# position — the LLMClient.embed hook (task 01) fires before
# each token-budgeted batch with (first_chunk_1based, total);
# the file's IDENTITY is the sibling ``progress`` hook's (the
# two hooks are paired — a progress call names the file, and
# every chunk position until the next one belongs to it).
def _chunk_hook(chunk: int, total: int) -> None:
_status.current_chunk = chunk
_status.chunks_total = total
# Phase 98 (task 01): the phase machine — the import phase
# starts NOW (the model-check + clone/pull prelude above
# reported ``phase: null``).
_status.phase = "import"
summary: ImportSummary = await import_sources(
sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root,
sources, llm, prune=True, progress=_hook, on_chunk_progress=_chunk_hook,
ignore_by_root=ignore_by_root,
include_hidden_by_root=include_hidden_by_root,
doc_dates_by_root=doc_dates_by_root,
)
overview = False
if summary.added + summary.updated > 0:
_status.phase = "overview" # phase 98 (task 01)
# Phase 129 (task 03): the import phase ends here — the
# chunk position (a sub-position of a done file) clears
# with the phase change; the file counts keep their final
# values (the phase-64 convention covers files only).
_status.current_chunk = 0
_status.chunks_total = 0
overview = await regenerate_overview(llm)
# Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03):
# the folder summaries — the drill-down ls's per-level
@@ -419,6 +487,11 @@ async def _run_sync() -> None:
# position through the (long) span (its total is the
# affected count).
_status.phase = "summaries"
# Phase 129 (task 03): the import phase ends here too
# (the changed-KB branch skips no phase) — same
# chunk-position clear as the overview transition.
_status.current_chunk = 0
_status.chunks_total = 0
folder_stats = await generate_folder_summaries(
fs_db, llm, changed_docs=summary.changed_paths,
on_progress=_summary_hook,
@@ -435,6 +508,11 @@ async def _run_sync() -> None:
# phase + hook (the fill's total is the missing
# count — the hook's loop-start ``total``).
_status.phase = "summaries"
# Phase 129 (task 03): the import phase ends here
# too (the unchanged-walk gap-fill branch) — same
# chunk-position clear.
_status.current_chunk = 0
_status.chunks_total = 0
folder_stats = await generate_folder_summaries(
fs_db, llm, only_missing=True, on_progress=_summary_hook
)
@@ -475,6 +553,12 @@ async def _run_sync() -> None:
_status.current_file = None # phase 64: keep the final counts
_status.phase = None # phase 98: keep the final summary counts
_status.current_summary = None
# Phase 129 (task 03): the chunk position is a sub-position of
# a file that's done — it clears in the terminal (unlike the
# FILE counts, which keep their final values for the result
# line's context).
_status.current_chunk = 0
_status.chunks_total = 0
_status.detail = {
"files": summary.files,
"added": summary.added,
@@ -498,3 +582,9 @@ async def _run_sync() -> None:
_status.current_file = None # phase 64: keep the final counts
_status.phase = None # phase 98: keep the final summary counts
_status.current_summary = None
# Phase 129 (task 03): the chunk position clears in the
# failed terminal too (the convention split — the FILE counts
# keep their final values next to the error, the sub-position
# of a file that's done/abandoned does not).
_status.current_chunk = 0
_status.chunks_total = 0
app/rag/importer.py
+104 -10
View File
@@ -144,7 +144,10 @@ is NOT cleared on success — it is the retrieval display's record
also, task 05).
``import_sources`` accepts an optional per-file ``progress`` callback
(phase 64, task 01) reporting the file being processed right now.
(phase 64, task 01) reporting the file being processed right now, and
an optional ``on_chunk_progress`` callback (phase 129, task 02)
reporting the in-flight file's embedding-chunk position at batch
granularity (see the function docstring).
"""
from __future__ import annotations
@@ -198,7 +201,14 @@ class Embedder(Protocol):
settings: Settings
embed_batches: int
async def embed(self, texts: list[str]) -> list[list[float]]: ...
async def embed(
self,
texts: list[str],
*,
# Phase 129 (task 01): the per-batch position hook — ``None``
# (every pre-phase caller) is a zero-cost guarded no-op.
on_progress: Callable[[int, int], None] | None = None,
) -> list[list[float]]: ...
async def chat(self, messages: list[dict[str, Any]], model: str | None = None) -> str: ...
# ^ the one-shot completion the summarizer uses — the ``lite`` model
@@ -419,6 +429,7 @@ async def import_sources(
limit: int | None = None,
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
@@ -442,6 +453,32 @@ async def import_sources(
exceeds the limit, but ``total`` stays the full pre-walk count (an
incomplete walk must not misreport the denominator).
``on_chunk_progress`` (phase 129, task 02) is an optional
chunk-position hook for the file currently being indexed: while that
file's content chunks embed, it is called — through
:meth:`app.rag.llm.LLMClient.embed`'s ``on_progress`` — before EACH
embed batch with ``(first_chunk_1based, total_chunks)``: the 1-based
position of the first chunk of the in-flight batch and the file's
total content-chunk count (batch granularity, the phase-129 A6
assumption — the transport embeds token-budgeted batches per HTTP
call). It carries the in-flight FILE's chunk position only, never
the file identity — the per-file ``progress`` hook owns that: the
caller pairs them (a ``progress`` call names the file, and every
chunk position reported until the next ``progress`` call belongs to
it; the position climbs through the file's batches and restarts at
1 for the next file). Same "only when provided" rule as
``progress``: ``None`` (every existing caller — the CLI, the chat
path) is a zero-cost guarded no-op, and the run is byte-identical
to a hook-less run. Only the content-chunk embeds report (the
shared :func:`_replace_content_chunks` — text, image, and PDF
branches; the image branch's one description chunk reports
``1/1``, which IS its only chunk): the document's summary-chunk
embed (:func:`_store_summary`) NEVER receives the hook, so the
counter counts content chunks only (no ``1/1`` noise after the
real progress ends), and the post-index enrichment pass / the
PDF's embedded-image docs (separate documents, not the walked
file) do not report either.
``ignore_by_root`` (phase 89, A1/A2) maps ``str(root)`` — the root path
string exactly as passed in *sources* — to that source's RAW ignore-path
lines (the importer normalizes them via ``_ignore_for_root``, the single
@@ -574,6 +611,7 @@ async def import_sources(
await _index_file(
session, source=source, rel=rel, full_path=path, llm=llm,
summary=summary, raw_date=dates_map.get(rel),
on_chunk=on_chunk_progress,
)
except EmbeddingError as e:
# A pathological file (e.g. content the embedding endpoint
@@ -608,7 +646,13 @@ async def import_sources(
async def _replace_content_chunks(
session: Session, *, doc: Document, llm: Embedder, content: str, rel: str
session: Session,
*,
doc: Document,
llm: Embedder,
content: str,
rel: str,
on_chunk: Callable[[int, int], None] | None = None,
) -> int:
"""Phase 1+2 on *content*: chunk it (format-aware, PLAN §5), replace
the doc's content chunk rows (embeddings NULL), and embed the new
@@ -622,6 +666,20 @@ async def _replace_content_chunks(
verbatim by the text path (:func:`_index_file`), the image path
(task 03), and the task-04 enrichment pass — one copy, three call
sites.
``on_chunk`` (phase 129, task 02) is the in-flight file's
chunk-position hook, forwarded as ``on_progress`` to the content
embed: it fires before each top-level embed batch with
``(first_chunk_1based, total_chunks)``. The token-cap re-chunk retry
(the ``while True`` above that halves *target*) re-embeds the NEW
(smaller-target) chunk list, and the hook reports against the NEW
total automatically — no special case: the announced total is
always the chunk list actually in flight. The empty-content
``doc.chunks = []`` break never embeds, so the hook never fires for
it. ``None`` (the default; every pre-phase call site — the CLI, the
phase-125 enrichment pass, and the PDF's embedded-image docs, which
index separate documents rather than the walked file) keeps the call
byte-identical.
"""
settings = llm.settings
target = max(400, settings.chunk_target_chars)
@@ -635,7 +693,12 @@ async def _replace_content_chunks(
if not doc.chunks:
break
try:
vectors = await llm.embed([c.content for c in doc.chunks])
# Phase 129 (task 02): the in-flight file's chunk-position
# hook (``on_progress`` — the task-01 contract; ``None`` is a
# zero-cost no-op).
vectors = await llm.embed(
[c.content for c in doc.chunks], on_progress=on_chunk
)
for row, vec in zip(doc.chunks, vectors, strict=True):
row.embedding = vec
break
@@ -660,6 +723,7 @@ async def _index_file(
llm: Embedder,
summary: ImportSummary,
raw_date: datetime | None = None,
on_chunk: Callable[[int, int], None] | None = None,
) -> None:
"""Upsert one file: doc row + chunk rows + embeddings, one transaction.
@@ -668,6 +732,15 @@ async def _index_file(
map, or ``None`` (every non-git case): the file's mtime is read
here, once, and becomes the source date (the D2 fallback).
``on_chunk`` (phase 129, task 02) is the in-flight file's
chunk-position hook (see :func:`import_sources`) — passed through to
ALL THREE branches' content embeds: the text path's
:func:`_replace_content_chunks` directly, the image branch via
:func:`_index_image_file` (its one description chunk reports
``1/1`` — it IS the file's only chunk), and the PDF branch via
:func:`_index_pdf_file` (its content embed is the primary target).
``None`` (the default) keeps every branch byte-identical.
Image files (phase 122, toggle on) delegate to
:func:`_index_image_file` — the binary path (bytes digest,
persistent copy, ``content`` = the vision description) — BEFORE
@@ -683,7 +756,7 @@ async def _index_file(
if match_extension(full_path, image_set) is not None:
return await _index_image_file(
session, source=source, rel=rel, full_path=full_path, llm=llm,
summary=summary, raw_date=raw_date,
summary=summary, raw_date=raw_date, on_chunk=on_chunk,
)
# Phase 126: PDF files take the PDF index path — AFTER the image
# branch, BEFORE any text read (a PDF is binary; ``extract_pdf_text``
@@ -691,7 +764,7 @@ async def _index_file(
if full_path.suffix.lower() == ".pdf":
return await _index_pdf_file(
session, source=source, rel=rel, full_path=full_path, llm=llm,
summary=summary, raw_date=raw_date,
summary=summary, raw_date=raw_date, on_chunk=on_chunk,
)
content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
@@ -793,7 +866,7 @@ async def _index_file(
# lives in the shared helper — the global policy stays intact for the
# rest of the KB).
chunks_n = await _replace_content_chunks(
session, doc=doc, llm=llm, content=content, rel=rel
session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
)
session.commit()
@@ -952,6 +1025,7 @@ async def _index_pdf_file(
llm: Embedder,
summary: ImportSummary,
raw_date: datetime | None = None,
on_chunk: Callable[[int, int], None] | None = None,
) -> None:
"""The phase-126 PDF branch of :func:`_index_file` — a PDF is
indexed from its EXTRACTED TEXT (LOCKED A1/A2/A4), one transaction
@@ -985,6 +1059,14 @@ async def _index_pdf_file(
``raw_date`` follows the text path: the caller's git last-commit
date or ``None`` (the mtime fallback, read before the unchanged
early-return — the unchanged path refreshes from the same source).
``on_chunk`` (phase 129, task 02) is the in-flight file's
chunk-position hook: it is forwarded to the PDF's CONTENT embed
(the :func:`_replace_content_chunks` call — the primary target),
but NOT to the embedded-image pass (those image docs are separate
documents — their ``1/1`` positions would be noise after the
file's real progress ends, the A6 exclusion). ``None`` keeps the
branch byte-identical.
"""
# Extract the text (fail-soft: an unopenable PDF yields "" — the
# document may still carry embedded images worth indexing).
@@ -1066,9 +1148,11 @@ async def _index_pdf_file(
# Phase 1+2 — the shared chunk/replace/embed pipeline (the page-
# aware chunker dispatches on the ``.pdf`` suffix; empty content
# clears the chunk rows and returns 0), committed atomically with
# the document.
# the document. Phase 129 (task 02): the in-flight file's
# chunk-position hook rides the content embed (the embedded-image
# pass does not receive it — separate documents, the A6 exclusion).
chunks_n = await _replace_content_chunks(
session, doc=doc, llm=llm, content=content, rel=rel
session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
)
session.commit()
@@ -1815,6 +1899,7 @@ async def _index_image_file(
llm: Embedder,
summary: ImportSummary,
raw_date: datetime | None = None,
on_chunk: Callable[[int, int], None] | None = None,
) -> None:
"""The phase-122 image branch of :func:`_index_file` — a standalone
image is indexed from its BYTES, never its text:
@@ -1855,6 +1940,13 @@ async def _index_image_file(
pre-phase). The UNCHANGED path never re-scans (the stored context
is part of the row's committed content; a changed source state
re-indexes the file through this path).
``on_chunk`` (phase 129, task 02) is the in-flight file's
chunk-position hook: it is forwarded to the description's content
embed (:func:`_replace_content_chunks`). A description (enriched or
not) chunks to ONE chunk, so the hook reports ``1/1`` — acceptable,
it IS the file's only chunk. ``None`` keeps the branch
byte-identical.
"""
settings = llm.settings
data = full_path.read_bytes()
@@ -1996,8 +2088,10 @@ async def _index_image_file(
# the chunk rows (embeddings NULL), embed, commit — one transaction
# per file; the endpoint token-cap retry lives in the shared helper,
# and a description is short, so it never fires in practice).
# Phase 129 (task 02): the in-flight file's chunk-position hook
# rides this embed (one description chunk → ``1/1``).
chunks_n = await _replace_content_chunks(
session, doc=doc, llm=llm, content=content, rel=rel
session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
)
session.commit()
app/rag/llm.py
+34 -3
View File
@@ -22,7 +22,7 @@ from __future__ import annotations
import asyncio
import json
import logging
from collections.abc import AsyncGenerator
from collections.abc import AsyncGenerator, Callable
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any, Literal, cast
@@ -343,12 +343,43 @@ class LLMClient:
batches.append(cur)
return batches
async def embed(self, texts: list[str]) -> list[list[float]]:
"""Embed *texts* in token-budgeted batches (order always kept)."""
async def embed(
self,
texts: list[str],
*,
on_progress: Callable[[int, int], None] | None = None,
) -> list[list[float]]:
"""Embed *texts* in token-budgeted batches (order always kept).
Progress hook (phase 129, task 01): when *on_progress* is given,
it is called once per TOP-LEVEL batch, BEFORE the batch's request
(a failing batch still announces its start), with
``(first_chunk_1based, total_chunks)`` — the 1-based index of the
first text of the in-flight batch and the total number of texts
in this call — so a caller can render ``chunk n/m`` (the importer
passes this through for the sync label; PLAN §7.4 "never stale").
``len(out) + 1`` is that first index: each batch's vectors extend
the accumulator 1:1, so ``len(out)`` is exactly the number of
chunks embedded by earlier batches. The :meth:`_embed_batch`
halving backstop does NOT sub-report: a halved batch leaves the
announced position approximate (the hook already fired for the
original batch's start) — documented; the backstop is rare by
construction (phase 114's ``_TooLarge`` handling). ``None`` (the
default — every pre-phase caller: the chat retrieval path,
:meth:`embed_one`, the summarizer, the importer) is a zero-cost
guarded no-op: call sequence and return value are
byte-identical to the pre-hook client.
"""
if not texts:
return []
out: list[list[float]] = []
for chunk in self._batch_texts(texts):
# Phase 129 (task 01): announce THIS batch's start before the
# request — the hook precedes the try, so even a failing batch
# reports its position.
if on_progress is not None:
on_progress(len(out) + 1, len(texts))
try:
vecs = await self._embed_batch(chunk)
except EmbeddingError:
frontend/assets/sources.js
+78 -33
View File
@@ -629,7 +629,10 @@ export async function mount(root) {
* Elements: #sync-btn (the button), #sync-label (the text —
* Phase 128 (task 01): the PARTS — .sync-label-main, the
* shrinks-to-fit prefix + file/folder, + .sync-label-count, the
* pinned "(n/m)" position (hidden when absent)), #sync-icon (the
* pinned "(n/m)" position (hidden when absent); phase 129 (task
* 04): + .sync-label-chunk, the pinned "· chunk n/m" embed-batch
* position of the in-flight file (hidden in every state but the
* import phase after the file's first batch)), #sync-icon (the
* spinner icon), #sync-result (aria-live result line),
* #sync-error-banner / #sync-error-text (error banner).
*/
@@ -637,6 +640,7 @@ export async function mount(root) {
const syncLabel = syncBtn ? syncBtn.querySelector(".sync-label") : null;
const syncLabelMain = syncLabel ? syncLabel.querySelector(".sync-label-main") : null;
const syncLabelCount = syncLabel ? syncLabel.querySelector(".sync-label-count") : null;
const syncLabelChunk = syncLabel ? syncLabel.querySelector(".sync-label-chunk") : null;
const syncIcon = syncBtn ? syncBtn.querySelector(".sync-icon") : null;
const syncResult = root.querySelector("#sync-result");
const syncErrorBanner = root.querySelector("#sync-error-banner");
@@ -665,22 +669,27 @@ export async function mount(root) {
return `${pad(d.getHours())}:${pad(d.getMinutes())}`;
}
/* Phase 128 (task 01): the label PARTS writer — the SINGLE point
* that writes #sync-label. mainText lands in the MAIN part
* (.sync-label-main — the shrinks-to-fit prefix + file/folder, the
* part CSS ellipsizes); countText lands in the COUNT part
* (.sync-label-count — the pinned "(n/m)" position, NEVER clipped):
* present → its textContent + hidden=false, absent → emptied +
* hidden=true (the count span is hidden whenever there is no
* "(n/m)" to show). Then the UNTRUNCATED full label —
* mainText + (countText ? " " + countText : "") — rides the button
* title + #sync-result (the aria-live announcer) exactly as before
* (A4 — the screen reader hears the whole thing; the visible span
* carries main + count, the space between them is the CSS gap).
* textContent only — the house rule, no HTML strings. Every label
* state goes through here (the running builders, the idle / settle
* labels, the 202/409 click, the re-attach). */
function setSyncLabel(mainText, countText = "") {
/* Phase 128 (task 01) + Phase 129 (task 04): the label PARTS
* writer — the SINGLE point that writes #sync-label. mainText lands
* in the MAIN part (.sync-label-main — the shrinks-to-fit prefix +
* file/folder, the part CSS ellipsizes); countText lands in the
* COUNT part (.sync-label-count — the pinned "(n/m)" position,
* NEVER clipped); chunkText (phase 129) lands in the CHUNK part
* (.sync-label-chunk — the pinned "· chunk n/m" embed-batch
* position of the in-flight file). The COUNT and CHUNK parts share
* one contract: present → its textContent + hidden=false, absent →
* emptied + hidden=true (each span is hidden whenever there is
* nothing of its kind to show). Then the UNTRUNCATED full label —
* mainText + (countText ? " " + countText : "") + (chunkText ?
* " " + chunkText : "") — rides the button title + #sync-result
* (the aria-live announcer) exactly as before (A4 — the screen
* reader hears the whole "Syncing… <file> (2/3) · chunk 14/38";
* the visible spans carry main + count + chunk, the spaces between
* them are the CSS gap). textContent only — the house rule, no
* HTML strings. Every label state goes through here (the running
* builders, the idle / settle labels, the 202/409 click, the
* re-attach). */
function setSyncLabel(mainText, countText = "", chunkText = "") {
if (syncLabelMain) syncLabelMain.textContent = mainText;
if (syncLabelCount) {
if (countText) {
@@ -691,7 +700,19 @@ export async function mount(root) {
syncLabelCount.hidden = true;
}
}
const full = countText ? `${mainText} ${countText}` : mainText;
if (syncLabelChunk) {
if (chunkText) {
syncLabelChunk.textContent = chunkText;
syncLabelChunk.hidden = false;
} else {
syncLabelChunk.textContent = "";
syncLabelChunk.hidden = true;
}
}
const full =
mainText +
(countText ? ` ${countText}` : "") +
(chunkText ? ` ${chunkText}` : "");
if (syncBtn) syncBtn.title = full; // A4: the untruncated label rides the title, always
if (syncResult) syncResult.textContent = full; // A4: the announcer hears the whole thing
}
@@ -713,7 +734,11 @@ export async function mount(root) {
* post-import phases (overview / summaries) take their own label
* via fmtSyncPhaseLabel (below) — this builder keeps that
* prelude/import fall-through byte-identical and serves the UPLOAD
* job unchanged (its status has no phase — D2). */
* job unchanged (its status has no phase — D2). Phase 129 (task
* 04): this builder still returns NO chunk part — the pinned
* "· chunk n/m" part is added by fmtSyncPhaseLabel's import
* fall-through (the only site that sees the status's chunk
* fields), so the upload label and the 202/409 click stay bare. */
function fmtSyncLabel(kind, currentFile, done, total) {
const prefix = kind === "upload" ? "Importing" : "Syncing…";
const main = currentFile ? `${prefix} ${currentFile}` : prefix;
@@ -729,26 +754,42 @@ export async function mount(root) {
* <source/folder>" + count "(n/m)" — the folder part is omitted
* while current_summary is null (the phase's first poll) and the
* COUNT IS ALWAYS EMITTED in this phase (the hook always carries a
* total); anything else (the null prelude, "import") → the
* phase-64 fmtSyncLabel parts, byte-identical (same builder, same
* fields, same order). The UPLOAD job never passes a status (its
* endpoint has no phase — D2) and keeps the bare fmtSyncLabel
* parts. The untruncated full label rides the button title +
* #sync-result like every running label (A4 — setSyncLabel). */
* total) and the CHUNK part stays empty (the summaries phase's own
* (n/m) is its position — the status's chunk fields are cleared at
* this transition); anything else (the null prelude, "import") →
* the phase-64 fmtSyncLabel parts, byte-identical (same builder,
* same fields, same order) + the phase-129 CHUNK part: "· chunk
* n/m" ONLY while chunks_total > 0 (a file's embed batch is in
* flight — n the 1-based first chunk of the batch, m the file's
* total content chunks, batch granularity), empty before the first
* batch (0/0) so the part hides. The UPLOAD job never passes a
* status (its endpoint has no phase — D2) and keeps the bare
* fmtSyncLabel parts. The untruncated full label rides the button
* title + #sync-result like every running label (A4 —
* setSyncLabel). */
function fmtSyncPhaseLabel(status) {
if (status.phase === "overview") {
return { main: "Writing KB overview…", count: "" };
return { main: "Writing KB overview…", count: "", chunk: "" };
}
if (status.phase === "summaries") {
const folder = status.current_summary ? ` ${status.current_summary}` : "";
return {
main: `Summarizing folders…${folder}`,
count: `(${status.summaries_done}/${status.summaries_total})`,
chunk: "", // this phase's own (n/m) is its position — no chunk part
};
}
// null phase (the prelude) or "import" — today's byte-identical
// sync label (the phase-64 contract, unchanged).
return fmtSyncLabel("sync", status.current_file, status.files_done, status.files_total);
// sync label (the phase-64 contract, unchanged) + the phase-129
// chunk part (hidden before the in-flight file's first batch).
const { main, count } = fmtSyncLabel(
"sync", status.current_file, status.files_done, status.files_total
);
const chunk =
status.chunks_total > 0
? `· chunk ${status.current_chunk}/${status.chunks_total}`
: "";
return { main, count, chunk };
}
function fmtSyncResult(detail) {
@@ -780,19 +821,23 @@ export async function mount(root) {
* writer (setSyncLabel) lands them — the main part (CSS-ellipsized),
* the pinned count part (hidden when absent), and the FULL
* untruncated label on the button title + #sync-result (A4 — set
* for EVERY running label; CSS ellipsizes the main part only). Same
* never-stale mechanics as before (disabled, aria-busy, spinning
* icon, no is-error).
* for EVERY running label; CSS ellipsizes the main part only).
* Phase 129 (task 04): the third, pinned CHUNK part ("· chunk n/m"
* — the in-flight file's embed-batch position) is destructured the
* same way (absent from the bare builders — the upload job and the
* 202/409 click — so the "" default keeps the part hidden there).
* Same never-stale mechanics as before (disabled, aria-busy,
* spinning icon, no is-error).
*/
function enterSyncRunningState(kind, currentFile, done, total, status) {
if (!syncBtn) return;
syncBtn.disabled = true;
syncBtn.setAttribute("aria-busy", "true");
const { main, count } =
const { main, count, chunk = "" } =
kind === "sync" && status
? fmtSyncPhaseLabel(status)
: fmtSyncLabel(kind, currentFile, done, total);
setSyncLabel(main, count); // the parts + the A4 untruncated title / announcer
setSyncLabel(main, count, chunk); // the parts + the A4 untruncated title / announcer
syncBtn.setAttribute("aria-label", "Sync sources");
syncBtn.classList.remove("is-error");
if (syncIcon) syncIcon.classList.add("is-spinning");
frontend/assets/styles.css
+15 -4
View File
@@ -2064,10 +2064,14 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und
<file> (n/m)") — a long source/relative/path ellipsizes inside the
pill; the full path lives in the button title + #sync-result (the
aria-live announcer). Phase 128 (task 02): the label is a FLEX ROW
of two parts — .sync-label-main (the prefix + file/folder: min-width
of parts — .sync-label-main (the prefix + file/folder: min-width
0, the shrinkable item whose ellipsis engages) + .sync-label-count
(the pinned "(n/m)": flex 0 0 auto, NEVER clipped — the
.sync-result mono language, a position reads as data). */
.sync-result mono language, a position reads as data). Phase 129
(task 04): a THIRD pinned part — .sync-label-chunk (the "· chunk
n/m" embed-batch position of the in-flight file, the same pinned
mono language; hidden in every state but the import phase after
the file's first batch). */
.sync-label {
display: inline-flex;
align-items: baseline;
@@ -2086,6 +2090,11 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und
white-space: nowrap;
font-family: var(--mono); /* the .sync-result mono language (position reads as data) */
}
.sync-label-chunk {
flex: 0 0 auto; /* NEVER clipped — the · chunk n/m position is the point (phase 129) */
white-space: nowrap;
font-family: var(--mono); /* the same pinned-part language as the count (position reads as data) */
}
/* Last-result announcer — soft ink, small mono */
.sync-result {
display: block;
@@ -5329,8 +5338,10 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und
width; the History page-head already wraps the pill below its
title block. The Sync label's min(16rem, 40vw) cap lifts and the
label FILLS the pill (phase 128: flex 1 1 auto) — .sync-label-main
ellipsizes against the width left by the pinned .sync-label-count,
so the (n/m) stays on-screen at the right edge, never clipped;
ellipsizes against the width left by the pinned .sync-label-count
(+ the phase-129 .sync-label-chunk part while a file's embed batch
is in flight), so the position parts stay on-screen at the right
edge, never clipped;
the refresh glyph joins its visible label. The aria-labels keep
the accessible names. */
.page-head-row { flex-wrap: wrap; }
frontend/index.html
+7 -3
View File
@@ -448,7 +448,7 @@
<h1>Knowledge base</h1>
<button type="button" class="sync-btn" id="sync-btn" aria-label="Sync sources" hidden>
<svg class="sync-icon" aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M21 12a9 9 0 1 1-9-9c2.52 0 4.93 1 6.74 2.74L21 8"/><path d="M21 3v5h-5"/></svg>
<span class="sync-label" id="sync-label"><span class="sync-label-main">Sync sources</span><span class="sync-label-count" hidden></span></span>
<span class="sync-label" id="sync-label"><span class="sync-label-main">Sync sources</span><span class="sync-label-count" hidden></span><span class="sync-label-chunk" hidden></span></span>
</button>
</div>
<p class="page-sub">
@@ -460,9 +460,13 @@
<!-- #sync-result is the aria-live announcer: the last sync
result ("N added · …") when a sync settles, and — phase 64 —
the LIVE file label while a sync runs ("Syncing… <file>
(n/m)"), UNTRUNCATED (the button's label span ellipsizes;
(n/m)"), UNTRUNCATED (the button's label spans ellipsize;
screen readers hear the full source/relative path, which
also rides the button title). Phase 90: an in-flight upload
also rides the button title). Phase 129: during the import
phase the label gains the pinned chunk part ("· chunk n/m"
— the file's embed-batch position, batch granularity) and
the announcer/title carry it too ("Syncing… <file> (n/m)
· chunk n/m"). Phase 90: an in-flight upload
run adopts the button with the BARE "Importing…" label (the
run is unpack + register only — no scan, so its status
never carries a file or counts). After an upload settles it
tests/e2e/test_sync_chunk_progress.py
+944
View File
@@ -0,0 +1,944 @@
"""Phase 129 task 05 E2E (Playwright, mock-only): while a file imports,
the sync status (endpoint + button label) shows the file's
embedding-chunk position — ``Syncing… chunkprog/big.md (1/2) · chunk
14/38`` — with ``chunks_total`` the file's TRUE content-chunk count and
``current_chunk`` the 1-based first chunk of the in-flight
token-budgeted batch (batch granularity, assumption A6), climbing 1 →
N through the file's embed batches, restarting per file, and cleared
(0/0, chunk part hidden) at every point the import phase ends.
The dedicated story suite for ``129_sync_chunk_progress`` (A16 — one
Playwright file per phase, run in isolation): the owner's report was
that ``Syncing… name/of/file/here (2/3)`` "needs to show what chunk
we're on out of how many".
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic mock plus the phase-64 ``tests/e2e/slow_llm.py`` delay
proxy that stretches the run past the recorder's ~100 ms cadence and
the UI's 2 s status poll, so every batch boundary and the rendered
labels are observed deterministically, not as races.
Timing fixture (the phase-98 sizing rationale, re-sized for batches):
``SLOW_DELAY_S = 1.5`` — the proxy delays EVERY LLM call, embeds
included, so each in-flight embed batch's position holds for ~1.5 s:
that outlives the tight-poll recorder's ~100 ms cadence by ~15× (every
batch boundary is densely sampled — the whole import span is sampled,
not just its ends), and the big file's import span (~30 batches
≈ 45 s) outlives the UI's 2 s poll ~20× (any window ≥ the poll period
catches a tick — the rendered-label assertions are deterministic, not
races). The whole changed sync is ≈ 35 LLM calls (2 model probes +
30 big-file embed batches + 1 small-file embed + 1 overview +
1 folder summary) ≈ 55–60 s — inside the generous settle budgets.
Sizing (why the file is ~60 KB): ``big.md`` is ~60 KB of markdown
(20 ``##`` sections × 3 ~1 KB paragraphs) → 61 content chunks at the
app's (code-default, pinned) 1200-char effective target — well over
the 30-chunk bar — which the token-budgeted batcher groups into ~30
requests of 2–3 chunks each: many enough batches that the climb is a
real sequence, few enough that the run stays inside the budgets. The
expected chunk count is computed IN THE TEST by calling
``chunk_document`` — the SAME function (and arguments: the importer's
``max(400, target)`` floor + the overlap setting) the importer's
``_replace_content_chunks`` uses — on the fixture file read back from
disk, and the expected batch boundaries by ``LLMClient._batch_texts``
on that list (the task-01 unit pattern). The three sizing knobs are
pinned to the code defaults in the app-under-test env (the leak-guard
pattern — an operator's gitignored ``.env`` tuning
``BOR_CHUNK_TARGET_CHARS`` must not change the suite's ground truth).
The suite runs ONE module app (the phase-79 full-inventory pattern —
it binds its own port) over the shared E2E Postgres: the SLOW app
(``BOR_LLM_BASE_URL`` = the slow proxy in front of the mock).
KB fixture — a host temp dir (``tmp_path_factory``; the app runs on
the same host) registered as a ``kind=local`` source (the
``test_local_directory_sources.py`` / phase-94 API-registration
pattern; no git anywhere): ``chunkprog/`` with EXACTLY TWO files, both
root-level (no subfolders → the summaries candidate set is just the
source root): ``big.md`` (the ~60 KB file above) + ``small.md`` (one
tiny chunk). The sorted walk processes ``big.md`` FIRST, so the file
counter reads ``(1/2)`` during the big span and ``(2/2)`` during the
small one — and the small file's single chunk is the proof that the
chunk position RESTARTS per file (the big file ends at its last
batch's first index, the small file reports ``(1, 1)``).
Autouse cleanup (the phase-96 pattern): before each test wait for no
running sync on the app, then truncate the shared registry + KB
tables (documents/chunks/query_log/steering_notes/kb_overview/
git_sources/folder_summaries) — each test seeds its own source and
runs its own sync; after each test the same truncate.
Test → observable mapping (Playwright Mapping Rule):
1. ``test_endpoint_reports_the_chunk_position`` (endpoint — the slow
leg, the phase-64 tight-poll ~100 ms recorder on
``GET /api/sync/status``): the prelude ticks are phase-less with
the chunk fields 0/0; while ``phase == "import"`` and
``current_file == "chunkprog/big.md"`` — ``files_done/total ==
1/2`` — every tick after the first batch has
``chunks_total == <the deterministic chunk_document count>`` and
the deduped ``current_chunk`` sequence climbs STRICTLY from 1,
every value one of the expected batch first-indices (batch
granularity — assert the climb + the total, not a contiguous run),
and the LAST tick before the file ends carries the final batch's
first index (which equals ``chunks_total`` only when the last
batch is a single chunk); while the tiny second file is in flight
(``files_done/total == 2/2``) the position RESTARTS — a later tick
has ``current_chunk == 1`` with ``chunks_total == 1``; the
overview + summaries phase ticks are already cleared (0/0 — the
import phase ends at the transition); the terminal has
``current_chunk == 0`` and ``chunks_total == 0`` while
``files_done == files_total == 2`` (the keep-final-file-counts
split — the file counts keep their final values, the sub-file
chunk position clears with the file, phase 129).
2. ``test_label_shows_the_chunk_part`` (UI — the slow leg, the page's
own 2 s poll): after clicking **Sync sources**, the page's own
poll renders the import span's ``#sync-label`` parts —
``.sync-label-main`` == ``Syncing… chunkprog/big.md``,
``.sync-label-count`` == ``(1/2)``, and ``.sync-label-chunk``
VISIBLE matching ``· chunk \\d+/\\d+`` with the total == the
expected chunk count — and the button ``title`` (and
``#sync-result``, the aria-live announcer) carries the UNTRUNCATED
concatenation main + " " + count + " " + chunk (A4 — asserted
atomically in the same in-browser evaluation, so the parts + title
can never straddle a poll tick); after the file's embed span the
CHUNK PART IS CLEARED — once the label has left the import phase
(overview / summaries / the terminal), ``.sync-label-chunk`` is
hidden + empty while the label itself keeps rendering in sync
(the title tracks the rendered parts, A4); on settle the button
reads the terminal ``Synced HH:MM`` (the phase-32 contract) with
the count + chunk parts hidden and ``#sync-result`` carrying the
counts line.
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import threading
import time
from collections.abc import Iterator
from datetime import datetime
from pathlib import Path
from typing import Any
import httpx
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.rag.chunker import chunk_document
from app.rag.llm import LLMClient
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
MOCK_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module's app binds its own port
# instead (a same-port second uvicorn dies on bind and would drive the
# wrong server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_CHUNKPROG", "8165"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
#: The slow-LLM proxy's port (the conftest's mock LLM stays on
#: MOCK_PORT — the proxy fronts it).
SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT_CHUNKPROG", "8906"))
SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}"
#: Per-LLM-request delay on the proxy — sized so each in-flight EMBED
#: BATCH's position outlives the recorder's ~100 ms cadence by ~15×
#: (the phase-64/98 rationale): the proxy delays EVERY call — embeds
#: included — so the whole import span is densely sampled, and the
#: big file's span (~45 s) outlives the UI's 2 s poll ~20× (see the
#: module docstring's timing fixture paragraph).
SLOW_DELAY_S = "1.5"
# --------------------------------------------------------------------------
# Fixture constants (deterministic, token-controlled)
# --------------------------------------------------------------------------
#: The local source — the temp directory's basename (``kind=local`` →
#: the directory's basename is the source name, phase 38).
SOURCE = "chunkprog"
#: The two fixture files (sorted walk: ``big.md`` FIRST → the file
#: counter reads (1/2) on the big span, (2/2) on the small one).
BIG_FILE = f"{SOURCE}/big.md"
SMALL_FILE = f"{SOURCE}/small.md"
N_FILES = 2
#: The big file's shape — 20 ``##`` sections × 3 ~1 KB paragraphs
#: (≈ 60 KB → 61 content chunks, 30+ at the default target; see the
#: module docstring's sizing paragraph).
N_BIG_SECTIONS = 20
N_BIG_PARAS = 3
#: The tiny second file — one chunk (``chunks_total == 1`` proves the
#: chunk position restarts per file).
SMALL_MD = "# Small\n\nA tiny second fixture file that chunks as one.\n"
#: The three sizing knobs, pinned to the CODE DEFAULTS in both the
#: app-under-test env (the leak guards below) and the test process's
#: expected-value computation — an operator's gitignored ``.env``
#: (which may legitimately tune ``BOR_CHUNK_TARGET_CHARS``) cannot
#: change the suite's ground truth. Derived from the class fields, so
#: they never drift from app/config.py (the conftest leak-guard
#: pattern).
CHUNK_TARGET = _Settings.model_fields["chunk_target_chars"].default
CHUNK_OVERLAP = _Settings.model_fields["chunk_overlap_chars"].default
EMBED_BATCH_SIZE = _Settings.model_fields["embed_batch_size"].default
#: The test process's mirror of the app-under-test sizing settings
#: (explicit kwargs beat any ``.env`` for the fields the expected
#: values depend on).
_EXPECTED_SETTINGS = _Settings(
chunk_target_chars=CHUNK_TARGET,
chunk_overlap_chars=CHUNK_OVERLAP,
embed_batch_size=EMBED_BATCH_SIZE,
)
#: "Synced HH:MM" — the local-time last-result label (sources.js's
#: fmtSyncTime), any hour/minute (the test_sync_button.py pattern).
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}$")
#: The settled boot label — idle ("Sync sources") or the RE-ATTACHED
#: terminal from the previous test's sync (the module app is shared,
#: its in-memory sync state survives the test boundary, and the
#: phase-32 boot re-attach renders the last result).
IDLE_OR_SETTLED = re.compile(r"^(Sync sources|Synced \d{1,2}:\d{2})$")
#: The post-import label mains (overview / summaries / terminal) —
#: the states AFTER the big file's embed span where the chunk part
#: must be cleared while the label keeps rendering (test 2's step 2).
POST_IMPORT_MAIN_RE = re.compile(
r"^(Writing KB overview…|Summarizing folders…(?: \S+)?|Synced \d{1,2}:\d{2})$"
)
#: The import-span chunk part (phase 129 task 04's unit-pinned shape):
#: "· chunk n/m" — ``m`` is asserted against the expected chunk count
#: by the in-browser snapshot (``n`` is batch-granular — any of the
#: expected first-indices).
CHUNK_PART_RE = re.compile(r"^· chunk (\d+)/(\d+)$")
#: Generous settle budgets: a changed sync against the slowed LLM is
#: ≈ 55–60 s; the UI's 2 s poll settles at most one tick after the
#: terminal state lands.
SETTLE_TIMEOUT_MS = 120_000
def _big_md_content() -> str:
"""The deterministic ~60 KB big fixture file (see the module
docstring's sizing paragraph): an H1 preamble (one chunk) + 20
``##`` sections of 3 ~1 KB single-line paragraphs each (the
section's paragraph packing sub-splits each into ~3 chunks — no
randomness, so the expected count is stable run to run)."""
parts = ["# Big Fixture\n"]
parts.append(
"Intro paragraph for the big chunkprog fixture file. This preamble "
"stays under the section target so it chunks as one.\n"
)
for k in range(1, N_BIG_SECTIONS + 1):
parts.append(f"\n## Section {k}\n")
for p in range(1, N_BIG_PARAS + 1):
base = (
f"Paragraph {p} of section {k} in the chunkprog big fixture "
f"carries deterministic filler number {k * 100 + p} of the "
f"large markdown file, written out to size. "
)
parts.append((base * 8)[:1050] + "\n\n")
return "".join(parts)
def _expected_big_chunks(content: str) -> list[str]:
"""The big file's expected content chunks — the SAME function and
arguments the importer's ``_replace_content_chunks`` uses
(``chunk_document(content, rel, max(400, target), overlap)`` — the
importer's token-cap floor rule; ``rel`` = the source-relative
path): the endpoint's ``chunks_total`` must equal
``len(chunk_document(...))`` — the file's TRUE chunk count, not a
batch count (the phase-129 A6 assumption)."""
settings = _EXPECTED_SETTINGS
target = max(400, settings.chunk_target_chars)
return chunk_document(content, "big.md", target, settings.chunk_overlap_chars)
def _expected_batch_firsts(chunks: list[str]) -> list[int]:
"""The expected ``current_chunk`` values — the 1-based FIRST chunk
of each token-budgeted embed batch, computed with the same
``LLMClient._batch_texts`` partition the app's embed loop walks
(the task-01 unit pattern — call it directly for the boundaries)."""
client = LLMClient(_EXPECTED_SETTINGS)
firsts: list[int] = []
acc = 0
for batch in client._batch_texts(chunks):
firsts.append(acc + 1)
acc += len(batch)
return firsts
def _dedup_consecutive(values: list[int]) -> list[int]:
"""Collapse consecutive duplicates (the ~100 ms recorder samples
each 1.5 s batch window ~15× — the CLIMB lives in the distinct
sequence)."""
out: list[int] = []
for v in values:
if not out or out[-1] != v:
out.append(v)
return out
# ---------------------------------------------------------------------------
# The in-browser snapshot functions (Playwright Mapping Rule — the
# rendered DOM is the observable).
#
# Each returns null while the awaited label state is absent, and the
# PART texts + the button title once it is present — read in the SAME
# in-browser evaluation that verified the label state (a 2 s poll tick
# updates the parts + the title together in one JS task, so the atomic
# check can never straddle one — the phase-128 pattern).
# ---------------------------------------------------------------------------
#: Test 2, step 1 — the in-flight IMPORT label with the VISIBLE chunk
#: part: the main part carries "Syncing… chunkprog/big.md" (the
#: untruncated A4 text — the big file is the first of the two, so the
#: count part is "(1/2)"), the pinned chunk part is visible matching
#: "· chunk n/m" with m == the expected chunk count (the arg), and the
#: button title + #sync-result carry the UNTRUNCATED concatenation
#: (main + " " + count + " " + chunk — A4: the announcer hears the
#: whole "Syncing… <file> (1/2) · chunk n/m").
_JS_IMPORT_CHUNK_SNAPSHOT = """
(n) => {
const label = document.querySelector('#sync-label');
const btn = document.querySelector('#sync-btn');
if (!label || !btn) return null;
const main = label.querySelector('.sync-label-main');
const count = label.querySelector('.sync-label-count');
const chunk = label.querySelector('.sync-label-chunk');
if (!main || !count || !chunk || count.hidden || chunk.hidden) return null;
if (main.textContent !== 'Syncing… chunkprog/big.md') return null;
if (count.textContent !== '(1/2)') return null;
const m = chunk.textContent.match(/^· chunk (\\d+)\\/(\\d+)$/);
if (!m || Number(m[2]) !== n) return null;
const full = main.textContent + ' ' + count.textContent + ' ' + chunk.textContent;
if (btn.getAttribute('title') !== full) return null;
const result = document.querySelector('#sync-result');
if (!result || result.textContent !== full) return null;
return {
mainText: main.textContent,
countText: count.textContent,
chunkText: chunk.textContent,
title: btn.getAttribute('title'),
resultText: result.textContent,
};
}
"""
#: Test 2, step 2 — a state AFTER the big file's embed span (the
#: overview / summaries / terminal label — the import phase is over):
#: the CHUNK PART IS CLEARED — hidden + empty — while the label
#: itself keeps rendering in sync (the title carries the rendered
#: parts' concatenation, A4; the count part is present only in the
#: summaries label).
_JS_POST_IMPORT_SNAPSHOT = """
() => {
const label = document.querySelector('#sync-label');
const btn = document.querySelector('#sync-btn');
if (!label || !btn) return null;
const main = label.querySelector('.sync-label-main');
const count = label.querySelector('.sync-label-count');
const chunk = label.querySelector('.sync-label-chunk');
if (!main || !count || !chunk) return null;
const t = main.textContent;
const postRe = /^(Writing KB overview…|Summarizing folders…(?: \\S+)?|Synced \\d{1,2}:\\d{2})$/;
if (!postRe.test(t)) return null;
if (!chunk.hidden || chunk.textContent !== '') return null;
const countText = count.hidden ? '' : count.textContent;
const full = t + (countText ? ' ' + countText : '');
if (btn.getAttribute('title') !== full) return null;
return {
mainText: t,
countText,
chunkHidden: chunk.hidden,
chunkText: chunk.textContent,
title: btn.getAttribute('title'),
};
}
"""
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def chunkprog_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The one-source temp tree (see the module docstring): the app
server runs on the same host, so the paths are visible to it.
The directory NAME is the source name (``kind=local``, phase 38).
Exactly two files: the ~60 KB ``big.md`` (30+ chunks) and the
tiny one-chunk ``small.md`` (the per-file restart proof)."""
root = tmp_path_factory.mktemp("bor_chunkprog")
src = root / SOURCE
src.mkdir()
(src / "big.md").write_text(_big_md_content(), encoding="utf-8")
(src / "small.md").write_text(SMALL_MD, encoding="utf-8")
assert len(list(src.rglob("*.md"))) == N_FILES
return src
def _app_env(base_url: str, tmp_path_factory: pytest.TempPathFactory) -> dict[str, str]:
"""The per-module app env (the conftest pattern, cf.
``test_oneshot_llm_retry.py`` / ``test_kb_tree.py``): the
``base_url`` picks the LLM (the slow proxy in front of the mock),
``BOR_GIT_SOURCES`` forced empty (the sync source is this suite's
own DB-registered local row), the sizing knobs pinned to the code
defaults (the leak guards — see the module constants), and the
other leak-guarded code defaults."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url
)
# Mock-calibrated threshold (conftest pattern): this suite never
# asks the chat model anything — the gate is never on a path.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
# Phase 67: instant retry waits + the code-default budget (the
# conftest leak-guard pattern).
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
# reads it from cwd) — override it with an EMPTY value (the env var
# beats the .env file): the registry must hold EXACTLY the local
# directory this suite registers.
env["BOR_GIT_SOURCES"] = ""
# Leak guards (conftest pattern): an operator's local (gitignored)
# .env cannot leak corpus-specific settings into the app under
# test — including the chunking / batching knobs this suite's
# ground truth is computed from (see the module constants).
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
_Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
env["BOR_CHUNK_TARGET_CHARS"] = str(CHUNK_TARGET)
env["BOR_CHUNK_OVERLAP_CHARS"] = str(CHUNK_OVERLAP)
env["BOR_EMBED_BATCH_SIZE"] = str(EMBED_BATCH_SIZE)
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
return env
@pytest.fixture(scope="module")
def slow_llm(mock_llm: int) -> Iterator[int]:
"""The delay-injecting reverse proxy in front of the mock LLM
(``tests/e2e/slow_llm.py``) — the timing fixture for the whole
suite: it delays EVERY LLM call, embeds included, so each in-
flight embed batch's position outlives the recorder's ~100 ms
cadence by ~15× and the big file's import span outlives the UI's
2 s poll ~20× (see ``SLOW_DELAY_S``)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S
env["E2E_MOCK_PORT"] = str(MOCK_PORT)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app",
"--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{SLOW_URL}/v1/models")
yield SLOW_PORT
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_server(
mock_llm: int,
slow_llm: int,
chunkprog_dir: Path,
tmp_path_factory: pytest.TempPathFactory,
) -> Iterator[str]:
"""The SLOW app under test (the whole suite):
``BOR_LLM_BASE_URL`` is the slow proxy in front of the mock (the
timing fixture — see the module docstring). ``chunkprog_dir`` is a
dependency only for the fixture ordering (the temp tree exists
before the first test)."""
env = _app_env(f"{SLOW_URL}/v1", tmp_path_factory)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
share one Postgres, so a leftover git_sources row or document would
pollute the file counts the syncs and the chunk-position
assertions pin exactly."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources, folder_summaries"
)
)
db.commit() # without the commit the TRUNCATE rolls back (the house pattern)
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
"""No background sync may leak across tests (the run state lives in
the app's memory, and a still-running run would keep importing into
the NEXT test's truncated KB): wait for the app's sync status to
be non-running BEFORE the truncate. Own admin session (the
endpoint is admin-only) — usually a no-op: the tests settle only
after their run's terminal state."""
with httpx.Client(base_url=base_url, timeout=5.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
if r.json()["state"] != "running":
return
time.sleep(0.2)
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
@pytest.fixture(autouse=True)
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
"""Per-test KB isolation (the phase-96 cleanup pattern): the
shared E2E Postgres is truncated before AND after every test —
each test seeds its own source and runs its own sync, so the KB
rows + folder_summaries rows of the temp source are exactly this
test's own. The pre-wait keeps a leaked run from the previous
test (the module app is shared) from racing the truncate."""
_wait_sync_settled(app_url)
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _register_source(base_url: str, path: str) -> None:
"""Register the temp directory through the authenticated API (the
``test_local_directory_sources.py`` / phase-94 pattern — a plain
directory is an API/DB-only operation)."""
with httpx.Client(base_url=base_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.post(
"/api/git-sources", json={"kind": "local", "path": path}
)
assert r.status_code == 201, r.text
class _TickRecorder:
"""The deterministic layer, concurrent with the test's own client.
A daemon thread that tight-polls (~100 ms cadence)
``GET /api/sync/status`` with its OWN admin session (``httpx`` —
the test's client drives itself in the meantime; the Playwright
sync API is not thread-safe, so the thread never touches it),
recording every tick from the first poll: the idle prelude, the
running ticks, and the terminal body. The thread only READS the
same endpoint the UI's 2 s poll reads — it starts no jobs and
cannot skew the run. (The phase-64 recorder pattern.)
The stale-terminal guard (the module app is shared — a previous
test's terminal lives in the app's memory): a terminal is
accepted only after a RUNNING tick was observed (this suite's runs
outlive the 100 ms cadence by ~15×)."""
def __init__(self, app_url: str, path: str) -> None:
self._url = f"{app_url}{path}"
self._login_url = f"{app_url}/api/login"
self._t0 = time.time()
self._ticks: list[dict[str, Any]] = []
self._terminal: dict[str, Any] | None = None
self._stop = threading.Event()
self._thread: threading.Thread | None = None
def _terminal_is_this_run(self, body: dict[str, Any], saw_running: bool) -> bool:
if saw_running:
return True
started = body.get("started_at")
if not started:
return False
try:
started_at = datetime.fromisoformat(str(started)).timestamp()
except ValueError:
return False
return started_at >= self._t0 - 2.0 # tolerance for the pre-submit gap
def start(self) -> None:
def run() -> None:
with httpx.Client(timeout=5.0) as client:
# Its own admin session — the recorder's reads must not
# depend on (or disturb) the test's client's cookie.
client.post(self._login_url, json={"password": ADMIN_PASSWORD})
saw_running = False
while not self._stop.is_set():
try:
r = client.get(self._url)
if r.status_code == 200:
body = r.json()
self._ticks.append(body)
if body["state"] == "running":
saw_running = True
elif (
body["state"] in ("success", "failed")
and self._terminal_is_this_run(body, saw_running)
):
self._terminal = body
return
except Exception: # noqa: BLE001 — blip: retry next tick
pass
self._stop.wait(0.1)
self._thread = threading.Thread(target=run, daemon=True)
self._thread.start()
def stop(self, timeout_s: float = 180.0) -> dict[str, Any]:
"""Join until a terminal tick is recorded (or fail the test)."""
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
if self._terminal is not None:
break
time.sleep(0.05)
self._stop.set()
if self._thread is not None:
self._thread.join(timeout=5)
assert self._terminal is not None, (
"the recorder saw no terminal state "
f"(last ticks: {self._ticks[-3:] if self._ticks else 'none'})"
)
return self._terminal
@property
def running_ticks(self) -> list[dict[str, Any]]:
return [t for t in self._ticks if t["state"] == "running"]
def _start_sync(page: Page, app_url: str, sync_dir: Path) -> None:
"""Register the source, land the admin on the sources view, wait
for the settled boot label (the phase-64 pattern — also proves the
phase-32 boot re-attach completed, so no later re-render can
clobber the running state), then click **Sync sources** and pin
the click's immediate 202 state (the bare "Syncing…" — the count
AND the chunk parts hidden: no file yet, no batch in flight)."""
_register_source(app_url, str(sync_dir))
login(page, app_url) # lands on /sources.html (the button's home)
btn = page.locator("#sync-btn")
expect(btn).to_be_visible(timeout=30_000)
expect(page.locator("#sync-label")).to_have_text(IDLE_OR_SETTLED)
expect(btn).to_be_enabled()
btn.click()
# The click's immediate state (the 202 moment — the bare prefix,
# the phase-64 contract): disabled, aria-busy, spinning icon, and
# the label's MAIN part alone ("Syncing…" — the 202 click carries
# no status, so both pinned parts hide, phase 128/129 writers).
expect(btn).to_be_disabled()
expect(btn).to_have_attribute("aria-busy", "true")
expect(btn.locator(".sync-icon")).to_have_class(re.compile(r"\bis-spinning\b"))
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator(".sync-label-count")).to_be_hidden()
expect(page.locator(".sync-label-chunk")).to_be_hidden()
expect(page.locator("#sync-error-banner")).to_be_hidden()
def _wait_terminal(app_url: str, timeout_s: float = 180.0) -> dict[str, Any]:
"""Poll the status endpoint (own admin session) until the UI-
triggered run reaches a terminal state — the source of truth for
the settle (the UI's 2 s poll renders it at most one tick later).
A FAILED terminal fails the test with the terminal body (the
``detail`` / error text) — never an opaque label timeout."""
with httpx.Client(base_url=app_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
body = r.json()
if body["state"] in ("success", "failed"):
assert body["state"] == "success", body
return body
time.sleep(0.2)
raise AssertionError(f"the run never reached a terminal state: {body}")
# ---------------------------------------------------------------------------
# 1. Endpoint — the chunk position climbs 1 → N through the big file's
# embed batches (chunks_total = the file's true chunk count),
# restarts per file, and clears at every point the import ends
# ---------------------------------------------------------------------------
def test_endpoint_reports_the_chunk_position(
app_url: str, chunkprog_dir: Path, db_ready: None
) -> None:
"""A KB-changing sync of the two-file temp source is tight-polled
by the ~100 ms recorder from the 202 to the terminal (the
phase-64 pattern): the recorded ticks show the phase-129 chunk
position end to end — the prelude is 0/0, the big file's import
span reports ``chunks_total == <the deterministic chunk_document
count>`` with ``current_chunk`` climbing strictly 1 → the final
batch's first index (batch granularity — every recorded value is
one of the expected ``_batch_texts`` boundaries, the sequence's
LAST value before the file ends is the final batch's first index),
the tiny second file RESTARTS the position (a later tick has
``current_chunk == 1`` with ``chunks_total == 1``), the
overview/summaries ticks are already cleared (0/0 at the
transitions), and the terminal keeps ``files_done == files_total
== 2`` while the chunk position is 0/0 (the keep-final-file-
counts split — the file counts survive, the sub-file position
clears with the file)."""
# The ground truth — computed from the fixture file ON DISK with
# the SAME function + arguments the importer uses (see the helper
# docstrings): the true chunk count + the batch first-indices.
big_content = (chunkprog_dir / "big.md").read_text(encoding="utf-8")
big_chunks = _expected_big_chunks(big_content)
n_chunks = len(big_chunks)
assert n_chunks >= 30, n_chunks # the task's sizing bar (30+ chunks)
firsts = _expected_batch_firsts(big_chunks)
assert len(firsts) >= 3, firsts # multi-batch — a real climb to assert
_register_source(app_url, str(chunkprog_dir))
recorder = _TickRecorder(app_url, "/api/sync/status")
recorder.start()
with httpx.Client(base_url=app_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.post("/api/sync")
assert r.status_code == 202, r.text
terminal = recorder.stop(timeout_s=180)
# --- the terminal: the keep-final-file-counts split — the FILE
# counts keep their final values, the sub-file chunk position is
# 0/0 (a sub-position of a file that's done — the phase-129
# convention), and the file/phase clear like phase 64/98 ---
assert terminal["state"] == "success", terminal
detail = terminal["detail"]
assert detail["added"] == N_FILES, detail
assert terminal["current_file"] is None
assert terminal["files_done"] == N_FILES
assert terminal["files_total"] == N_FILES
assert terminal["current_chunk"] == 0, terminal
assert terminal["chunks_total"] == 0, terminal
assert terminal["phase"] is None, terminal
running = recorder.running_ticks
assert running, "the recorder caught no running tick"
# --- the prelude (model check): phase null, chunk fields 0/0
# (reset with the run) — two probes × 1.5 s under the slow proxy:
# the recorder cannot miss it. ---
prelude = [t for t in running if t["phase"] is None]
assert prelude, f"no phase-less prelude tick: {running[:6]}"
assert all(t["current_file"] is None for t in prelude), prelude
assert all(
t["current_chunk"] == 0 and t["chunks_total"] == 0 for t in prelude
), prelude
imp = [t for t in running if t["phase"] == "import"]
assert imp, f"no import-phase tick: {[t['phase'] for t in running]}"
big_idx = [i for i, t in enumerate(imp) if t["current_file"] == BIG_FILE]
small_idx = [i for i, t in enumerate(imp) if t["current_file"] == SMALL_FILE]
assert big_idx, f"no big-file import tick: {[t['current_file'] for t in imp][:8]}"
assert small_idx, f"no small-file import tick: {[t['current_file'] for t in imp][:8]}"
# The sorted walk processes big.md FIRST — and the files never
# interleave: the big span ends before the small span starts.
assert max(big_idx) < min(small_idx), (max(big_idx), min(small_idx))
# --- the big file (files 1/2): chunks_total is the file's TRUE
# chunk count (not a batch count) and current_chunk climbs
# strictly through the batch boundaries ---
big_ticks = [imp[i] for i in big_idx]
assert all(
t["files_done"] == 1 and t["files_total"] == N_FILES for t in big_ticks
), big_ticks[:4]
with_total = [t for t in big_ticks if t["chunks_total"] > 0]
assert with_total, f"no tick carried a chunk total: {big_ticks[:4]}"
assert all(t["chunks_total"] == n_chunks for t in with_total), (
[t["chunks_total"] for t in with_total], n_chunks
)
# The pre-first-batch ticks (if sampled) are 0/0 — no other
# (chunk, total) combination is legal for this file.
for t in big_ticks:
if t["chunks_total"] == 0:
assert t["current_chunk"] == 0, t
seq = _dedup_consecutive([t["current_chunk"] for t in with_total])
assert seq[0] == 1, f"the climb does not start at 1: {seq}"
assert seq == sorted(set(seq)), f"the climb is not strictly increasing: {seq}"
assert set(seq) <= set(firsts), f"off-partition positions: {seq} vs {firsts}"
# The LAST tick before the file ends carries the final batch's
# first index (== chunks_total only when the last batch is a
# single chunk) — batch granularity, not a contiguous run.
assert seq[-1] == firsts[-1], (seq, firsts)
assert len(seq) >= 3, f"too few batch boundaries observed: {seq}"
# --- the tiny second file (files 2/2): the position RESTARTS —
# one chunk → the single batch reports (1, 1) ---
small_ticks = [imp[i] for i in small_idx]
assert all(
t["files_done"] == N_FILES and t["files_total"] == N_FILES
for t in small_ticks
), small_ticks[:4]
with_total_s = [t for t in small_ticks if t["chunks_total"] > 0]
assert with_total_s, f"no tick carried the small file's chunk total: {small_ticks[:4]}"
assert all(
t["current_chunk"] == 1 and t["chunks_total"] == 1 for t in with_total_s
), with_total_s[:4]
# --- the phase transitions: the import phase ends at overview /
# summaries — the chunk position is cleared (0/0) at the
# transition (the two 1.5 s spans outlive the 100 ms cadence ~15×,
# so the recorder samples both). ---
ov = [t for t in running if t["phase"] == "overview"]
sm = [t for t in running if t["phase"] == "summaries"]
assert ov, f"no overview-phase tick: {[t['phase'] for t in running]}"
assert sm, f"no summaries-phase tick: {[t['phase'] for t in running]}"
for t in [*ov, *sm]:
assert t["current_chunk"] == 0 and t["chunks_total"] == 0, t
# ---------------------------------------------------------------------------
# 2. UI — the label's chunk part: "Syncing… chunkprog/big.md (1/2) ·
# chunk n/m" while a batch is in flight, the untruncated label on
# the title + announcer, the part cleared after the file's span
# ---------------------------------------------------------------------------
def test_label_shows_the_chunk_part(
page: Page, app_url: str, chunkprog_dir: Path, db_ready: None
) -> None:
"""A fresh admin page on ``/sources.html`` clicks **Sync sources**;
the page's OWN 2 s poll renders the import span's ``#sync-label``
as the phase-128/129 PARTS — the main part
``Syncing… chunkprog/big.md``, the pinned count part ``(1/2)``,
and the VISIBLE chunk part matching ``· chunk n/m`` with the total
== the expected chunk count (the big file's span is ~45 s under
the slow proxy — the poll cannot miss it) — with the button
``title`` + ``#sync-result`` carrying the UNTRUNCATED
concatenation (A4, asserted atomically in the same in-browser
evaluation). After the file's embed span, once the label has left
the import phase (overview / summaries / terminal), the chunk part
is CLEARED — hidden + empty — while the label keeps rendering in
sync (the title tracks the rendered parts); on settle the button
reads the terminal ``Synced HH:MM`` (the phase-32 contract) with
the count + chunk parts hidden and ``#sync-result`` carrying the
counts line."""
page.set_default_timeout(30_000)
big_content = (chunkprog_dir / "big.md").read_text(encoding="utf-8")
n_chunks = len(_expected_big_chunks(big_content))
assert n_chunks >= 30, n_chunks
_start_sync(page, app_url, chunkprog_dir)
# Step 1 — the import span with the chunk part (the page's own 2 s
# poll; the big file's ~45 s span outlives the poll ~20×). The
# in-browser snapshot verifies the parts + the title +
# #sync-result TOGETHER (one JS task per poll tick — the atomic
# check can never straddle one) and returns the texts.
snap = page.wait_for_function(
_JS_IMPORT_CHUNK_SNAPSHOT, arg=n_chunks, timeout=SETTLE_TIMEOUT_MS
).json_value()
assert snap["mainText"] == f"Syncing… {SOURCE}/big.md", snap
assert snap["countText"] == "(1/2)", snap
m = CHUNK_PART_RE.match(snap["chunkText"])
assert m, snap
assert int(m.group(2)) == n_chunks, snap # the TRUE chunk count, not a batch count
assert int(m.group(1)) >= 1, snap
# A4: the untruncated concatenation rides the title AND the
# aria-live announcer (the "· chunk" part included).
full = f"Syncing… {SOURCE}/big.md (1/2) {snap['chunkText']}"
assert snap["title"] == full and "· chunk" in snap["title"], snap
assert snap["resultText"] == full, snap
# Step 2 — after the file's embed span: the chunk part is CLEARED
# (hidden + empty) once the label has left the import phase — the
# overview / summaries / terminal states — while the label keeps
# rendering (the title still tracks the rendered parts, A4). The
# terminal is persistent, so the wait terminates deterministically
# (the intermediate post-import states may be sampled or skipped
# by the 2 s poll — any of them satisfies the pin).
snap2 = page.wait_for_function(
_JS_POST_IMPORT_SNAPSHOT, timeout=SETTLE_TIMEOUT_MS
).json_value()
assert POST_IMPORT_MAIN_RE.match(snap2["mainText"]), snap2
assert snap2["chunkHidden"] is True, snap2
assert snap2["chunkText"] == "", snap2
assert (
snap2["title"]
== snap2["mainText"] + (" " + snap2["countText"] if snap2["countText"] else "")
), snap2
# The success settle (the phase-32 contract, preserved): "Synced
# HH:MM" + the counts result line, the button re-enabled, the
# count AND chunk parts hidden + empty, no error — the chunk part
# rendered ONLY during the import phase (test 1 pins the
# endpoint-side clearing at the transitions + terminal).
_wait_terminal(app_url)
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
btn = page.locator("#sync-btn")
expect(btn).to_be_enabled()
expect(btn).not_to_have_attribute("aria-busy")
expect(page.locator(".sync-label-count")).to_be_hidden()
expect(page.locator(".sync-label-count")).to_have_text("")
expect(page.locator(".sync-label-chunk")).to_be_hidden()
expect(page.locator(".sync-label-chunk")).to_have_text("")
expect(page.locator("#sync-result")).to_have_text(f"{N_FILES} added")
expect(page.locator("#sync-error-banner")).to_be_hidden()
tests/e2e/test_sync_label_mobile.py
+46 -14
View File
@@ -64,7 +64,13 @@ Test → observable mapping (Playwright Mapping Rule):
``\\(\\d+/\\d+\\)`` once a file is in flight; ``.sync-label-main``
carries ``Syncing… <file>`` — the FULL source/relative/path
(visually ellipsized, but the DOM text and the button ``title``
are untruncated, A4).
are untruncated, A4). Phase 129: the label's THIRD pinned part —
``.sync-label-chunk``, "· chunk n/m" (the in-flight file's
embed-batch position, batch granularity) — renders while a file's
embed batch is in flight (``chunks_total > 0``) and rides the
untruncated ``title`` too; the snapshot accepts it (optional —
the pre-first-batch frame has no chunk part yet; the phase-129
dedicated suite pins it strictly).
2. ``test_mobile_label_shows_summaries_count`` — same mobile
viewport; through to the summaries phase: ``.sync-label-main``
matches ``Summarizing folders…`` (the folder part optional — the
@@ -74,12 +80,14 @@ Test → observable mapping (Playwright Mapping Rule):
the running label (the phase-32 terminal ``Synced HH:MM``) and the
count span is hidden + empty.
3. ``test_desktop_label_unchanged`` — viewport 1280×800; the same
sync: during import the full rendered label text (main + count
concatenation) matches ``Syncing… <full source/relative/path>
(n/m)`` — the pre-phase byte-identical string — and the label's
width respects the ``min(16rem, 40vw)`` cap (bounding box width
≤ 256 px + slack); the ``title`` carries the same untruncated
string (A4).
sync: during import the full rendered label text (main + count +
optional phase-129 chunk concatenation) matches ``Syncing… <full
source/relative/path> (n/m)`` (the pre-phase byte-identical
string, the chunk part appended while a batch is in flight) and
the label's width respects the ``min(16rem, 40vw)`` cap (bounding
box width ≤ 256 px + slack — the shrinkable main part absorbs
the extra pinned part); the ``title`` carries the same
untruncated string (A4).
4. ``test_pre_phase_css_reproduces_the_clip`` — the pin check: the
SAME mobile run with the pre-phase-128 ``.sync-label`` rules
re-injected (CSSOM ``insertRule`` into the origin's own linked
@@ -179,8 +187,11 @@ FILE_RE = r"synclabel/(?:alpha|bravo)/(?:alpha|bravo)-[ab]\.md"
IMPORT_MAIN_RE = re.compile(rf"^Syncing… {FILE_RE}$")
IMPORT_COUNT_RE = re.compile(r"^\([1-4]/4\)$")
#: The pre-phase byte-identical FULL string (main + " " + count — the
#: single span's text before phase 128; the A4 title keeps it).
IMPORT_FULL_RE = re.compile(rf"^Syncing… {FILE_RE} \([1-4]/4\)$")
#: single span's text before phase 128; the A4 title keeps it), plus
#: the OPTIONAL phase-129 chunk part (" · chunk n/m" — the in-flight
#: file's embed-batch position, batch granularity) the title carries
#: while a file's embed batch is in flight.
IMPORT_FULL_RE = re.compile(rf"^Syncing… {FILE_RE} \([1-4]/4\)(?: · chunk \d+/\d+)?$")
#: The summaries label's PARTS (phase 98 task 02, unit-pinned): main
#: "Summarizing folders… [source/folder]" (the folder part omitted
@@ -236,7 +247,13 @@ def _md(title: str, body: str) -> str:
#: <full source/relative/path>" (the untruncated A4 text — the visual
#: ellipsis never touches textContent), the count part is VISIBLE
#: (not ``hidden``) with the "(n/4)" position, and the button title
#: carries the full untruncated label (main + " " + count, A4).
#: carries the full untruncated label (main + " " + count, A4). Phase
#: 129: the THIRD pinned part — .sync-label-chunk ("· chunk n/m", the
#: in-flight file's embed-batch position) — rides the title too while
#: a file's embed batch is in flight (chunks_total > 0): the title
#: check accepts main + " " + count (+ " " + chunk when the part is
#: rendered) — optional, since the pre-first-batch frame has no chunk
#: part (the phase-129 dedicated suite pins the part strictly).
_JS_IMPORT_SNAPSHOT = """
() => {
const label = document.querySelector('#sync-label');
@@ -248,8 +265,12 @@ _JS_IMPORT_SNAPSHOT = """
const reMain = /^Syncing… synclabel\\/(?:alpha|bravo)\\/(?:alpha|bravo)-[ab]\\.md$/;
const reCount = /^\\([1-4]\\/4\\)$/;
if (!reMain.test(main.textContent) || !reCount.test(count.textContent)) return null;
const chunk = label.querySelector('.sync-label-chunk');
const chunkText = (chunk && !chunk.hidden) ? chunk.textContent : '';
const title = btn.getAttribute('title');
if (title !== main.textContent + ' ' + count.textContent) return null;
const full = main.textContent + ' ' + count.textContent
+ (chunkText ? ' ' + chunkText : '');
if (title !== full) return null;
const rect = (el) => {
const b = el.getBoundingClientRect();
return { left: b.left, right: b.right, width: b.width, height: b.height };
@@ -257,6 +278,7 @@ _JS_IMPORT_SNAPSHOT = """
return {
mainText: main.textContent,
countText: count.textContent,
chunkText,
title,
labelRect: rect(label),
countRect: rect(count),
@@ -658,7 +680,14 @@ def test_mobile_label_fits_viewport_with_count_in_import(
# A4: the button title carries the UNTRUNCATED label — the full
# path, not the ellipsized visual tail (the in-browser check
# already required title == main + " " + count; pin it here too).
assert snap["title"] == snap["mainText"] + " " + snap["countText"], snap
# Phase 129: the pinned chunk part ("· chunk n/m") rides the title
# while a file's embed batch is in flight (optional — the
# pre-first-batch frame has none; the phase-129 dedicated suite
# pins it strictly).
full = snap["mainText"] + " " + snap["countText"]
if snap["chunkText"]:
full += " " + snap["chunkText"]
assert snap["title"] == full, snap
# The geometry contract (the fix): label inside the viewport, the
# count visible and never clipped.
_assert_count_not_clipped(snap, MOBILE_VIEWPORT["width"])
@@ -727,9 +756,12 @@ def test_desktop_label_unchanged(
_JS_IMPORT_SNAPSHOT, timeout=SETTLE_TIMEOUT_MS
).json_value()
# The pre-phase byte-identical FULL string (main + " " + count):
# "Syncing… <full source/relative/path> (n/m)".
# The pre-phase byte-identical FULL string (main + " " + count +
# the phase-129 chunk part while a batch is in flight):
# "Syncing… <full source/relative/path> (n/m)" (+ " · chunk n/m").
full = snap["mainText"] + " " + snap["countText"]
if snap["chunkText"]:
full += " " + snap["chunkText"]
assert IMPORT_FULL_RE.match(full), full
assert snap["title"] == full, snap
# The count part is visible and pinned inside the label (no
tests/e2e/test_sync_upload_progress.py
+36 -13
View File
@@ -64,8 +64,9 @@ Contract under test:
documents** (the upload unpacked + registered only) and the sync
button settled idle with no error UI; clicking **Sync sources**
then imports the uploaded row with the LIVE "Syncing… <file> (n/m)"
label (both layers) and lands the counts ("N added", the catalog
refreshes);
label (both layers — the phase-129 pinned chunk part "· chunk n/m"
rides the label + title while a file's embed batch is in flight)
and lands the counts ("N added", the catalog refreshes);
* **upload processing (phase 90 A2)**: the upload button shows the
BARE "Processing…" for the whole background run (no file, no
"(n/m)", no title — proven across a held first status GET); every
@@ -75,9 +76,11 @@ Contract under test:
result line points at the Sync button; the KB stays empty;
* **sync live file (A4)**: a multi-file local source; clicking
**Sync sources** on ``/sources.html`` shows "Syncing…" (bare, the
pre-64 click state) then "Syncing… <file> (n/m)" (both layers),
then the pre-64 success settle — "Synced HH:MM" + the counts result
line — preserved, plus the file in the label;
pre-64 click state) then "Syncing… <file> (n/m)" (both layers — the
phase-129 pinned chunk part "· chunk n/m" rides the label + title
while a file's embed batch is in flight), then the pre-64 success
settle — "Synced HH:MM" + the counts result line — preserved, plus
the file in the label;
* **reload re-attach (A1 + A2, phase 90)**: starting an upload and
reloading ``/git-sources.html`` (the sub-second run may still be in
flight — the boot re-attach enters the bare Processing state — or
@@ -607,20 +610,29 @@ def test_upload_toast_then_navigate_away(
# tick — the file rides the MAIN part, "(n/m)" the pinned COUNT
# span (phase 128: the space between them is the CSS gap, so the
# rendered concatenation carries none — the FULL string with the
# space rides the button title, A4).
# space rides the button title, A4). Phase 129: the pinned CHUNK
# part ("· chunk n/m" — the file's embed-batch position, batch
# granularity) renders while a file's embed batch is in flight —
# both the rendered label (no space — the CSS gap) and the title
# (with the space, A4) carry it.
expect(page.locator("#sync-label")).to_have_text(
re.compile(rf"Syncing… {re.escape(UPLOAD_NAME)}/.+\.md\(\d+/{N_FILES}\)"),
re.compile(
rf"Syncing… {re.escape(UPLOAD_NAME)}/.+\.md\(\d+/{N_FILES}\)(?:· chunk \d+/\d+)?"
),
timeout=SETTLE_TIMEOUT_MS,
)
# Phase 128: the "(n/m)" is NEVER clipped — it sits in its own
# visible count span, and the button title carries the full
# untruncated string (main + " " + count, A4).
# untruncated string (main + " " + count [+ " " + chunk, phase
# 129], A4).
expect(page.locator("#sync-label .sync-label-count")).to_be_visible(
timeout=SETTLE_TIMEOUT_MS
)
expect(btn).to_have_attribute(
"title",
re.compile(rf"^Syncing… {re.escape(UPLOAD_NAME)}/.+\.md \(\d+/{N_FILES}\)$"),
re.compile(
rf"^Syncing… {re.escape(UPLOAD_NAME)}/.+\.md \(\d+/{N_FILES}\)(?: · chunk \d+/\d+)?$"
),
timeout=SETTLE_TIMEOUT_MS,
)
@@ -791,20 +803,31 @@ def test_sync_live_file_label(
# tick — the file rides the MAIN part, "(n/m)" the pinned COUNT
# span (phase 128: the space between them is the CSS gap, so the
# rendered concatenation carries none — the FULL string with the
# space rides the button title, A4).
# space rides the button title, A4). Phase 129: the pinned CHUNK
# part ("· chunk n/m" — the file's embed-batch position, batch
# granularity) renders while a file's embed batch is in flight —
# both the rendered label (no space — the CSS gap) and the title
# (with the space, A4) carry it.
expect(page.locator("#sync-label")).to_have_text(
re.compile(rf"Syncing… {re.escape(SYNC_SOURCE_DIR)}/.+\.md\(\d+/{N_FILES}\)"),
re.compile(
rf"Syncing… {re.escape(SYNC_SOURCE_DIR)}/.+\.md\(\d+/{N_FILES}\)(?:· chunk \d+/\d+)?"
),
timeout=SETTLE_TIMEOUT_MS,
)
# Phase 128: the "(n/m)" is NEVER clipped — it sits in its own
# visible count span, and the button title carries the full
# untruncated string (main + " " + count, A4).
# untruncated string (main + " " + count [+ " " + chunk, phase
# 129], A4).
expect(page.locator("#sync-label .sync-label-count")).to_be_visible(
timeout=SETTLE_TIMEOUT_MS
)
expect(btn).to_have_attribute(
"title",
re.compile(rf"^Syncing… {re.escape(SYNC_SOURCE_DIR)}/.+\.md \(\d+/{N_FILES}\)$"),
re.compile(
"^Syncing… "
rf"{re.escape(SYNC_SOURCE_DIR)}/.+\.md \(\d+/{N_FILES}\)"
r"(?: · chunk \d+/\d+)?$"
),
timeout=SETTLE_TIMEOUT_MS,
)
tests/fakes.py
+22 -1
View File
@@ -1,6 +1,7 @@
"""Shared test fakes (no network, deterministic)."""
from __future__ import annotations
from collections.abc import Callable
from typing import Any
from app.config import Settings
@@ -22,6 +23,16 @@ class FakeEmbedder:
text-summary body above is string-only; a subclass handling the
multimodal image description (task 03's vision mock) overrides
``chat``.
``embed`` accepts the phase-129 (task 01) ``on_progress`` hook and
honors it with the real client's single-batch semantics: this fake
embeds every call in ONE top-level batch, so the hook — when given
and *texts* is non-empty — fires exactly once with
``(1, len(texts))`` before the "request"; empty *texts* fires it
never (the real client's early return). Subclasses that model
multi-batch transport override :meth:`embed` and own the hook
sequence themselves (passing ``on_progress=None`` to
:meth:`super().embed` per sub-batch).
"""
def __init__(self, dim: int = 768) -> None:
@@ -31,7 +42,17 @@ class FakeEmbedder:
self.calls: list[list[str]] = []
self.chat_calls: list[list[dict[str, str]]] = []
async def embed(self, texts: list[str]) -> list[list[float]]:
async def embed(
self,
texts: list[str],
*,
on_progress: Callable[[int, int], None] | None = None,
) -> list[list[float]]:
# Phase 129 (task 01): the single top-level batch announces its
# start when a hook is given (empty texts → no call — the real
# client's early return).
if texts and on_progress is not None:
on_progress(1, len(texts))
self.calls.append(list(texts))
self.embed_batches += 1
return [[0.01 * (i % 97) for i in range(self.dim)] for _ in texts]
tests/integration/test_chat_api.py
+12 -2
View File
@@ -19,7 +19,7 @@ import logging
import math
import re
import uuid
from collections.abc import Iterator
from collections.abc import Callable, Iterator
from datetime import UTC, datetime
from pathlib import Path
from typing import TYPE_CHECKING, Any, cast
@@ -124,7 +124,17 @@ class FakeRagLLM:
#: fake is byte-identical to the plain fake's output.
self.tool_script: list[list[StreamPiece | ToolCallPiece]] = list(tool_script or [])
async def embed(self, texts: list[str]) -> list[list[float]]:
async def embed(
self,
texts: list[str],
*,
on_progress: Callable[[int, int], None] | None = None,
) -> list[list[float]]:
# Phase 129 (task 01) hook, single-batch fake semantics (see
# tests.fakes.FakeEmbedder): this fake embeds every call in one
# top-level batch, so the hook fires once with (1, len(texts)).
if texts and on_progress is not None:
on_progress(1, len(texts))
self.embed_batches += 1
return [_token_vec(t) for t in texts]
tests/integration/test_sync_api.py
+36 -1
View File
@@ -285,6 +285,11 @@ class FakeImportSources:
# only, same root-string keying; local rows contribute
# nothing).
self.doc_dates_maps: list[dict[str, dict[str, datetime]]] = []
# Phase 129 (task 03): the chunk-position hook the runner
# passes (a live closure while wired, None if the wiring
# regresses) — the canned fake never fires it (the runner's
# status chunk fields stay 0/0, like a no-batch import).
self.chunk_hooks: list[object] = []
async def __call__(
self,
@@ -295,6 +300,7 @@ class FakeImportSources:
limit: int | None = None,
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -303,6 +309,7 @@ class FakeImportSources:
self.llms.append(llm)
self.prune_flags.append(prune)
self.progress_hooks.append(progress)
self.chunk_hooks.append(on_chunk_progress)
self.ignore_maps.append(ignore_by_root or {})
self.include_hidden_maps.append(include_hidden_by_root or {})
self.doc_dates_maps.append(doc_dates_by_root or {})
@@ -459,6 +466,10 @@ def test_admin_sync_success_reports_full_detail(
"current_file": None,
"files_done": 0,
"files_total": 0,
# Phase 129 (task 03): the chunk-position keys — 0/0 idle
# (the dataclass defaults).
"current_chunk": 0,
"chunks_total": 0,
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
"phase": None,
"current_summary": None,
@@ -627,6 +638,10 @@ class _GatedImport:
self.started = started
self.release = release
self.hook_calls: list[tuple[str, str, int, int]] = []
# Phase 129 (task 03): the chunk-position hook the runner
# passes (recorded for the wiring pin — the gated fake never
# fires it, so the status chunk fields stay 0/0).
self.chunk_hook: Callable[[int, int], None] | None = None
async def __call__(
self,
@@ -637,6 +652,7 @@ class _GatedImport:
limit: int | None = None,
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -644,6 +660,7 @@ class _GatedImport:
if progress is not None:
progress("repo", "notes/deep.md", 1, 3)
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
self.chunk_hook = on_chunk_progress
self.started.set()
await asyncio.to_thread(self.release.wait)
return self.summary
@@ -750,12 +767,14 @@ def test_status_reports_the_phase_machine_across_the_run(
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
_login(sync_client)
# Idle: the four phase keys ride along as null/0/0/0.
# Idle: the four phase keys ride along as null/0/0/0, and the
# phase-129 chunk keys as 0/0 (the dataclass defaults).
body = sync_client.get("/api/sync/status").json()
assert body["state"] == "idle"
assert body["phase"] is None
assert body["current_summary"] is None
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
assert body["current_chunk"] == 0 and body["chunks_total"] == 0
assert sync_client.post("/api/sync").status_code == 202
try:
@@ -767,12 +786,19 @@ def test_status_reports_the_phase_machine_across_the_run(
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_summary"] is None
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
# Phase 129 (task 03): no embed batch has fired the chunk hook
# (the gated fake never does) — the position stays 0/0.
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
gated_import.release.set()
assert gated_overview.started.wait(5.0), "the run never reached the overview"
s = sync_client.get("/api/sync/status").json()
assert s["state"] == "running"
assert s["phase"] == "overview"
assert s["current_summary"] is None
# Phase 129 (task 03): the import→overview transition cleared
# the chunk position (the convention split — the FILE counts
# below keep their final values).
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
gated_overview.release.set()
assert gated_folders.started.wait(5.0), "the run never reached the summaries"
s = sync_client.get("/api/sync/status").json()
@@ -785,6 +811,9 @@ def test_status_reports_the_phase_machine_across_the_run(
# The import's final position is kept through the summary span
# (the "number pauses" span — the user's report).
assert s["files_done"] == 1 and s["files_total"] == 3
# Phase 129 (task 03): the import→summaries transition kept
# the chunk position cleared (0/0 through the summary span).
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
gated_folders.release.set()
finally:
gated_import.release.set()
@@ -797,6 +826,11 @@ def test_status_reports_the_phase_machine_across_the_run(
assert body["phase"] is None
assert body["current_summary"] is None
assert body["summaries_done"] == 3 and body["summaries_total"] == 3
# Phase 129 (task 03): the terminal keeps the chunk position
# cleared (0/0) — the wiring pin: the runner passed a live chunk
# closure to the import (it just never fired in this canned run).
assert body["current_chunk"] == 0 and body["chunks_total"] == 0
assert gated_import.chunk_hook is not None
# Wiring: the summary hook was the runner's live closure (the
# counters above came through it); the file hook fired once.
assert gated_folders.hook is not None
@@ -1432,6 +1466,7 @@ def test_import_error_is_reported_with_credentials_masked(
limit: int | None = None,
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
tests/unit/test_frontend_sync_upload.py
+311 -24
View File
@@ -63,12 +63,39 @@ visible span carries main + count, the space is the CSS gap). Every
other label state (idle "Sync sources", settle "Synced HH:MM", the
202/409 click, the re-attach) goes through setSyncLabel(main) —
count hidden.
Phase 129 (task 04) adds the label's THIRD pinned part — the chunk
position: #sync-label ships a hidden .sync-label-chunk span after the
count span (the pinned "· chunk n/m" — the in-flight file's
embed-batch position, batch granularity: n the 1-based first chunk of
the in-flight batch, m the file's total content chunks). The ONLY
builder that reads the status's chunk fields is fmtSyncPhaseLabel's
import fall-through — "· chunk n/m" renders while chunks_total > 0
and hides in every other state (the prelude, the 202/409 click, the
upload run — its status has no chunk fields, overview, summaries —
its own (n/m) is its position, idle / settle). setSyncLabel(mainText,
countText, chunkText) lands the part under the same present/absent
contract as the count part, and the UNTRUNCATED full label — main +
" " + count + " " + chunk — rides the button title + #sync-result
(A4 — the in-flight import reads "Syncing… <file> (2/3) · chunk
14/38"). The part keeps the pinned mono CSS (flex 0 0 auto, nowrap,
--mono, the rule right after .sync-label-count). A node harness (the
test-document-viewer pattern) behaviorally pins the rendered parts +
the full title across every label state.
"""
from __future__ import annotations
import json
import re
import shutil
import subprocess
from pathlib import Path
import pytest
#: The node harness needs node (the test-document-viewer pattern).
HAVE_NODE = shutil.which("node") is not None
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
SOURCES_JS = FRONTEND / "assets" / "sources.js"
STYLES_CSS = FRONTEND / "assets" / "styles.css"
@@ -219,21 +246,26 @@ def test_fmt_sync_label_returns_the_parts_contract() -> None:
def test_set_sync_label_is_the_single_parts_writer() -> None:
"""setSyncLabel(mainText, countText = "") — Phase 128 (task 01):
the SINGLE point that writes #sync-label. The MAIN part gets
mainText (textContent only — the house rule, no HTML strings);
the COUNT part gets countText when present (textContent +
hidden=false — the "(n/m)" position is visible, NEVER clipped) and
is emptied + hidden=true when absent (the count span is hidden
whenever there is no "(n/m)" to show). Then the UNTRUNCATED full
label — mainText + (countText ? " " + countText : "") — rides the
button title + #sync-result (the aria-live announcer) exactly as
before (A4 — the screen reader hears the whole thing)."""
"""setSyncLabel(mainText, countText = "", chunkText = "") — Phase
128 (task 01) + Phase 129 (task 04): the SINGLE point that writes
#sync-label. The MAIN part gets mainText (textContent only — the
house rule, no HTML strings); the COUNT part gets countText when
present (textContent + hidden=false — the "(n/m)" position is
visible, NEVER clipped) and is emptied + hidden=true when absent;
the CHUNK part (phase 129) gets chunkText under the SAME contract
(present → the pinned "· chunk n/m" is visible, absent → emptied
+ hidden=true — the part renders only while a file's embed batch
is in flight). Then the UNTRUNCATED full label — mainText +
(countText ? " " + countText : "") + (chunkText ? " " + chunkText
: "") — rides the button title + #sync-result (the aria-live
announcer) (A4 — the screen reader hears the whole "Syncing…
<file> (2/3) · chunk 14/38")."""
body = _fn(_js(), "setSyncLabel")
assert 'function setSyncLabel(mainText, countText = "")' in body
assert 'function setSyncLabel(mainText, countText = "", chunkText = "")' in body
# the parts: textContent only (the house rule), null-guarded refs
assert "syncLabelMain.textContent = mainText" in body
assert "syncLabelCount.textContent = countText" in body
assert "syncLabelChunk.textContent = chunkText" in body
# present count → visible; absent count → hidden, emptied
assert "syncLabelCount.hidden = false" in body
assert 'syncLabelCount.textContent = ""' in body
@@ -241,11 +273,22 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
i_vis = body.find("syncLabelCount.hidden = false")
i_hid = body.find("syncLabelCount.hidden = true")
assert -1 < i_vis < i_hid, "the present-count branch precedes the absent one"
# present chunk → visible; absent chunk → hidden, emptied (same contract)
assert "syncLabelChunk.hidden = false" in body
assert 'syncLabelChunk.textContent = ""' in body
assert "syncLabelChunk.hidden = true" in body
i_cvis = body.find("syncLabelChunk.hidden = false")
i_chid = body.find("syncLabelChunk.hidden = true")
assert -1 < i_cvis < i_chid, "the present-chunk branch precedes the absent one"
# A4: the untruncated full label rides the title + the announcer —
# main + " " + count (the join space), bare main when no count.
# main + " " + count + " " + chunk (the join spaces = the CSS
# gaps), bare main when both are absent.
assert (
'const full = countText ? `${mainText} ${countText}` : mainText'
) in body, "the full label is main + ' ' + count (bare main when absent)"
"const full =\n"
" mainText +\n"
" (countText ? ` ${countText}` : \"\") +\n"
" (chunkText ? ` ${chunkText}` : \"\");"
) in body, ("the full label is main + ' ' + count + ' ' + chunk (spaces = the CSS gaps)")
i_full = body.find("const full")
i_title = body.find("syncBtn.title = full")
i_result = body.find("syncResult.textContent = full")
@@ -281,8 +324,11 @@ def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
assert "current_summary" not in ov and "summaries_done" not in ov, (
"the overview label carries no folder and no counts"
)
# "summaries": the exact template — optional folder, counts always.
i_fall = body.find("return fmtSyncLabel")
# "summaries": the exact template — optional folder, counts always,
# NO chunk part (the summaries phase's own (n/m) is its position —
# the status's chunk fields are cleared at this transition, so the
# branch never reads them).
i_fall = body.find("fmtSyncLabel(")
assert i_sum < i_fall, "the summaries branch precedes the fall-through"
sum_ = body[i_sum:i_fall]
assert 'status.current_summary ? ` ${status.current_summary}` : ""' in sum_, (
@@ -295,12 +341,40 @@ def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
"count: `(${status.summaries_done}/${status.summaries_total})`"
) in sum_, "the count part is ALWAYS emitted in the summaries phase"
assert "total > 0" not in sum_, ("no conditional — the hook always carries a total")
assert 'chunk: ""' in ov, "the overview label carries no chunk part"
assert 'chunk: ""' in sum_, (
"the summaries label carries no chunk part (its (n/m) is its position)"
)
assert "chunks_total" not in ov, "the overview branch never reads the chunk fields"
assert "chunks_total" not in sum_, "the summaries branch never reads the chunk fields"
# anything else (null prelude, "import"): byte-identical to today's
# sync label — same builder, same fields, same order.
# sync label — same builder, same fields, same order — plus the
# phase-129 chunk part: "· chunk n/m" ONLY while chunks_total > 0
# (a file's embed batch is in flight — n the 1-based first chunk
# of the batch, m the file's total content chunks, batch
# granularity), "" otherwise (0/0 before the first batch — the
# part hides). The fall-through is the ONLY branch that reads the
# status's chunk fields.
i_destr = body.find("const { main, count } = fmtSyncLabel(")
assert -1 < i_destr < i_fall, "the fall-through destructures the phase-64 parts"
fall = body[i_destr:]
assert (
'return fmtSyncLabel("sync", status.current_file, '
"status.files_done, status.files_total);"
) in body[i_fall:]
'const { main, count } = fmtSyncLabel(\n'
' "sync", status.current_file, status.files_done, status.files_total\n'
" );"
) in fall, "the phase-64 parts builder, unchanged"
assert "status.chunks_total > 0" in fall, (
"the chunk part renders only while a batch is in flight"
)
assert "`· chunk ${status.current_chunk}/${status.chunks_total}`" in fall
i_guard = fall.find("status.chunks_total > 0")
i_else = fall.find(": \"\";", i_guard)
assert -1 < i_guard < i_else, "0/0 (before the first batch) → the empty chunk part"
assert "return { main, count, chunk }" in fall
# the bare builder (the upload job + the 202/409 click) returns NO
# chunk part — those labels stay bare.
bare = _fn(_js(), "fmtSyncLabel")
assert "chunk" not in bare, "fmtSyncLabel stays chunk-free (upload + the click stay bare)"
def test_enter_running_state_is_phase_aware_and_writes_the_parts() -> None:
@@ -312,8 +386,10 @@ def test_enter_running_state_is_phase_aware_and_writes_the_parts() -> None:
endpoint has no phase) and the 202/409 click keep the bare
fmtSyncLabel parts, byte-identical. Phase 128 (task 01): the entry
itself no longer writes the label — it hands the parts to the
SINGLE writer (setSyncLabel(main, count)), which lands the main
part, the pinned count part (hidden when absent), and the FULL
SINGLE writer (setSyncLabel(main, count, chunk) — phase 129:
the chunk part destructures with a "" default), which lands the
main part, the pinned count part (hidden when absent), the pinned
chunk part (phase 129 — hidden when absent), and the FULL
untruncated label on the button title + #sync-result (A4 — set
for EVERY running label; CSS ellipsizes the main part only)."""
body = _fn(_js(), "enterSyncRunningState")
@@ -332,8 +408,14 @@ def test_enter_running_state_is_phase_aware_and_writes_the_parts() -> None:
i_fall = body.find("fmtSyncLabel(kind, currentFile, done, total)")
assert 'kind === "sync" && status' in body
assert -1 < i_phase < i_fall, "the phase label precedes the fall-through"
# the parts are handed to the single writer (A4 lives in it).
i_set = body.find("setSyncLabel(main, count)")
# the parts are handed to the single writer (A4 lives in it) —
# phase 129: the chunk part destructures with a "" default (the
# bare builders — the upload job, the 202/409 click — return no
# chunk part, so the part stays hidden there).
assert "const { main, count, chunk = \"\" } =" in body, (
"the chunk part destructures with the '' default (bare builders return none)"
)
i_set = body.find("setSyncLabel(main, count, chunk)")
assert -1 < i_fall < i_set, "the parts are written after the split"
assert body.count("setSyncLabel(") == 1, "one writer call per entry"
assert "syncBtn.title =" not in body, ("the title write lives in setSyncLabel (A4)")
@@ -369,6 +451,211 @@ def test_settle_and_idle_labels_go_through_the_parts_writer() -> None:
assert "syncLabel.textContent" not in js, ("the parts writer is the single entry point")
# =====================================================================
# Phase 129 (task 04) — the label's third part: the chunk position
# =====================================================================
def test_sync_label_markup_ships_the_hidden_chunk_part() -> None:
"""#sync-label (the shell's RAG view) ships the phase-129 THIRD
part after the count span — the pinned "· chunk n/m" span
(.sync-label-chunk), hidden + empty by default (the same ship
contract as the count span: no chunk position at idle — the part
renders only while a file's embed batch is in flight)."""
html = _html()
btn = html[html.find('id="sync-btn"') : html.find("</button>", html.find('id="sync-btn"'))]
label = re.search(
r'<span\b[^>]*id="sync-label"[^>]*>'
r'<span class="sync-label-main">Sync sources</span>'
r'<span class="sync-label-count" hidden></span>'
r'<span class="sync-label-chunk" hidden></span>'
r"</span>",
btn,
)
assert label, (
"the #sync-label parts markup (main + hidden count + hidden chunk) is missing"
)
def test_sync_label_chunk_css_is_the_pinned_part() -> None:
""".sync-label-chunk (phase 129 task 04): the same pinned-part
language as .sync-label-count — flex 0 0 auto (NEVER clipped),
white-space nowrap, the --mono font (a position reads as data) —
and it sits right AFTER the count rule (both mono parts read as
data)."""
css = _css()
count_block = re.search(r"\.sync-label-count\s*\{", css)
chunk_block = re.search(r"\.sync-label-chunk\s*\{([^}]*)\}", css)
assert chunk_block, "styles.css must style .sync-label-chunk"
for prop in ("flex: 0 0 auto", "white-space: nowrap", "font-family: var(--mono)"):
assert prop in chunk_block.group(1), f".sync-label-chunk must carry {prop!r}"
assert count_block and count_block.start() < chunk_block.start(), (
"the chunk rule sits right after the count rule"
)
#: The node harness script (the test-document-viewer pattern): the
#: REAL extracted label functions run against stub element refs, and
#: every label state is snapshotted as a JSON row —
#: [tag, title, announcer, main, count, countHidden, chunk, chunkHidden].
#: No JS template literals in the script body (plain strings only —
#: the script is assembled by concatenation, no placeholders).
_CHUNK_NODE_HEAD = """
const syncLabelMain = { textContent: "" };
const syncLabelCount = { textContent: "", hidden: true };
const syncLabelChunk = { textContent: "", hidden: true };
const syncResult = { textContent: "" };
const syncBtn = {
disabled: false,
title: "",
attrs: {},
setAttribute(k, v) { this.attrs[k] = v; },
removeAttribute(k) { delete this.attrs[k]; },
classList: { add() {}, remove() {} },
};
const syncIcon = { classList: { add() {}, remove() {} } };
"""
_CHUNK_NODE_TAIL = """
const show = (tag) => {
console.log(JSON.stringify([
tag, syncBtn.title, syncResult.textContent,
syncLabelMain.textContent,
syncLabelCount.textContent, syncLabelCount.hidden,
syncLabelChunk.textContent, syncLabelChunk.hidden,
]));
};
// 1. import, batch in flight (the task's fixture values, 14/38):
enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 3,
current_chunk: 14, chunks_total: 38,
});
show("import-inflight");
// 2. import before the first batch (0/0) — the chunk part hides:
enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 3,
current_chunk: 0, chunks_total: 0,
});
show("import-prebatch");
// 3. the prelude (null phase, no file yet):
enterSyncRunningState("sync", null, 0, 0, {
phase: null, current_file: null, files_done: 0, files_total: 0,
current_chunk: 0, chunks_total: 0,
});
show("prelude");
// 4. the 202/409 click (no status yet):
enterSyncRunningState("sync", null, 0, 0);
show("click");
// 5. the upload run (no status — its endpoint has no chunk fields, D2):
enterSyncRunningState("upload", null, 0, 0);
show("upload");
// 6. overview (the chunk fields are cleared at this transition):
enterSyncRunningState("sync", null, 4, 4, {
phase: "overview", current_file: null, files_done: 4, files_total: 4,
current_chunk: 0, chunks_total: 0,
});
show("overview");
// 7. summaries (the phase's own (n/m) is its position):
enterSyncRunningState("sync", null, 4, 4, {
phase: "summaries", current_summary: "src/alpha", files_done: 4, files_total: 4,
summaries_done: 2, summaries_total: 3, current_chunk: 0, chunks_total: 0,
});
show("summaries");
// 8. the idle settle goes through the writer with no chunk text:
setSyncLabel("Sync sources");
show("idle");
"""
def _run_node(script: str) -> str:
proc = subprocess.run(["node", "-e", script], capture_output=True, text=True, timeout=60)
assert proc.returncode == 0, f"node failed: {proc.stderr}"
return proc.stdout
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_chunk_part_renders_only_in_the_import_phase_under_node() -> None:
"""Behavioral check (node) of the REAL label path — the extracted
fmtSyncLabel / fmtSyncPhaseLabel / setSyncLabel /
enterSyncRunningState run against stub element refs (the
test-document-viewer pattern). The pinned "· chunk n/m" part (the
task's fixture values):
* import, batch in flight (chunks_total 38 > 0): the chunk part
shows "· chunk 14/38" and the UNTRUNCATED title + #sync-result
read exactly "Syncing… src/a/b.md (2/3) · chunk 14/38" (A4);
* import before the first batch (0/0): the chunk part hides and
the label stays "Syncing… src/a/b.md (2/3)";
* every OTHER state — prelude (null phase), the 202/409 click (no
status), the upload run (no status, D2), overview, summaries (its
own (n/m) is its position), the idle settle — leaves the chunk
part hidden + empty.
"""
js = _js()
fns = "\n".join(
_fn(js, name)
for name in ("fmtSyncLabel", "fmtSyncPhaseLabel", "setSyncLabel", "enterSyncRunningState")
)
script = _CHUNK_NODE_HEAD + fns + _CHUNK_NODE_TAIL
rows = [json.loads(line) for line in _run_node(script).splitlines() if line]
by_tag = {row[0]: row for row in rows}
# row = [tag, title, announcer, main, count, countHidden, chunk, chunkHidden]
# 1. the in-flight import — the chunk part renders, the full
# untruncated label rides the title + the announcer (A4).
row = by_tag["import-inflight"]
assert row[3] == "Syncing… src/a/b.md", row
assert row[4] == "(2/3)" and row[5] is False, row
assert row[6] == "· chunk 14/38" and row[7] is False, (
"the chunk part shows the batch position (1-based first chunk / file total)"
)
assert row[1] == "Syncing… src/a/b.md (2/3) · chunk 14/38", row[1]
assert row[2] == row[1], "the announcer hears the whole thing (A4)"
# 2. the import before the first batch (0/0) — the part hides, the
# label is the byte-identical phase-64 string.
row = by_tag["import-prebatch"]
assert row[1] == "Syncing… src/a/b.md (2/3)", row[1]
assert row[2] == row[1]
assert row[6] == "" and row[7] is True, row
# 3. every other state — the chunk part is hidden + empty, and the
# title is the byte-identical pre-phase-129 label for that state.
for tag, expected_title in (
("prelude", "Syncing…"),
("click", "Syncing…"),
("upload", "Importing"),
("overview", "Writing KB overview…"),
("summaries", "Summarizing folders… src/alpha (2/3)"),
("idle", "Sync sources"),
):
row = by_tag[tag]
assert row[1] == expected_title, (tag, row)
assert row[2] == row[1], (tag, row)
assert row[6] == "" and row[7] is True, f"{tag} must hide the chunk part"
def test_every_other_state_passes_no_chunk_text() -> None:
"""Source-level pin of the render-only-while-chunks_total>0
contract: the import fall-through is the ONLY place in sources.js
that reads the status's chunk fields (the summaries/overview
branches and the bare builders never touch them), and every
NON-running label emission passes no chunk text (the part hides):
the settle/idle labels go through setSyncLabel(label) — the only
three-arg call site is the running entry."""
js = _js()
body = _fn(js, "fmtSyncPhaseLabel")
i_fall = body.find("fmtSyncLabel(")
pre = body[:i_fall]
assert "chunks_total" not in pre and "current_chunk" not in pre, (
"the overview/summaries branches never read the chunk fields"
)
fall = body[i_fall:]
assert "status.chunks_total" in fall and "status.current_chunk" in fall
assert js.count("setSyncLabel(main, count, chunk)") == 1
assert "setSyncLabel(label)" in js, "the settle/idle labels stay two-arg (no chunk)"
# ---------- the two-job tick (startSyncPolling) ----------
tests/unit/test_image_anchor_context.py
+7 -1
View File
@@ -47,6 +47,7 @@ from __future__ import annotations
import asyncio
import base64
from collections.abc import Callable
from pathlib import Path
from typing import Any
@@ -730,7 +731,12 @@ class _EmbedFailingVision(FakeEmbedder):
first = user.split()
return "Summary of " + (first[0] if first else "<empty>")
async def embed(self, texts):
async def embed(
self,
texts: list[str],
*,
on_progress: Callable[[int, int], None] | None = None,
) -> list[list[float]]:
raise EmbeddingError("simulated embed endpoint failure (test sentinel)")
tests/unit/test_image_documents.py
+8 -2
View File
@@ -66,6 +66,7 @@ import asyncio
import base64
import logging
import re
from collections.abc import Callable
from pathlib import Path
from typing import Any
@@ -199,13 +200,18 @@ class _SummaryEmbedFailsLLM(FakeEmbedder):
super().__init__()
self._embed_batches = 0
async def embed(self, texts: list[str]) -> list[list[float]]:
async def embed(
self,
texts: list[str],
*,
on_progress: Callable[[int, int], None] | None = None,
) -> list[list[float]]:
self._embed_batches += 1
if self._embed_batches == 2:
raise EmbeddingError(
"simulated summary-chunk embed failure (test sentinel)"
)
return await super().embed(texts)
return await super().embed(texts, on_progress=on_progress)
class _VisionMockLLM(FakeEmbedder):
tests/unit/test_importer.py
+218 -4
View File
@@ -16,6 +16,7 @@ from __future__ import annotations
import asyncio
import logging
import os
from collections.abc import Callable
from datetime import UTC, datetime
from pathlib import Path
@@ -27,6 +28,7 @@ from sqlalchemy.orm import Session
import app.rag.importer as importer
from app.config import Settings
from app.models import Chunk, Document
from app.rag.chunker import chunk_document
from app.rag.importer import (
EXCLUDED_DIRS,
ImportSummary,
@@ -48,10 +50,15 @@ DEFAULT_EXTS = frozenset({".md", ".markdown", ".txt", ".yaml", ".yml", ".json",
class _PoisonEmbedder(FakeEmbedder):
"""Fails (like a real endpoint) on any text containing 'poison'."""
async def embed(self, texts: list[str]) -> list[list[float]]:
async def embed(
self,
texts: list[str],
*,
on_progress: Callable[[int, int], None] | None = None,
) -> list[list[float]]:
if any("poison" in t for t in texts):
raise EmbeddingError("embeddings endpoint refused the input (simulated)")
return await super().embed(texts)
return await super().embed(texts, on_progress=on_progress)
class _FailingChatEmbedder(FakeEmbedder):
@@ -67,13 +74,18 @@ class _CapEmbedder(FakeEmbedder):
"""Simulates the endpoint's ~1024-token input cap at ~1.1 chars/token:
any single text over 1000 chars is rejected (URL-dense worst case)."""
async def embed(self, texts: list[str]) -> list[list[float]]:
async def embed(
self,
texts: list[str],
*,
on_progress: Callable[[int, int], None] | None = None,
) -> list[list[float]]:
if any(len(t) > 1000 for t in texts):
raise EmbeddingError(
"a single 1100-char chunk exceeded the endpoint's per-request "
"input token cap — lower BOR_CHUNK_TARGET_CHARS and re-import"
)
return await super().embed(texts)
return await super().embed(texts, on_progress=on_progress)
def _embedder_with_extensions(extensions: str) -> FakeEmbedder:
@@ -1430,3 +1442,205 @@ def test_prune_records_changed_paths_only_when_summary_given(db, tmp_path: Path)
assert counters.changed_paths == [(source, "gone2.md")]
finally:
_cleanup_source(db, root.name)
# ---------- phase 129 (task 02): on_chunk_progress plumbing ----------
class _BatchingEmbedder(FakeEmbedder):
"""Mimics ``LLMClient``'s top-level token-budgeted batching (the
task-01 contract on the importer side): *texts* is split into
consecutive groups of *batch_size* and the hook fires
``(first_chunk_1based, total)`` before EACH group — the group is then
embedded as one request (``super().embed`` with ``on_progress=None``:
this class owns the hook sequence, so the base's single-batch
announce is suppressed per group)."""
def __init__(self, batch_size: int) -> None:
super().__init__()
self.batch_size = batch_size
async def embed(
self,
texts: list[str],
*,
on_progress: Callable[[int, int], None] | None = None,
) -> list[list[float]]:
out: list[list[float]] = []
for start in range(0, len(texts), self.batch_size):
group = texts[start : start + self.batch_size]
if on_progress is not None:
on_progress(start + 1, len(texts))
out.extend(await super().embed(group, on_progress=None))
return out
def _markdown_with_sections(stem: str, n: int) -> str:
"""One markdown file with a preamble (h1 title + intro) and *n* short
``##`` sections — each section is under the 2000-char target, so the
file chunks to exactly *n* + 1 content chunks (the preamble + one per
section; verified against ``chunk_document`` below)."""
lines = [f"# {stem}", "", f"Intro line for {stem.lower()}.", ""]
for i in range(1, n + 1):
lines += [f"## {stem} {i}", "", f"Body text for section {i}.", ""]
return "\n".join(lines)
def _chunk_count(content: str, rel: str) -> int:
"""The content-chunk count the importer's pipeline will produce for
*content* (the same ``chunk_document`` call and defaults as
``_replace_content_chunks``: ``max(400, chunk_target_chars)`` target,
``chunk_overlap_chars`` overlap)."""
s = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
return len(chunk_document(content, rel, max(400, s.chunk_target_chars), s.chunk_overlap_chars))
def test_on_chunk_progress_reports_per_file_batch_sequences(
db, tmp_path: Path
) -> None:
"""A multi-batch file's embed position is fully observable: the hook
fires before each embed batch with ``(first_chunk_1based, total)`` —
``total`` is the file's content-chunk count, the first indices climb
by the previous batch's size, and a SECOND file in the same run gets
its OWN sequence (the two files' totals never mix)."""
root = tmp_path / "chunkprog"
root.mkdir()
content_a = _markdown_with_sections("Alpha", 8) # preamble + 8 = 9 chunks
content_b = _markdown_with_sections("Beta", 4) # preamble + 4 = 5 chunks
(root / "alpha.md").write_text(content_a)
(root / "beta.md").write_text(content_b)
n_a, n_b = _chunk_count(content_a, "alpha.md"), _chunk_count(content_b, "beta.md")
assert (n_a, n_b) == (9, 5)
batch_size = 3
def expected(n: int) -> list[tuple[int, int]]:
return [(start + 1, n) for start in range(0, n, batch_size)]
# The task's multi-batch case: file A spans >= 2 batches.
assert len(expected(n_a)) >= 2
llm = _BatchingEmbedder(batch_size=batch_size)
chunk_events: list[tuple[int, int]] = []
file_events: list[tuple[str, str, int, int]] = []
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
progress=lambda s, r, d, t: file_events.append((s, r, d, t)),
on_chunk_progress=lambda f, t: chunk_events.append((f, t)),
)
)
assert (summary.added, summary.chunks) == (2, n_a + n_b)
finally:
_cleanup_source(db, root.name)
# The per-file ``progress`` hook names each file in walk order ...
assert file_events == [
(root.name, "alpha.md", 1, 2),
(root.name, "beta.md", 2, 2),
]
# ... and the chunk sequence is exactly file A's batches followed by
# file B's OWN restarted sequence (its totals never leak into A's).
assert chunk_events == expected(n_a) + expected(n_b)
def test_on_chunk_progress_single_batch_file_and_summary_excluded(
db, tmp_path: Path
) -> None:
"""A file whose chunks fit in ONE embed batch reports exactly one
``(1, N)`` event, and the document's summary-chunk embed (the SECOND
``llm.embed`` call) NEVER fires the hook — the counter counts content
chunks only (A6: no ``1/1`` noise after the real progress ends)."""
root = tmp_path / "chunkprog-one"
root.mkdir()
content = _markdown_with_sections("Gamma", 3) # preamble + 3 = 4 chunks
(root / "gamma.md").write_text(content)
n = _chunk_count(content, "gamma.md")
assert n == 4
events: list[tuple[int, int]] = []
llm = FakeEmbedder()
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_chunk_progress=lambda f, t: events.append((f, t)),
)
)
assert summary.added == 1
finally:
_cleanup_source(db, root.name)
assert events == [(1, n)]
# Two embed calls happened (the content batch + the excluded summary
# chunk) — the hook fired for the first only.
assert [len(c) for c in llm.calls] == [n, 1]
def test_on_chunk_progress_image_branch_reports_one_of_one(
db, tmp_path: Path
) -> None:
"""The image branch's ONE description-chunk embed reports exactly
``(1, 1)`` — it IS the file's only chunk — and the mirrored
summary-chunk embed never fires the hook. The REAL description seam
is used (the ``_VisionLLM`` multimodal ``chat``)."""
root = tmp_path / "imgchunk"
root.mkdir()
(root / "pic.png").write_bytes(_png_bytes(4, 4, (0.8, 0.2, 0.2)))
events: list[tuple[int, int]] = []
llm = _VisionLLM()
llm.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
images=True,
image_dir=str(tmp_path / "images"),
)
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_chunk_progress=lambda f, t: events.append((f, t)),
)
)
assert (summary.files, summary.added) == (1, 1)
finally:
_cleanup_source(db, root.name)
assert events == [(1, 1)]
# The content embed (the 1 description chunk) + the excluded
# summary-chunk embed — the hook fired for the first only.
assert [len(c) for c in llm.calls] == [1, 1]
def test_on_chunk_progress_pdf_branch_reports_content_chunks(
db, tmp_path: Path
) -> None:
"""The PDF branch's CONTENT embed is the primary target: a two-page
PDF (two ``Page N:`` chunks) reports ``(1, 2)`` once through the
hook, and the summary-chunk embed never fires it."""
root = tmp_path / "pdfchunk"
root.mkdir()
doc = fitz.open()
doc.new_page().insert_text((72, 72), "First page text about networking.")
doc.new_page().insert_text((72, 72), "Second page text about storage.")
doc.save(root / "report.pdf")
doc.close()
events: list[tuple[int, int]] = []
llm = _VisionLLM()
llm.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
images=True,
image_dir=str(tmp_path / "images"),
import_extensions="md,txt,pdf",
)
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_chunk_progress=lambda f, t: events.append((f, t)),
)
)
assert (summary.files, summary.added) == (1, 1)
finally:
_cleanup_source(db, root.name)
assert events == [(1, 2)]
# The content embed (the 2 page chunks, one batch) + the excluded
# summary-chunk embed — the hook fired for the first only.
assert [len(c) for c in llm.calls] == [2, 1]
tests/unit/test_llm_client.py
+115 -1
View File
@@ -54,12 +54,14 @@ class _FakeEmbeddingsService:
self,
dim: int = 768,
fail: Exception | None = None,
fail_from: int = 1,
drop_index: int = -1,
http_error: int | None = None,
too_large_min: int | None = None,
) -> None:
self.dim = dim
self.fail = fail
self.fail_from = fail_from # 1-based: fail on this call number onward
self.drop_index = drop_index
self.http_error = http_error
self.too_large_min = too_large_min
@@ -67,7 +69,7 @@ class _FakeEmbeddingsService:
async def create(self, *, model: str, input: list[str]) -> _Response:
self.calls.append(list(input))
if self.fail is not None:
if self.fail is not None and len(self.calls) >= self.fail_from:
raise self.fail
return _Response(
[
@@ -275,6 +277,118 @@ def test_single_oversized_text_fails_actionably() -> None:
assert llm.embed_batches == 0
# ---------- embed on_progress hook (phase 129, task 01) ----------
def test_embed_on_progress_reports_batch_starts_against_batch_texts() -> None:
"""The hook fires once per TOP-LEVEL batch, before the request, with
``(first_chunk_1based, total)`` — asserted against ``_batch_texts``'s
actual partition: each first index = the previous first + the
previous batch's size, total always ``len(texts)``."""
service = _FakeEmbeddingsService()
llm, _ = _make_client(service, embed_batch_size=4) # count cap binds
texts = [f"t{i}" for i in range(11)] # 4 + 4 + 3 ⇒ three top-level batches
events: list[tuple[int, int]] = []
vecs = asyncio.run(
llm.embed(texts, on_progress=lambda first, total: events.append((first, total)))
)
batches = llm._batch_texts(texts)
assert [len(b) for b in batches] == [4, 4, 3]
# Walk the partition: each first index = the previous first + the
# previous batch's size (offset counts chunks of earlier batches).
expected: list[tuple[int, int]] = []
offset = 0
for batch in batches:
expected.append((offset + 1, len(texts)))
offset += len(batch)
assert events == expected == [(1, 11), (5, 11), (9, 11)]
# first-indices strictly increasing, total always the full input count
firsts = [first for first, _ in events]
assert firsts == sorted(firsts) and len(set(firsts)) == len(firsts)
assert all(total == 11 for _, total in events)
assert len(vecs) == 11
assert llm.embed_batches == 3
def test_embed_on_progress_token_budget_partition() -> None:
"""The same contract holds when the TOKEN budget (not the count cap)
binds the partition — one oversized text per request."""
service = _FakeEmbeddingsService()
llm, _ = _make_client(service, embed_batch_size=16) # token budget binds
texts = ["x" * 2000 for _ in range(5)] # 2×2000 > 2800-char budget
events: list[tuple[int, int]] = []
asyncio.run(
llm.embed(texts, on_progress=lambda first, total: events.append((first, total)))
)
batches = llm._batch_texts(texts)
assert [len(b) for b in batches] == [1, 1, 1, 1, 1]
assert events == [(1, 5), (2, 5), (3, 5), (4, 5), (5, 5)]
def test_embed_on_progress_fires_for_a_failing_batch() -> None:
"""The hook precedes the request's try/except: a failing batch still
announced its start — the position is visible at the failure point."""
service = _FakeEmbeddingsService(
fail=RuntimeError("connection refused"), fail_from=2
) # first batch succeeds, the second fails
llm, _ = _make_client(service) # embed_batch_size=2 ⇒ [a,b] + [c]
events: list[tuple[int, int]] = []
def hook(first: int, total: int) -> None:
events.append((first, total))
with pytest.raises(EmbeddingError, match="connection refused"):
asyncio.run(llm.embed(["a", "b", "c"], on_progress=hook))
assert events == [(1, 3), (3, 3)]
assert llm.embed_batches == 1 # the failing batch did not count
def test_embed_on_progress_halved_batches_do_not_subreport() -> None:
"""The halving backstop is rare by construction and does NOT sub-report:
a halved batch leaves the announced position approximate (documented in
the ``embed`` docstring) — one top-level batch ⇒ exactly one hook call."""
service = _FakeEmbeddingsService(too_large_min=3)
llm, _ = _make_client(service, embed_batch_size=16) # all 8 fit one request
texts = [f"t{i}" for i in range(8)]
events: list[tuple[int, int]] = []
vecs = asyncio.run(
llm.embed(texts, on_progress=lambda first, total: events.append((first, total)))
)
assert events == [(1, 8)]
assert len(vecs) == 8
def test_embed_on_progress_none_is_a_noop_identical_to_default() -> None:
"""Explicit ``None`` (and the default) keep the pre-hook client: the
same request sequence, the same vectors, the same counter."""
service_default = _FakeEmbeddingsService()
llm_default, _ = _make_client(service_default)
service_none = _FakeEmbeddingsService()
llm_none, _ = _make_client(service_none)
texts = [f"t{i}" for i in range(5)]
vecs_default = asyncio.run(llm_default.embed(texts))
vecs_none = asyncio.run(llm_none.embed(texts, on_progress=None))
assert vecs_default == vecs_none
assert service_default.calls == service_none.calls
assert [len(c) for c in service_default.calls] == [2, 2, 1]
assert llm_default.embed_batches == llm_none.embed_batches == 3
def test_embed_empty_texts_never_calls_on_progress() -> None:
"""The early return for an empty input list precedes the batch loop —
no hook calls, no requests."""
service = _FakeEmbeddingsService()
llm, http = _make_client(service)
events: list[tuple[int, int]] = []
vecs = asyncio.run(
llm.embed([], on_progress=lambda first, total: events.append((first, total)))
)
assert vecs == []
assert events == []
assert http.bodies == []
# ---------- chat streaming (phase 03) ----------
tests/unit/test_pdf.py
+8 -3
View File
@@ -12,7 +12,7 @@ from __future__ import annotations
import asyncio
import hashlib
import logging
from collections.abc import Sequence
from collections.abc import Callable, Sequence
from datetime import timedelta
from pathlib import Path
from types import SimpleNamespace
@@ -116,10 +116,15 @@ class _FirstImageEmbedFailsLLM(FakeEmbedder):
first = user.split()
return "Summary of " + (first[0] if first else "<empty>")
async def embed(self, texts: list[str]) -> list[list[float]]:
async def embed(
self,
texts: list[str],
*,
on_progress: Callable[[int, int], None] | None = None,
) -> list[list[float]]:
if any(self.EMBED_FAIL in t for t in texts):
raise EmbeddingError("simulated embed failure (test sentinel)")
return await super().embed(texts)
return await super().embed(texts, on_progress=on_progress)
def _pdf_llm[
tests/unit/test_sync_button.py
+326 -27
View File
@@ -38,16 +38,34 @@ file" (remove-when-null) rule is gone.
Phase 128 (task 01): the label becomes PARTS — #sync-label ships a
main span (the idle text pre-JS) + a hidden count span (the pinned
"(n/m)"), the builders return { main, count }, and the single writer
setSyncLabel(main, count) lands the parts + the untruncated full
label (main + " " + count) on the title + #sync-result; the
running-state / settle pins move onto that shape (their labels stay
byte-identical as full strings).
setSyncLabel(main, count, chunk — the chunk part is phase 129) lands
the parts + the untruncated full label (main + " " + count +
" " + chunk) on the title + #sync-result; the running-state /
settle pins move onto that shape (their labels stay byte-identical
as full strings). Phase 129 (task 04): #sync-label ships a THIRD
hidden span (the pinned "· chunk n/m" — the in-flight file's
embed-batch position), rendered only while a file's embed batch is in
flight (the import phase after the first batch — hidden in every
other state); the markup pin below pins the exact three-part ship
shape.
Phase 129 (task 03): the status contract gains the in-flight file's
EMBED-chunk position — ``current_chunk`` (the 1-based first chunk of
the in-flight token-budgeted batch) + ``chunks_total`` (the file's
total content chunks): the idle dict pins them 0/0, the running
dict's pins gain them, and the new mid-run pair test reads the
positions between ticks (1/38 → 13/38) through the runner's own
``_chunk_hook`` closure; the convention split is pinned — the chunk
position clears at every point the import phase ends (the
overview/summaries transitions + both terminals) while the FILE
counts keep their final values.
"""
from __future__ import annotations
import asyncio
import re
import threading
import time
from collections.abc import Callable, Iterator, Sequence
from datetime import datetime
from pathlib import Path
@@ -130,23 +148,31 @@ def test_sync_button_has_icon_and_label_parts() -> None:
Phase 128 (task 01): #sync-label is PARTS — the MAIN span
(.sync-label-main) carries the idle text (the pre-JS label renders
from the static markup; the part CSS may ellipsize it), and the
COUNT span (.sync-label-count — the pinned "(n/m)" position, never
clipped) ships hidden + empty (there is no count at idle). The
class/id stay (the module query + the E2E label assertions)."""
from the static markup; the part CSS may ellipsize it), the
COUNT span (.sync-label-count — the pinned "(n/m)" position,
never clipped) ships hidden + empty (there is no count at idle),
and — phase 129 (task 04) — the CHUNK span (.sync-label-chunk —
the pinned "· chunk n/m" embed-batch position) ships hidden +
empty too (no chunk position at idle — the part renders only
while a file's embed batch is in flight). The class/id stay (the
module query + the E2E label assertions)."""
text = _text(SHELL_HTML)
btn = text[text.find('id="sync-btn"') : text.find("</button>", text.find('id="sync-btn"'))]
assert re.search(r'<svg[^>]*class="sync-icon"[^>]*aria-hidden="true"', btn)
# Phase 128 (task 01): the parts markup — the exact shape (the main
# span holds the idle text, the count span is hidden + empty).
# Phase 128 (task 01) + Phase 129 (task 04): the parts markup —
# the exact three-part shape (the main span holds the idle text,
# the count + chunk spans are hidden + empty).
label = re.search(
r'<span\b[^>]*id="sync-label"[^>]*>'
r'<span class="sync-label-main">Sync sources</span>'
r'<span class="sync-label-count" hidden></span>'
r'<span class="sync-label-chunk" hidden></span>'
r"</span>",
btn,
)
assert label, ("the #sync-label parts markup (main + hidden count) is missing")
assert label, (
"the #sync-label parts markup (main + hidden count + hidden chunk) is missing"
)
assert 'class="sync-label"' in label.group(0)
@@ -299,9 +325,11 @@ def test_sources_js_running_state_is_never_stale() -> None:
Phase 128 (task 01): the entry hands the label PARTS (the
phase-aware split — sync + status → fmtSyncPhaseLabel, the
fall-through → the byte-identical fmtSyncLabel) to the SINGLE
writer, setSyncLabel(main, count), which lands the main part, the
pinned count part (hidden when absent), and the FULL untruncated
label on the button title + #sync-result (A4) — the direct span /
writer, setSyncLabel(main, count, chunk — phase 129: the chunk
part destructures with a "" default), which lands the main part,
the pinned count part (hidden when absent), the pinned chunk part
(phase 129 — hidden when absent), and the FULL untruncated label
on the button title + #sync-result (A4) — the direct span /
title / announcer writes are GONE from the entry (they live in
setSyncLabel)."""
js = _text(SOURCES_JS)
@@ -313,7 +341,7 @@ def test_sources_js_running_state_is_never_stale() -> None:
assert "fmtSyncLabel(kind, currentFile, done, total)" in body
# …and the single writer lands the parts (A4 in it — the title is
# set for EVERY running label: the file-only rule is adjusted away).
assert "setSyncLabel(main, count)" in body
assert "setSyncLabel(main, count, chunk)" in body
assert body.count("setSyncLabel(") == 1, "the entry writes ONLY through the parts writer"
assert "syncBtn.title =" not in body, ("the title write lives in setSyncLabel (A4)")
assert "syncLabel.textContent" not in body, ("no direct span write — the parts")
@@ -759,7 +787,14 @@ class _GatedImport:
progress-shaped call goes through the real hook closure — the
closure under test), parks on a threading gate so the test can read
the status mid-run, then returns the canned summary (or raises
``fail``)."""
``fail``).
Phase 129 (task 03): optionally ALSO fires the in-flight file's
chunk-position pairs (``chunk_pairs``) through the runner's OWN
``_chunk_hook`` closure: pair one lands before the main park
(readable mid-run), and every later pair lands after the main
park + its own park (``chunk_release``), so each position is
readable on the status between ticks."""
def __init__(
self,
@@ -767,6 +802,8 @@ class _GatedImport:
started: threading.Event,
release: threading.Event,
fail: BaseException | None = None,
chunk_pairs: Sequence[tuple[int, int]] = (),
chunk_release: threading.Event | None = None,
) -> None:
self.summary = summary
self.started = started
@@ -774,6 +811,9 @@ class _GatedImport:
self.fail = fail
self.hook_calls: list[tuple[str, str, int, int]] = []
self.prune_flags: list[bool] = []
self.chunk_pairs = list(chunk_pairs)
self.chunk_release = chunk_release
self.chunk_hook_calls: list[tuple[int, int]] = []
async def __call__(
self,
@@ -784,6 +824,7 @@ class _GatedImport:
limit: int | None = None,
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -792,8 +833,26 @@ class _GatedImport:
if progress is not None:
progress("repo", "notes/deep.md", 1, 3)
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
self.started.set()
await asyncio.to_thread(self.release.wait) # park without freezing the worker loop
if on_chunk_progress is not None and self.chunk_pairs:
# Phase 129 (task 03): the file's chunk-position pairs,
# fired through the runner's OWN hook closure (the closure
# under test) — pair one before the main park, each later
# pair after the main park + its own park (each park lets
# the test read the status mid-run). The runner passes the
# hook only when the wiring is intact — a regression to
# None skips this branch and the pair tests time out.
on_chunk_progress(*self.chunk_pairs[0])
self.chunk_hook_calls.append(self.chunk_pairs[0])
self.started.set()
await asyncio.to_thread(self.release.wait)
for pair in self.chunk_pairs[1:]:
on_chunk_progress(*pair)
self.chunk_hook_calls.append(pair)
if self.chunk_release is not None:
await asyncio.to_thread(self.chunk_release.wait)
else:
self.started.set()
await asyncio.to_thread(self.release.wait) # park without freezing the worker loop
if self.fail is not None:
raise self.fail
return self.summary
@@ -896,9 +955,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
fresh_sync_status: None,
) -> None:
"""Idle: the three phase-64 progress keys ride along as null/0/0,
the four phase-98 phase keys as null/0/0/0, and EVERY pre-existing
key is unchanged — the full response dict is pinned, so the
current UI and every existing consumer keep working."""
the two phase-129 chunk keys as 0/0, the four phase-98 phase keys
as null/0/0/0, and EVERY pre-existing key is unchanged — the full
response dict is pinned, so the current UI and every existing
consumer keep working."""
assert sync_api.sync_status() == {
"state": "idle",
"started_at": None,
@@ -908,6 +968,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
"current_file": None,
"files_done": 0,
"files_total": 0,
# Phase 129 (task 03): the chunk-position keys — 0/0 idle
# (the dataclass defaults, no special-casing).
"current_chunk": 0,
"chunks_total": 0,
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
"phase": None,
"current_summary": None,
@@ -938,7 +1002,8 @@ def test_mid_run_status_reports_current_file(
assert clone.started.wait(5.0), "the run never reached the clone phase"
# Clone/pull prelude: running — but no file yet (A4: bare
# "Syncing…") and no phase yet either (D1: the prelude stays
# null — the bare label's pin).
# null — the bare label's pin), and no chunk position yet
# (phase 129: 0/0 before the first embed batch).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] is None
@@ -946,11 +1011,14 @@ def test_mid_run_status_reports_current_file(
assert s["current_summary"] is None
assert s["files_done"] == 0 and s["files_total"] == 0
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Import phase: the hook's file is live on the status, and the
# phase machine names the phase (the summary span has not
# started — its counters are still 0/0).
# started — its counters are still 0/0); no embed batch has
# fired the chunk hook (this canned import passes no pairs),
# so the chunk position stays 0/0.
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
@@ -959,6 +1027,7 @@ def test_mid_run_status_reports_current_file(
assert s["files_total"] == 3
assert s["current_summary"] is None
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
fake_import.release.set()
finally:
clone.release.set()
@@ -974,7 +1043,9 @@ def test_mid_run_status_reports_current_file(
# Success terminal: current_file null, final file counts retained;
# the phase machine clears phase + current_summary — the fake
# folder step never fired the summary hook, so the summary counts
# stay 0/0 (the keep-final-counts convention keeps what fired).
# stay 0/0 (the keep-final-counts convention keeps what fired);
# the chunk position clears with the file (phase 129 — it is a
# sub-position of a done file, never a keep-final counter).
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
@@ -982,6 +1053,7 @@ def test_mid_run_status_reports_current_file(
assert s["current_file"] is None
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["error"] is None
assert s["started_at"] is not None and s["finished_at"] is not None
@@ -993,7 +1065,13 @@ def test_failed_terminal_clears_current_file_keeps_counts(
) -> None:
"""Terminal (failed): a run that dies inside the import clears
``current_file`` but keeps the hook's final counts — the last
position is useful context next to the (sanitized) error."""
position is useful context next to the (sanitized) error.
Phase 129 (task 03): the convention split extends to the chunk
position — the (about-to-fail) import reports a live pair
(1/38), the failed terminal clears it to 0/0 (a sub-position of a
file that's done/abandoned), while the FILE counts keep their
final values next to the error."""
clone = _GatedClone(threading.Event(), threading.Event())
fake_import = _GatedImport(
ImportSummary(),
@@ -1003,6 +1081,7 @@ def test_failed_terminal_clears_current_file_keeps_counts(
"embeddings request to https://u:p@aipi.example.com/v1 "
"failed: connection refused"
),
chunk_pairs=[(1, 38)], # phase 129: the live pair before the raise
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
@@ -1010,11 +1089,15 @@ def test_failed_terminal_clears_current_file_keeps_counts(
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
clone.release.set()
assert fake_import.started.wait(5.0), "the progress hook never fired"
# While the (about-to-fail) import is parked: the file is live.
assert fake_import.started.wait(5.0), "the chunk pair never landed"
# While the (about-to-fail) import is parked: the file is live,
# and so is its chunk position (the pair the fake fired through
# the runner's own closure).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 1 and s["chunks_total"] == 38
fake_import.release.set()
finally:
clone.release.set()
@@ -1022,6 +1105,8 @@ def test_failed_terminal_clears_current_file_keeps_counts(
thread.join(10.0)
assert not thread.is_alive()
assert errors == []
# Wiring: the pair fired through the runner's own closure.
assert fake_import.chunk_hook_calls == [(1, 38)]
s = sync_api.sync_status()
assert s["state"] == "failed"
assert s["current_file"] is None # cleared in the terminal state
@@ -1029,8 +1114,222 @@ def test_failed_terminal_clears_current_file_keeps_counts(
assert s["current_summary"] is None
assert s["files_done"] == 1 and s["files_total"] == 3 # final counts kept
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
# Phase 129 (task 03): the chunk position clears in the FAILED
# terminal too — the convention split, pinned (the FILE counts
# above keep their final values, the sub-position does not).
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["detail"] == {}
error = s["error"] or ""
assert "*****@aipi.example.com" in error # credentials masked
assert "u:p" not in error
assert "connection refused" in error # the reason survives
# ---------- phase 129 (task 03): the in-flight file's embed-chunk ----------
# ---------- position (current_chunk / chunks_total) ----------------------
#
# Same seams as the phase-64 block above: the fake import fires the
# runner's OWN ``_chunk_hook`` closure (the task under test) with the
# file's chunk-position pairs, parking between pairs so the test reads
# the status BETWEEN ticks (the A6 batch-granularity contract: 14 =
# the first chunk of the in-flight token-budgeted batch).
def _poll_until(
predicate: Callable[[], bool],
timeout: float = 5.0,
what: str = "condition",
) -> None:
"""Poll (main thread) until *predicate* — the worker-thread run's
state is only observable between its parks."""
deadline = time.monotonic() + timeout
while not predicate():
if time.monotonic() >= deadline:
raise AssertionError(f"timed out waiting for {what}")
time.sleep(0.01)
def test_mid_run_status_reports_chunk_position_between_ticks(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""Phase 129 (task 03): the status carries the in-flight file's
embed-chunk position — the hook's (first, total) pair climbs
through the file's batches (1/38 → 13/38, read between ticks),
paired with the file the ``progress`` hook named; the prelude
reports 0/0 (no embed batch yet), and the success terminal clears
the position to 0/0 while the FILE counts keep their final values
(the convention split — the chunk position is a sub-position of a
done file, never a keep-final counter)."""
clone = _GatedClone(threading.Event(), threading.Event())
import_release = threading.Event()
chunk_release = threading.Event()
fake_import = _GatedImport(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
import_release,
chunk_pairs=[(1, 38), (13, 38)], # two batches of a 38-chunk file
chunk_release=chunk_release,
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
# Prelude: the chunk position is 0/0 (no file, no batch yet —
# alongside the bare-label pins of the phase-64 test above).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["current_file"] is None
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
clone.release.set()
assert fake_import.started.wait(5.0), "the first chunk pair never landed"
# Import phase, between ticks: pair one is live on the status,
# paired with the file the progress hook named (the two hooks
# are paired — the position belongs to that file).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 1 and s["chunks_total"] == 38
# Release pair one: the import fires pair two (13/38 — the
# second batch of the 38-chunk file) and parks again.
import_release.set()
_poll_until(
lambda: sync_api.sync_status()["current_chunk"] == 13,
what="the second chunk pair on the status",
)
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
# The position CLIMBED (batch granularity) — total unchanged
# (the file's chunk count, not the batch size).
assert s["current_chunk"] == 13 and s["chunks_total"] == 38
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
chunk_release.set()
finally:
clone.release.set()
import_release.set()
chunk_release.set()
thread.join(10.0)
assert not thread.is_alive()
assert errors == [], f"the runner raised: {errors!r}"
# Wiring: the pairs fired through the runner's OWN closure (the
# status reads above came through it; a wiring regression to
# None would have skipped the firing branch and timed out).
assert fake_import.chunk_hook_calls == [(1, 38), (13, 38)]
# Success terminal: the chunk position clears (0/0) ...
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
# ... while the FILE counts keep their final values (the
# convention split, pinned alongside the phase-64 terminal pin).
assert s["current_file"] is None
assert s["files_done"] == 1 and s["files_total"] == 3
def test_chunk_position_clears_at_phase_transitions_and_terminal(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""Phase 129 (task 03): the chunk position is CLEARED (0/0) at
every point the import phase ends. This run's summary changes the
KB, so BOTH post-import transitions run and are pinned: the
import→``"overview"`` transition and the import→``"summaries"``
transition (the changed-KB branch) each clear it to 0/0, and the
success terminal keeps it 0/0 — through the whole span the FILE
counts keep their final values (the convention split: the file
counts survive for the result line's context, the chunk position
is a sub-position of a file that's done)."""
clone = _GatedClone(threading.Event(), threading.Event())
import_release = threading.Event()
overview_started = threading.Event()
overview_release = threading.Event()
folders_started = threading.Event()
folders_release = threading.Event()
fake_import = _GatedImport(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
import_release,
chunk_pairs=[(7, 38)], # a live pair to clear at each transition
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
# Park the post-import steps (re-monkeypatched over the seams
# helper's no-ops): the test reads the status DURING the
# "overview" phase and DURING the "summaries" phase — the two
# import-phase-ending transitions the task pins.
async def gated_overview(llm: object, session: object = None) -> bool:
overview_started.set()
await asyncio.to_thread(overview_release.wait)
return True
async def gated_folders(
db: object,
llm: object,
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
folders_started.set()
await asyncio.to_thread(folders_release.wait)
return {"generated": 0, "failed": 0, "pruned": 0}
monkeypatch.setattr(sync_api, "regenerate_overview", gated_overview)
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
clone.release.set()
assert fake_import.started.wait(5.0), "the chunk pair never landed"
# Mid-import: the live pair (7/38) is on the status.
s = sync_api.sync_status()
assert s["phase"] == "import"
assert s["current_chunk"] == 7 and s["chunks_total"] == 38
# Release the import: it returns (added+updated > 0) → the
# phase machine moves to "overview" and the position clears.
import_release.set()
assert overview_started.wait(5.0), "the run never reached the overview"
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "overview"
# CLEARED at the import→overview transition (0/0) ... the file
# counts KEEP their final values (the convention split).
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["files_done"] == 1 and s["files_total"] == 3
overview_release.set()
# ... and the changed-KB branch moves to "summaries" (the
# second import-phase-ending transition — cleared there too,
# idempotently) before parking.
assert folders_started.wait(5.0), "the run never reached the summaries"
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "summaries"
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["files_done"] == 1 and s["files_total"] == 3
folders_release.set()
finally:
clone.release.set()
import_release.set()
overview_release.set()
folders_release.set()
thread.join(10.0)
assert not thread.is_alive()
assert errors == [], f"the runner raised: {errors!r}"
assert fake_import.chunk_hook_calls == [(7, 38)]
# Success terminal: still 0/0 (the terminal clears too — same
# spots as current_file), file counts kept.
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
assert s["current_file"] is None
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["files_done"] == 1 and s["files_total"] == 3