diff --git a/.agents/phases/todo/129_sync_chunk_progress/00_phase.md b/.agents/phases/complete/129_sync_chunk_progress/00_phase.md similarity index 100% rename from .agents/phases/todo/129_sync_chunk_progress/00_phase.md rename to .agents/phases/complete/129_sync_chunk_progress/00_phase.md diff --git a/.agents/phases/todo/129_sync_chunk_progress/01_llm_embed_hook.md b/.agents/phases/complete/129_sync_chunk_progress/01_llm_embed_hook.md similarity index 100% rename from .agents/phases/todo/129_sync_chunk_progress/01_llm_embed_hook.md rename to .agents/phases/complete/129_sync_chunk_progress/01_llm_embed_hook.md diff --git a/.agents/phases/todo/129_sync_chunk_progress/02_importer_chunk_progress.md b/.agents/phases/complete/129_sync_chunk_progress/02_importer_chunk_progress.md similarity index 100% rename from .agents/phases/todo/129_sync_chunk_progress/02_importer_chunk_progress.md rename to .agents/phases/complete/129_sync_chunk_progress/02_importer_chunk_progress.md diff --git a/.agents/phases/todo/129_sync_chunk_progress/03_sync_status_chunk_fields.md b/.agents/phases/complete/129_sync_chunk_progress/03_sync_status_chunk_fields.md similarity index 100% rename from .agents/phases/todo/129_sync_chunk_progress/03_sync_status_chunk_fields.md rename to .agents/phases/complete/129_sync_chunk_progress/03_sync_status_chunk_fields.md diff --git a/.agents/phases/todo/129_sync_chunk_progress/04_ui_chunk_part.md b/.agents/phases/complete/129_sync_chunk_progress/04_ui_chunk_part.md similarity index 100% rename from .agents/phases/todo/129_sync_chunk_progress/04_ui_chunk_part.md rename to .agents/phases/complete/129_sync_chunk_progress/04_ui_chunk_part.md diff --git a/.agents/phases/todo/129_sync_chunk_progress/05_e2e_chunk_progress.md b/.agents/phases/complete/129_sync_chunk_progress/05_e2e_chunk_progress.md similarity index 100% rename from .agents/phases/todo/129_sync_chunk_progress/05_e2e_chunk_progress.md rename to .agents/phases/complete/129_sync_chunk_progress/05_e2e_chunk_progress.md diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a1.err b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a1.md b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a1.md new file mode 100644 index 0000000..0be074c --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a1.md @@ -0,0 +1,12 @@ +All green. Final report: + +**Phase 129 final verification pass — complete** + +- Verified implementation on-spec: `LLMClient.embed` per-batch `on_progress` hook, `import_sources(on_chunk_progress=…)` plumbed through text/image/PDF branches (summary-chunk embed excluded), `SyncStatus.current_chunk/chunks_total` with reset/clear contract, UI `.sync-label-chunk` third pinned part +- `uv run pytest --cov=app --cov-report=term-missing`: **3041 passed, 0 failed**, `app/` coverage **99%** (>90% gate) +- `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` (isolated, DB up): **2 passed** — endpoint chunk-position climb (total = true `chunk_document` count) + label `· chunk n/m` part with untruncated `title`/`#sync-result` +- `uv run ruff check .`: all checks passed · `uv run pyright`: 0 errors, 0 warnings +- Completion criteria: all 5 ✅ (incl. no behavior change in completed phases — full existing suite green, CLI/chat paths byte-identical no-op) +- Docs: README.md — fixed 2 pre-existing inconsistencies: (1) archive-upload section claimed "unpacks + scans immediately" but phase 90 made uploads unpack+register only (scan deferred to the Sync button); (2) troubleshooting claimed a "120s client timeout" but the real guard is the 300 s pre-token `TURN_TIMEOUT_MS`. Phase-129 label behavior already documented ("Sync from the UI"). AGENTS.md — verified consistent (user stories, no-CDN test, gpgsign, quick-ref commands all checked) — no changes +- Removed stray untracked `NOTES.md` (8-byte `# Notes` artifact); no live-infra changes; nothing committed (left in working tree for the harness) +- Next pending phase: `130_chat_answer_clusters` diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a1.validate b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a1.validate new file mode 100644 index 0000000..0f8a218 --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a1.validate @@ -0,0 +1,114 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 18%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 66%] +........................................................................ [ 68%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 75%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 85%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 97%] +........................................................................ [ 99%] +................ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 156 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5152 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a2.err b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a2.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a2.md b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a2.md new file mode 100644 index 0000000..0b98e62 --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a2.md @@ -0,0 +1,12 @@ +All green. Final report: + +**Phase 129 final verification pass — 100% green** (previous work intact; this pass re-verified everything and fixed two pre-existing README inconsistencies) + +- Verified on-spec: `LLMClient.embed` per-batch `on_progress` hook; importer `on_chunk_progress` plumbing (text/image/PDF branches; summary embed excluded); `SyncStatus.current_chunk/chunks_total` reset+clear contract; UI `.sync-label-chunk` pinned part with untruncated `title`/`#sync-result` +- `uv run pytest --cov=app -q`: exit 0, ~3041 passed, 0 failed, `app/` coverage **99%** (>90% gate) +- `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` (isolated, DB up): **2 passed** (endpoint chunk climb with true `chunk_document` total + UI `· chunk n/m` part) +- `uv run ruff check .`: all checks passed · `uv run pyright`: 0 errors, 0 warnings +- Completion criteria: all 5 ✅ (incl. no behavior change in completed phases — full suite green, CLI/chat no-op paths byte-identical) +- Docs: README.md fixed 2 pre-existing inconsistencies (upload section claimed "unpacks + scans immediately" — phase 90 made it unpack+register only, scan deferred to Sync; troubleshooting claimed "120s client timeout" — actual guard is the 300 s pre-token `TURN_TIMEOUT_MS`). Phase-129 label behavior already documented. AGENTS.md re-verified consistent — no changes +- Removed stray untracked `NOTES.md` (8-byte artifact); no live-infra changes; no commits — all changes left in working tree for the harness +- Next pending phase: `130_chat_answer_clusters` diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a2.validate b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a2.validate new file mode 100644 index 0000000..0f8a218 --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__00_phase.a2.validate @@ -0,0 +1,114 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 18%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 66%] +........................................................................ [ 68%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 75%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 85%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 97%] +........................................................................ [ 99%] +................ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 156 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5152 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__01_llm_embed_hook.a1.err b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__01_llm_embed_hook.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__01_llm_embed_hook.a1.md b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__01_llm_embed_hook.a1.md new file mode 100644 index 0000000..7dd98aa --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__01_llm_embed_hook.a1.md @@ -0,0 +1,17 @@ +Task 01 complete. All gates green. + +**Implemented** +- `app/rag/llm.py`: `LLMClient.embed` gained keyword-only `on_progress: Callable[[int, int], None] | None = None`; fires `on_progress(len(out) + 1, len(texts))` before each top-level batch request (before the try, so a failing batch still announces its start). Docstring documents the contract, the halving-backstop's non-sub-reporting (approximate position, rare by construction), and the `None` zero-cost no-op. +- `tests/unit/test_llm_client.py`: 6 new tests — hook sequence asserted against `_batch_texts`' actual partition (count-cap and token-budget splits, first-indices strictly increasing, total always `len(texts)`), failing-batch announcement, halving does not sub-report, `None`/default byte-identical (same calls + vectors + counter), empty `texts` → no calls. Test fake `_FakeEmbeddingsService` gained a `fail_from` knob (test-only). +- No existing `embed`/`embed_one` call sites touched (`_batch_texts`/`_embed_batch`/`embed_batches`/`embed_one` internals untouched). + +**Results** +- `uv run pytest` → 3030/3030 passed, exit 0 (100% green; no F/E in progress output) +- `uv run pytest --cov=app --cov-report=term-missing` → exit 0, `app/` TOTAL **99%** (>90%); `app/rag/llm.py` 99% (only miss is pre-existing unreachable line 509) +- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors, 0 warnings + +**Docs**: no doc impact — internal optional hook only; no env vars, API, CLI, or setup changes (the sync endpoint/UI changes land in tasks 03/04). + +**Decisions**: `Callable` imported from `collections.abc` (matches `folder_summaries.py` hook precedent). + +**Next pending task**: `.agents/phases/todo/129_sync_chunk_progress/02_importer_chunk_progress.md` diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__01_llm_embed_hook.a1.validate b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__01_llm_embed_hook.a1.validate new file mode 100644 index 0000000..2f848fb --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__01_llm_embed_hook.a1.validate @@ -0,0 +1,114 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 68%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 95%] +........................................................................ [ 97%] +........................................................................ [ 99%] +...... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5135 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__02_importer_chunk_progress.a1.err b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__02_importer_chunk_progress.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__02_importer_chunk_progress.a1.md b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__02_importer_chunk_progress.a1.md new file mode 100644 index 0000000..45989dd --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__02_importer_chunk_progress.a1.md @@ -0,0 +1,16 @@ +Task 02 complete. Report: + +**Implemented** +- `app/rag/importer.py`: `import_sources(on_chunk_progress=…)` keyword-only hook (documented beside the `progress` paragraph) → `_index_file(on_chunk=…)` → all three branches (text, image, PDF) → `_replace_content_chunks(on_chunk=…)` → `llm.embed(…, on_progress=on_chunk)`; `Embedder` protocol extended with the task-01 `on_progress` kwarg +- Re-chunk retry naturally reports against the new chunk list (documented); `_store_summary`'s summary-chunk embed never receives the hook; CLI (`scripts/import_docs.py`) passes nothing — byte-identical +- Test fakes made protocol-conform (`tests/fakes.py` single-batch announce + 5 importer-facing fakes forward the kwarg); 4 new unit tests in `tests/unit/test_importer.py` (multi-batch per-file sequences, single-batch + summary exclusion, image `1/1`, PDF content chunks) + +**Verification** +- `uv run pytest --cov=app --cov-report=term-missing` → 3034 passed, `app/` 99% (>90%) +- `uv run ruff check . && uv run pyright` → clean (0 errors) + +**Docs:** no doc impact (internal plumbing; CLI/API/env/UI unchanged — those land in tasks 03/04). + +**Notable decision:** the PDF's embedded-image docs and the phase-125 post-index enrichment pass do NOT receive the hook — they index separate documents, and their `1/1` positions would be A6 "noise after the real progress ends"; the task's work items scope the hook to the walked file's three branch content embeds. + +**Next pending task:** `03_sync_status_chunk_fields.md` (SyncStatus fields, closure, clearing, endpoint). diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__02_importer_chunk_progress.a1.validate b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__02_importer_chunk_progress.a1.validate new file mode 100644 index 0000000..ace63b7 --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__02_importer_chunk_progress.a1.validate @@ -0,0 +1,114 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 18%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 68%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 75%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 97%] +........................................................................ [ 99%] +.......... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 139 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5135 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__03_sync_status_chunk_fields.a1.err b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__03_sync_status_chunk_fields.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__03_sync_status_chunk_fields.a1.md b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__03_sync_status_chunk_fields.a1.md new file mode 100644 index 0000000..019f725 --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__03_sync_status_chunk_fields.a1.md @@ -0,0 +1,16 @@ +All checks pass. The task is complete — here's the report: + +**Implemented (task 03 — `SyncStatus` chunk fields):** +- `app/api/sync.py`: `SyncStatus` gains `current_chunk`/`chunks_total` (defaults 0/0); module + dataclass + endpoint docstrings document the clearing contract +- `_run_sync`: run-start reset, `_chunk_hook` closure passed as `on_chunk_progress=` to `import_sources`, cleared at every import-phase-ending point (overview transition, both summaries transitions, both terminals) while file counts keep-final +- `GET /api/sync/status` exposes both fields after `files_*` +- Tests: `tests/unit/test_sync_button.py` (idle dict pin, mid-run pair test 1/38→13/38 read between ticks, phase-transition + terminal clearing test with gated overview/summaries, failed-terminal convention split, prelude 0/0 pins; `_GatedImport` extended with chunk pairs); `tests/integration/test_sync_api.py` (idle/phase-machine pins + fake signature updates) + +**Verification (all green):** +- `uv run pytest -q` (unit+integration): 100% pass +- `uv run pytest --cov=app --cov-report=term-missing`: TOTAL 99% (>90%); `app/api/sync.py` 100% +- `uv run ruff check .`: clean; `uv run pyright`: 0 errors, 0 warnings + +**Docs:** no doc impact — README/AGENTS.md don't enumerate the status JSON fields (the user-visible label change lands in task 04). + +**Next pending task:** `04_ui_chunk_part.md` (the label's `.sync-label-chunk` part + unit pins). diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__03_sync_status_chunk_fields.a1.validate b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__03_sync_status_chunk_fields.a1.validate new file mode 100644 index 0000000..4cf17d6 --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__03_sync_status_chunk_fields.a1.validate @@ -0,0 +1,114 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 18%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 68%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 75%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 97%] +........................................................................ [ 99%] +............ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 156 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5152 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a1.err b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a1.md b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a1.md new file mode 100644 index 0000000..8fc975f --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a1.md @@ -0,0 +1,20 @@ +Everything is green and the working tree is in the expected state (no git add/commit; tasks 01–03's changes remain for the harness's phase commit). + +**Task 04 complete — the label's chunk part: `Syncing… (2/3) · chunk 14/38`** + +- `index.html`: third part `` after the count span in `#sync-label`; announcer comment updated +- `styles.css`: `.sync-label-chunk` pinned-part rule (flex 0 0 auto / nowrap / --mono) right after `.sync-label-count`; container + mobile comments updated +- `sources.js`: null-guarded `syncLabelChunk` ref; `setSyncLabel(mainText, countText = "", chunkText = "")` (same present/absent contract; full = main + count + chunk rides title + `#sync-result`, A4); `fmtSyncPhaseLabel` import fall-through emits `· chunk n/m` only while `chunks_total > 0`, overview/summaries/bare builders pass none; `enterSyncRunningState` destructures `chunk = ""` default +- Unit pins: 4 new tests (markup, CSS, node harness of the real JS across all 8 label states incl. exact title `Syncing… src/a/b.md (2/3) · chunk 14/38`, only-fall-through-reads-chunk-fields) + updated phase-128 writer/markup pins in both unit files +- Fixed completed-phase regressions the new part caused: phase-128 E2E import snapshot/title regex and phase-90 E2E label/title regexes now accept the chunk part (Playwright `to_have_text` is full-match — verified empirically) + +**Results** +- `uv run pytest --cov=app --cov-report=term-missing` → 3040 passed, app/ 99% (>90%) +- `uv run ruff check .` → clean; `uv run pyright` → 0 errors +- E2E in isolation: `test_sync_label_mobile.py` 4/4, `test_sync_upload_progress.py` 4/4, `test_sync_summary_visibility.py` 3/3 +- Live probes: status endpoint reported `58/61` mid-import (file's true chunk count) and 0/0 terminal; real browser rendered the chunk part in 132 frames with the untruncated title + +- Docs: README sync States bullet documents the `· chunk n/m` label + title/announcer; no AGENTS.md impact +- Notable: the pre-first-batch window is milliseconds, so running-label E2E assertions had to be chunk-aware rather than relying on a 0/0 frame + +Next pending task: `05_e2e_chunk_progress.md` (dedicated Playwright suite, run in isolation). diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a1.validate b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a1.validate new file mode 100644 index 0000000..0f8a218 --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a1.validate @@ -0,0 +1,114 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 18%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 66%] +........................................................................ [ 68%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 75%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 85%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 97%] +........................................................................ [ 99%] +................ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 156 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5152 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a2.err b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a2.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a2.md b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a2.md new file mode 100644 index 0000000..1007c43 --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a2.md @@ -0,0 +1,10 @@ +All green — the prior rejection was a missing final report, not a failure; the working tree was left intact and now re-verified. + +**Report (task 04 — the label's chunk part `· chunk n/m`)** +- Implemented: `#sync-label` third pinned part (hidden `span.sync-label-chunk` in `index.html`; `.sync-label-chunk` mono/pinned CSS after `.sync-label-count`); `sources.js` — null-guarded ref, `setSyncLabel(mainText, countText = "", chunkText = "")` with the untruncated main+count+chunk riding `title` + `#sync-result` (A4), `fmtSyncPhaseLabel` import fall-through emits `· chunk n/m` only while `chunks_total > 0` (overview/summaries/prelude/upload/202-click/idle all pass none), `enterSyncRunningState` destructures `chunk = ""` +- Unit pins: 4 new tests in `tests/unit/test_frontend_sync_upload.py` (markup, CSS, node harness of the real JS across all label states incl. exact title `Syncing… src/a/b.md (2/3) · chunk 14/38`, fall-through-only-reads-chunk-fields) + updated phase-128 pins in `test_sync_button.py`; chunk-aware regexes in phase-128 and phase-90 E2E (Playwright `to_have_text` is full-match) +- Results: `uv run pytest --cov=app --cov-report=term-missing` → **3040 passed**, app/ **99%** (>90%); `uv run ruff check .` → clean; `uv run pyright` → 0 errors +- E2E (isolation): `test_sync_label_mobile.py` 4/4 (re-verified this session); `test_sync_upload_progress.py` 4/4 and `test_sync_summary_visibility.py` 3/4→3/3 passed on identical code last session +- Docs: README sync States bullet documents the chunk part + title/announcer; no AGENTS.md impact +- Decisions: chunk part renders only post-first-batch (the 0/0 window is milliseconds); live browser probe last session confirmed 132 rendered chunk frames + 0/0 terminal +- Next pending task: `05_e2e_chunk_progress.md` diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a2.validate b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a2.validate new file mode 100644 index 0000000..0f8a218 --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__04_ui_chunk_part.a2.validate @@ -0,0 +1,114 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 18%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 66%] +........................................................................ [ 68%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 75%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 85%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 97%] +........................................................................ [ 99%] +................ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 156 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5152 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a1.err b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a1.md b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a1.md new file mode 100644 index 0000000..9d328de --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a1.md @@ -0,0 +1,15 @@ +All gates are green. The task is complete — final report: + +**Phase 129, task 05 complete** — `tests/e2e/test_sync_chunk_progress.py` (new, mock-only, house pattern): +- **Test 1 (endpoint, slow leg):** ~100 ms tight-poll recorder proves `chunks_total` == the deterministic `chunk_document` count (61 — the file's TRUE chunk count, computed in-test with the importer's exact args; sizing knobs leak-guard-pinned to code defaults) while `current_chunk` climbs strictly 1 → 60 through the 30 token-budgeted batches (every value a `_batch_texts` boundary; last tick = final batch's first index); the tiny second file restarts to `(1, 1)` at `files_done/total == 2/2`; overview/summaries ticks + the terminal are 0/0 while file counts keep-final (2/2) +- **Test 2 (UI, slow leg):** the page's 2 s poll renders `Syncing… chunkprog/big.md` + `(1/2)` + visible `· chunk n/61`; title + `#sync-result` carry the untruncated concatenation (atomic in-browser check); after the file's span the chunk part is hidden + empty while the label keeps rendering in sync; settle = `Synced HH:MM`, parts hidden, `2 added` +- Fixture: `chunkprog/` local source, ~63.5 KB `big.md` (61 chunks) + one-chunk `small.md`; `SLOW_DELAY_S=1.5` so each embed batch outlives the 100 ms cadence ~15× and the import span (~45 s) the 2 s UI poll ~20× + +**Results (all green):** +- `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` → 2 passed (126 s, DB up) +- `uv run pytest --cov=app --cov-report=term-missing` → 3040 passed, `app/` coverage **99%** +- `uv run ruff check .` → clean; `uv run pyright` → 0 errors + +**Docs:** no impact (test-only task; README/AGENTS.md reference E2E files only as examples). +**Notable:** Playwright's `wait_for_function` takes `arg=` keyword-only in this version; ports 8165/8906 (unused by other suites). +**Next pending task:** phase 129 is complete (task 05 was its last); next phase is `130_chat_answer_clusters` in `.agents/phases/todo/`. diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a1.validate b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a1.validate new file mode 100644 index 0000000..0f8a218 --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a1.validate @@ -0,0 +1,114 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 18%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 66%] +........................................................................ [ 68%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 75%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 85%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 97%] +........................................................................ [ 99%] +................ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 156 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5152 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a2.err b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a2.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a2.md b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a2.md new file mode 100644 index 0000000..60e16b2 --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a2.md @@ -0,0 +1,9 @@ +Documentation check complete: README.md's sync paragraph matches the implemented behavior (label parts, batch-granularity chunk position, untruncated title/announcer) and neither file documents the status fields individually, so nothing is stale; AGENTS.md needs no changes (all referenced commands/files verified present). + +**Final report — Phase 129, task 05 (E2E chunk progress):** + +- **Implemented:** `tests/e2e/test_sync_chunk_progress.py` (new, mock-only, house pattern): test 1 — ~100 ms recorder on `GET /api/sync/status` proves `chunks_total` == the deterministic `chunk_document` count (61 — the file's TRUE chunk count, computed in-test with the importer's exact args, sizing knobs leak-guard-pinned to code defaults) while `current_chunk` climbs strictly 1 → 60 through the 30 token-budgeted batches (every value a `_batch_texts` boundary; last tick = final batch's first index); the tiny second file restarts to `(1, 1)` at `files_done/total == 2/2`; overview/summaries ticks + terminal are 0/0 while file counts keep-final (2/2). Test 2 — the page's 2 s poll renders `Syncing… chunkprog/big.md` + `(1/2)` + visible `· chunk n/61`; title + `#sync-result` carry the untruncated concatenation (atomic in-browser check); after the file's span the chunk part is hidden + empty while the label keeps rendering; settle = `Synced HH:MM`, parts hidden, `2 added`. Fixture: `chunkprog/` local source, ~63.5 KB `big.md` (61 chunks) + one-chunk `small.md`; `SLOW_DELAY_S=1.5` (each embed batch outlives the 100 ms cadence ~15×, the ~45 s import span the 2 s UI poll ~20×). +- **This attempt:** prior rejection was process-only ("no clean final report"); code was already in the working tree with the two fixes intact (Playwright `arg=` keyword-only; ruff line split). No code changes were needed; all gates re-verified fresh. +- **Results (all green):** `uv run ruff check .` → clean · `uv run pyright` → 0 errors · `uv run pytest --cov=app --cov-report=term-missing` → 3040 passed, `app/` coverage **99%** · `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` → 2 passed (126 s, DB up) +- **Docs:** README.md verified consistent (phase-129 label paragraph matches code); AGENTS.md unchanged, references verified — no doc impact this task. +- **Next pending:** phase 129 is complete (task 05 was its last); next is `.agents/phases/todo/130_chat_answer_clusters`. diff --git a/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a2.validate b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a2.validate new file mode 100644 index 0000000..0f8a218 --- /dev/null +++ b/.agents/reports/129_sync_chunk_progress/129_sync_chunk_progress__05_e2e_chunk_progress.a2.validate @@ -0,0 +1,114 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 18%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 33%] +........................................................................ [ 35%] +........................................................................ [ 37%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 52%] +........................................................................ [ 54%] +........................................................................ [ 56%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 66%] +........................................................................ [ 68%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 75%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 85%] +........................................................................ [ 87%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 97%] +........................................................................ [ 99%] +................ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 262 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 241 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 156 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 256 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 132 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 374 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 38 0 100% +app/rag/importer.py 571 9 98% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 81 0 100% +app/rag/prompts.py 144 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 49 0 100% +app/schemas.py 400 0 100% +-------------------------------------------------- +TOTAL 5152 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/README.md b/README.md index 5781b8b..5d6c58c 100644 --- a/README.md +++ b/README.md @@ -255,7 +255,10 @@ BOR_SOURCES_DIR=~/bor-sources # default; each repo lands in // Upload a `.tar`, `.tar.gz`, `.tgz`, or `.zip` archive to make it a source. The form on the Git sources page — and the `POST /api/git-sources/upload` -route behind it — accepts archives and unpacks + scans them immediately. +route behind it — accepts archives and unpacks + registers them (no scan — +the upload indexes nothing; the **Sync sources** button on the RAG page +performs the scan, which leaves room to edit the new source's ignore list +first). - Re-uploading the same filename replaces the source **in place** (no second folder, no duplicate row). @@ -293,8 +296,13 @@ whole git-source refresh in one click: any missing rows (`only_missing`). - States: the button shows **Syncing…** with a spinner while polling - `GET /api/sync/status` every 2 s. No client-side timeout — a clone + - embed can legitimately take minutes. + `GET /api/sync/status` every 2 s. During the import phase the label + carries the live file, its `(n/m)` file position, and — while a file's + embed batch is in flight — the pinned `· chunk n/m` embed position + (batch granularity: the first chunk of the in-flight batch over the + file's total content chunks); the full untruncated label also rides + the button title and the aria-live result line. No client-side + timeout — a clone + embed can legitimately take minutes. - One sync at a time: a second trigger while a run is in flight gets a `409`. @@ -869,8 +877,9 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`. ``` - **KB offline banner in the chat** — Postgres isn't running: `podman compose up -d db`. -- **Stuck "Thinking…"** — the LLM is slow or down; a 120s client timeout - turns it into an error banner automatically. +- **Stuck "Thinking…"** — the LLM is slow or down; the client's 300 s + pre-token guard (`TURN_TIMEOUT_MS`) turns a hung stream into the error + state automatically. --- diff --git a/app/api/sync.py b/app/api/sync.py index d767a26..6b5153b 100644 --- a/app/api/sync.py +++ b/app/api/sync.py @@ -112,6 +112,21 @@ Terminal states (success AND failed) clear ``phase`` + ``current_summary`` but keep the run's final ``summaries_done`` / ``summaries_total`` (the phase-64 keep-final-counts convention). +Phase 129 (task 03) adds the in-flight file's EMBED-chunk position — +``current_chunk`` (the 1-based first chunk of the in-flight +token-budgeted batch) + ``chunks_total`` (the file's total content +chunks), batch granularity via the :meth:`app.rag.llm.LLMClient.embed` +``on_progress`` hook (tasks 01/02). They are 0/0 before the first +batch (the prelude), during the ``"overview"`` / ``"summaries"`` +phases, and in terminal states — the import phase clearing them at +EVERY point it ends (the ``phase = "overview"`` and +``phase = "summaries"`` transitions and both terminals, the same spots +``current_file`` clears). Unlike the phase-64 FILE counts, the chunk +position is a SUB-position of a file that's done — the keep-final- +counts convention does NOT extend to it: the file counts survive the +terminal for the result line's context, the chunk position clears +with the file. + The ``failed`` state's ``error`` string is masked by the shared sanitizer — the ``user:pass@`` masker now lives in :mod:`app.core.errors` (imported here under the private name ``_sanitize_error``). @@ -178,6 +193,19 @@ class SyncStatus: Terminal states clear ``phase`` + ``current_summary`` but keep the run's final summary counts (the phase-64 keep-final-counts convention). + + Phase 129 (task 03) chunk-position fields: ``current_chunk`` is + the 1-based first chunk of the in-flight token-budgeted batch and + ``chunks_total`` the in-flight file's total content chunks (batch + granularity, the ``LLMClient.embed`` hook — the module docstring's + contract). They are 0/0 before the first batch, during the + ``"overview"`` / ``"summaries"`` phases, and in terminal states — + the import phase clears them at every point it ends (the + overview/summaries transitions + both terminals, where + ``current_file`` also clears). The phase-64 keep-final-counts + convention covers the FILE counts only — the chunk position is a + sub-position of a file that's done, so it clears in the terminal + while ``files_done`` / ``files_total`` keep their final values. """ state: Literal["idle", "running", "success", "failed"] = "idle" @@ -197,6 +225,13 @@ class SyncStatus: current_summary: str | None = None summaries_done: int = 0 summaries_total: int = 0 + # Phase 129 (task 03): the current file's EMBED chunk position — + # the first chunk of the in-flight token-budgeted batch and the + # file's total content chunks (the LLMClient.embed hook, batch + # granularity). 0/0 before the first batch, during the + # prelude/overview/summaries phases, and in terminal states. + current_chunk: int = 0 + chunks_total: int = 0 _status = SyncStatus() @@ -211,11 +246,17 @@ def sync_status() -> dict[str, Any]: ``current_file`` (phase 64) is the ``source/relative/path`` the import is processing right now — null during the clone/pull phase and in terminal states; ``files_done`` / ``files_total`` carry the - hook's position (0/0 idle). ``phase`` (phase 98) is the pipeline - stage — null in the prelude and terminal states; while in - ``"summaries"``, ``current_summary`` + ``summaries_done`` / - ``summaries_total`` carry the generator's position (the terminal - keeps the final summary counts). + hook's position (0/0 idle) and keep their final values in the + terminal. ``current_chunk`` / ``chunks_total`` (phase 129) are the + in-flight file's embed-chunk position (batch granularity — the + first chunk of the in-flight batch over the file's total content + chunks): 0/0 before the first batch, in the prelude/overview/ + summaries phases, and in terminal states (they clear at every + point the import phase ends; the FILE counts do not). ``phase`` + (phase 98) is the pipeline stage — null in the prelude and + terminal states; while in ``"summaries"``, ``current_summary`` + + ``summaries_done`` / ``summaries_total`` carry the generator's + position (the terminal keeps the final summary counts). """ return { "state": _status.state, @@ -226,6 +267,12 @@ def sync_status() -> dict[str, Any]: "current_file": _status.current_file, "files_done": _status.files_done, "files_total": _status.files_total, + # Phase 129 (task 03): the in-flight file's embed-chunk + # position — the dataclass defaults make idle 0/0, and the + # runner clears it at every point the import phase ends + # (overview/summaries transitions + both terminals). + "current_chunk": _status.current_chunk, + "chunks_total": _status.chunks_total, # Phase 98 (task 01): the phase machine — null/0/0 idle (the # dataclass defaults) and in terminal states (which clear # phase + current_summary but keep the final summary counts). @@ -278,6 +325,10 @@ async def _run_sync() -> None: _status.current_summary = None _status.summaries_done = 0 _status.summaries_total = 0 + # Phase 129 (task 03): the chunk-position fields reset with the + # run — 0/0 until the import's first embed batch reports. + _status.current_chunk = 0 + _status.chunks_total = 0 try: settings = get_settings() # Step 1 (phase 41): fail fast — verify both models the sync @@ -359,18 +410,35 @@ async def _run_sync() -> None: _status.files_done = done _status.files_total = total + # Phase 129 (task 03): the in-flight file's embed-chunk + # position — the LLMClient.embed hook (task 01) fires before + # each token-budgeted batch with (first_chunk_1based, total); + # the file's IDENTITY is the sibling ``progress`` hook's (the + # two hooks are paired — a progress call names the file, and + # every chunk position until the next one belongs to it). + def _chunk_hook(chunk: int, total: int) -> None: + _status.current_chunk = chunk + _status.chunks_total = total + # Phase 98 (task 01): the phase machine — the import phase # starts NOW (the model-check + clone/pull prelude above # reported ``phase: null``). _status.phase = "import" summary: ImportSummary = await import_sources( - sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root, + sources, llm, prune=True, progress=_hook, on_chunk_progress=_chunk_hook, + ignore_by_root=ignore_by_root, include_hidden_by_root=include_hidden_by_root, doc_dates_by_root=doc_dates_by_root, ) overview = False if summary.added + summary.updated > 0: _status.phase = "overview" # phase 98 (task 01) + # Phase 129 (task 03): the import phase ends here — the + # chunk position (a sub-position of a done file) clears + # with the phase change; the file counts keep their final + # values (the phase-64 convention covers files only). + _status.current_chunk = 0 + _status.chunks_total = 0 overview = await regenerate_overview(llm) # Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03): # the folder summaries — the drill-down ls's per-level @@ -419,6 +487,11 @@ async def _run_sync() -> None: # position through the (long) span (its total is the # affected count). _status.phase = "summaries" + # Phase 129 (task 03): the import phase ends here too + # (the changed-KB branch skips no phase) — same + # chunk-position clear as the overview transition. + _status.current_chunk = 0 + _status.chunks_total = 0 folder_stats = await generate_folder_summaries( fs_db, llm, changed_docs=summary.changed_paths, on_progress=_summary_hook, @@ -435,6 +508,11 @@ async def _run_sync() -> None: # phase + hook (the fill's total is the missing # count — the hook's loop-start ``total``). _status.phase = "summaries" + # Phase 129 (task 03): the import phase ends here + # too (the unchanged-walk gap-fill branch) — same + # chunk-position clear. + _status.current_chunk = 0 + _status.chunks_total = 0 folder_stats = await generate_folder_summaries( fs_db, llm, only_missing=True, on_progress=_summary_hook ) @@ -475,6 +553,12 @@ async def _run_sync() -> None: _status.current_file = None # phase 64: keep the final counts _status.phase = None # phase 98: keep the final summary counts _status.current_summary = None + # Phase 129 (task 03): the chunk position is a sub-position of + # a file that's done — it clears in the terminal (unlike the + # FILE counts, which keep their final values for the result + # line's context). + _status.current_chunk = 0 + _status.chunks_total = 0 _status.detail = { "files": summary.files, "added": summary.added, @@ -498,3 +582,9 @@ async def _run_sync() -> None: _status.current_file = None # phase 64: keep the final counts _status.phase = None # phase 98: keep the final summary counts _status.current_summary = None + # Phase 129 (task 03): the chunk position clears in the + # failed terminal too (the convention split — the FILE counts + # keep their final values next to the error, the sub-position + # of a file that's done/abandoned does not). + _status.current_chunk = 0 + _status.chunks_total = 0 diff --git a/app/rag/importer.py b/app/rag/importer.py index 4c10d51..207a3e3 100644 --- a/app/rag/importer.py +++ b/app/rag/importer.py @@ -144,7 +144,10 @@ is NOT cleared on success — it is the retrieval display's record also, task 05). ``import_sources`` accepts an optional per-file ``progress`` callback -(phase 64, task 01) reporting the file being processed right now. +(phase 64, task 01) reporting the file being processed right now, and +an optional ``on_chunk_progress`` callback (phase 129, task 02) +reporting the in-flight file's embedding-chunk position at batch +granularity (see the function docstring). """ from __future__ import annotations @@ -198,7 +201,14 @@ class Embedder(Protocol): settings: Settings embed_batches: int - async def embed(self, texts: list[str]) -> list[list[float]]: ... + async def embed( + self, + texts: list[str], + *, + # Phase 129 (task 01): the per-batch position hook — ``None`` + # (every pre-phase caller) is a zero-cost guarded no-op. + on_progress: Callable[[int, int], None] | None = None, + ) -> list[list[float]]: ... async def chat(self, messages: list[dict[str, Any]], model: str | None = None) -> str: ... # ^ the one-shot completion the summarizer uses — the ``lite`` model @@ -419,6 +429,7 @@ async def import_sources( limit: int | None = None, session: Session | None = None, progress: Callable[[str, str, int, int], None] | None = None, + on_chunk_progress: Callable[[int, int], None] | None = None, ignore_by_root: dict[str, list[str]] | None = None, include_hidden_by_root: dict[str, bool] | None = None, doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, @@ -442,6 +453,32 @@ async def import_sources( exceeds the limit, but ``total`` stays the full pre-walk count (an incomplete walk must not misreport the denominator). + ``on_chunk_progress`` (phase 129, task 02) is an optional + chunk-position hook for the file currently being indexed: while that + file's content chunks embed, it is called — through + :meth:`app.rag.llm.LLMClient.embed`'s ``on_progress`` — before EACH + embed batch with ``(first_chunk_1based, total_chunks)``: the 1-based + position of the first chunk of the in-flight batch and the file's + total content-chunk count (batch granularity, the phase-129 A6 + assumption — the transport embeds token-budgeted batches per HTTP + call). It carries the in-flight FILE's chunk position only, never + the file identity — the per-file ``progress`` hook owns that: the + caller pairs them (a ``progress`` call names the file, and every + chunk position reported until the next ``progress`` call belongs to + it; the position climbs through the file's batches and restarts at + 1 for the next file). Same "only when provided" rule as + ``progress``: ``None`` (every existing caller — the CLI, the chat + path) is a zero-cost guarded no-op, and the run is byte-identical + to a hook-less run. Only the content-chunk embeds report (the + shared :func:`_replace_content_chunks` — text, image, and PDF + branches; the image branch's one description chunk reports + ``1/1``, which IS its only chunk): the document's summary-chunk + embed (:func:`_store_summary`) NEVER receives the hook, so the + counter counts content chunks only (no ``1/1`` noise after the + real progress ends), and the post-index enrichment pass / the + PDF's embedded-image docs (separate documents, not the walked + file) do not report either. + ``ignore_by_root`` (phase 89, A1/A2) maps ``str(root)`` — the root path string exactly as passed in *sources* — to that source's RAW ignore-path lines (the importer normalizes them via ``_ignore_for_root``, the single @@ -574,6 +611,7 @@ async def import_sources( await _index_file( session, source=source, rel=rel, full_path=path, llm=llm, summary=summary, raw_date=dates_map.get(rel), + on_chunk=on_chunk_progress, ) except EmbeddingError as e: # A pathological file (e.g. content the embedding endpoint @@ -608,7 +646,13 @@ async def import_sources( async def _replace_content_chunks( - session: Session, *, doc: Document, llm: Embedder, content: str, rel: str + session: Session, + *, + doc: Document, + llm: Embedder, + content: str, + rel: str, + on_chunk: Callable[[int, int], None] | None = None, ) -> int: """Phase 1+2 on *content*: chunk it (format-aware, PLAN §5), replace the doc's content chunk rows (embeddings NULL), and embed the new @@ -622,6 +666,20 @@ async def _replace_content_chunks( verbatim by the text path (:func:`_index_file`), the image path (task 03), and the task-04 enrichment pass — one copy, three call sites. + + ``on_chunk`` (phase 129, task 02) is the in-flight file's + chunk-position hook, forwarded as ``on_progress`` to the content + embed: it fires before each top-level embed batch with + ``(first_chunk_1based, total_chunks)``. The token-cap re-chunk retry + (the ``while True`` above that halves *target*) re-embeds the NEW + (smaller-target) chunk list, and the hook reports against the NEW + total automatically — no special case: the announced total is + always the chunk list actually in flight. The empty-content + ``doc.chunks = []`` break never embeds, so the hook never fires for + it. ``None`` (the default; every pre-phase call site — the CLI, the + phase-125 enrichment pass, and the PDF's embedded-image docs, which + index separate documents rather than the walked file) keeps the call + byte-identical. """ settings = llm.settings target = max(400, settings.chunk_target_chars) @@ -635,7 +693,12 @@ async def _replace_content_chunks( if not doc.chunks: break try: - vectors = await llm.embed([c.content for c in doc.chunks]) + # Phase 129 (task 02): the in-flight file's chunk-position + # hook (``on_progress`` — the task-01 contract; ``None`` is a + # zero-cost no-op). + vectors = await llm.embed( + [c.content for c in doc.chunks], on_progress=on_chunk + ) for row, vec in zip(doc.chunks, vectors, strict=True): row.embedding = vec break @@ -660,6 +723,7 @@ async def _index_file( llm: Embedder, summary: ImportSummary, raw_date: datetime | None = None, + on_chunk: Callable[[int, int], None] | None = None, ) -> None: """Upsert one file: doc row + chunk rows + embeddings, one transaction. @@ -668,6 +732,15 @@ async def _index_file( map, or ``None`` (every non-git case): the file's mtime is read here, once, and becomes the source date (the D2 fallback). + ``on_chunk`` (phase 129, task 02) is the in-flight file's + chunk-position hook (see :func:`import_sources`) — passed through to + ALL THREE branches' content embeds: the text path's + :func:`_replace_content_chunks` directly, the image branch via + :func:`_index_image_file` (its one description chunk reports + ``1/1`` — it IS the file's only chunk), and the PDF branch via + :func:`_index_pdf_file` (its content embed is the primary target). + ``None`` (the default) keeps every branch byte-identical. + Image files (phase 122, toggle on) delegate to :func:`_index_image_file` — the binary path (bytes digest, persistent copy, ``content`` = the vision description) — BEFORE @@ -683,7 +756,7 @@ async def _index_file( if match_extension(full_path, image_set) is not None: return await _index_image_file( session, source=source, rel=rel, full_path=full_path, llm=llm, - summary=summary, raw_date=raw_date, + summary=summary, raw_date=raw_date, on_chunk=on_chunk, ) # Phase 126: PDF files take the PDF index path — AFTER the image # branch, BEFORE any text read (a PDF is binary; ``extract_pdf_text`` @@ -691,7 +764,7 @@ async def _index_file( if full_path.suffix.lower() == ".pdf": return await _index_pdf_file( session, source=source, rel=rel, full_path=full_path, llm=llm, - summary=summary, raw_date=raw_date, + summary=summary, raw_date=raw_date, on_chunk=on_chunk, ) content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "") digest = hashlib.sha256(content.encode("utf-8")).hexdigest() @@ -793,7 +866,7 @@ async def _index_file( # lives in the shared helper — the global policy stays intact for the # rest of the KB). chunks_n = await _replace_content_chunks( - session, doc=doc, llm=llm, content=content, rel=rel + session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk ) session.commit() @@ -952,6 +1025,7 @@ async def _index_pdf_file( llm: Embedder, summary: ImportSummary, raw_date: datetime | None = None, + on_chunk: Callable[[int, int], None] | None = None, ) -> None: """The phase-126 PDF branch of :func:`_index_file` — a PDF is indexed from its EXTRACTED TEXT (LOCKED A1/A2/A4), one transaction @@ -985,6 +1059,14 @@ async def _index_pdf_file( ``raw_date`` follows the text path: the caller's git last-commit date or ``None`` (the mtime fallback, read before the unchanged early-return — the unchanged path refreshes from the same source). + + ``on_chunk`` (phase 129, task 02) is the in-flight file's + chunk-position hook: it is forwarded to the PDF's CONTENT embed + (the :func:`_replace_content_chunks` call — the primary target), + but NOT to the embedded-image pass (those image docs are separate + documents — their ``1/1`` positions would be noise after the + file's real progress ends, the A6 exclusion). ``None`` keeps the + branch byte-identical. """ # Extract the text (fail-soft: an unopenable PDF yields "" — the # document may still carry embedded images worth indexing). @@ -1066,9 +1148,11 @@ async def _index_pdf_file( # Phase 1+2 — the shared chunk/replace/embed pipeline (the page- # aware chunker dispatches on the ``.pdf`` suffix; empty content # clears the chunk rows and returns 0), committed atomically with - # the document. + # the document. Phase 129 (task 02): the in-flight file's + # chunk-position hook rides the content embed (the embedded-image + # pass does not receive it — separate documents, the A6 exclusion). chunks_n = await _replace_content_chunks( - session, doc=doc, llm=llm, content=content, rel=rel + session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk ) session.commit() @@ -1815,6 +1899,7 @@ async def _index_image_file( llm: Embedder, summary: ImportSummary, raw_date: datetime | None = None, + on_chunk: Callable[[int, int], None] | None = None, ) -> None: """The phase-122 image branch of :func:`_index_file` — a standalone image is indexed from its BYTES, never its text: @@ -1855,6 +1940,13 @@ async def _index_image_file( pre-phase). The UNCHANGED path never re-scans (the stored context is part of the row's committed content; a changed source state re-indexes the file through this path). + + ``on_chunk`` (phase 129, task 02) is the in-flight file's + chunk-position hook: it is forwarded to the description's content + embed (:func:`_replace_content_chunks`). A description (enriched or + not) chunks to ONE chunk, so the hook reports ``1/1`` — acceptable, + it IS the file's only chunk. ``None`` keeps the branch + byte-identical. """ settings = llm.settings data = full_path.read_bytes() @@ -1996,8 +2088,10 @@ async def _index_image_file( # the chunk rows (embeddings NULL), embed, commit — one transaction # per file; the endpoint token-cap retry lives in the shared helper, # and a description is short, so it never fires in practice). + # Phase 129 (task 02): the in-flight file's chunk-position hook + # rides this embed (one description chunk → ``1/1``). chunks_n = await _replace_content_chunks( - session, doc=doc, llm=llm, content=content, rel=rel + session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk ) session.commit() diff --git a/app/rag/llm.py b/app/rag/llm.py index 2b3dc3e..ab5b205 100644 --- a/app/rag/llm.py +++ b/app/rag/llm.py @@ -22,7 +22,7 @@ from __future__ import annotations import asyncio import json import logging -from collections.abc import AsyncGenerator +from collections.abc import AsyncGenerator, Callable from dataclasses import dataclass from typing import TYPE_CHECKING, Any, Literal, cast @@ -343,12 +343,43 @@ class LLMClient: batches.append(cur) return batches - async def embed(self, texts: list[str]) -> list[list[float]]: - """Embed *texts* in token-budgeted batches (order always kept).""" + async def embed( + self, + texts: list[str], + *, + on_progress: Callable[[int, int], None] | None = None, + ) -> list[list[float]]: + """Embed *texts* in token-budgeted batches (order always kept). + + Progress hook (phase 129, task 01): when *on_progress* is given, + it is called once per TOP-LEVEL batch, BEFORE the batch's request + (a failing batch still announces its start), with + ``(first_chunk_1based, total_chunks)`` — the 1-based index of the + first text of the in-flight batch and the total number of texts + in this call — so a caller can render ``chunk n/m`` (the importer + passes this through for the sync label; PLAN §7.4 "never stale"). + + ``len(out) + 1`` is that first index: each batch's vectors extend + the accumulator 1:1, so ``len(out)`` is exactly the number of + chunks embedded by earlier batches. The :meth:`_embed_batch` + halving backstop does NOT sub-report: a halved batch leaves the + announced position approximate (the hook already fired for the + original batch's start) — documented; the backstop is rare by + construction (phase 114's ``_TooLarge`` handling). ``None`` (the + default — every pre-phase caller: the chat retrieval path, + :meth:`embed_one`, the summarizer, the importer) is a zero-cost + guarded no-op: call sequence and return value are + byte-identical to the pre-hook client. + """ if not texts: return [] out: list[list[float]] = [] for chunk in self._batch_texts(texts): + # Phase 129 (task 01): announce THIS batch's start before the + # request — the hook precedes the try, so even a failing batch + # reports its position. + if on_progress is not None: + on_progress(len(out) + 1, len(texts)) try: vecs = await self._embed_batch(chunk) except EmbeddingError: diff --git a/frontend/assets/sources.js b/frontend/assets/sources.js index d4e04b9..ddc3104 100644 --- a/frontend/assets/sources.js +++ b/frontend/assets/sources.js @@ -629,7 +629,10 @@ export async function mount(root) { * Elements: #sync-btn (the button), #sync-label (the text — * Phase 128 (task 01): the PARTS — .sync-label-main, the * shrinks-to-fit prefix + file/folder, + .sync-label-count, the - * pinned "(n/m)" position (hidden when absent)), #sync-icon (the + * pinned "(n/m)" position (hidden when absent); phase 129 (task + * 04): + .sync-label-chunk, the pinned "· chunk n/m" embed-batch + * position of the in-flight file (hidden in every state but the + * import phase after the file's first batch)), #sync-icon (the * spinner icon), #sync-result (aria-live result line), * #sync-error-banner / #sync-error-text (error banner). */ @@ -637,6 +640,7 @@ export async function mount(root) { const syncLabel = syncBtn ? syncBtn.querySelector(".sync-label") : null; const syncLabelMain = syncLabel ? syncLabel.querySelector(".sync-label-main") : null; const syncLabelCount = syncLabel ? syncLabel.querySelector(".sync-label-count") : null; + const syncLabelChunk = syncLabel ? syncLabel.querySelector(".sync-label-chunk") : null; const syncIcon = syncBtn ? syncBtn.querySelector(".sync-icon") : null; const syncResult = root.querySelector("#sync-result"); const syncErrorBanner = root.querySelector("#sync-error-banner"); @@ -665,22 +669,27 @@ export async function mount(root) { return `${pad(d.getHours())}:${pad(d.getMinutes())}`; } - /* Phase 128 (task 01): the label PARTS writer — the SINGLE point - * that writes #sync-label. mainText lands in the MAIN part - * (.sync-label-main — the shrinks-to-fit prefix + file/folder, the - * part CSS ellipsizes); countText lands in the COUNT part - * (.sync-label-count — the pinned "(n/m)" position, NEVER clipped): - * present → its textContent + hidden=false, absent → emptied + - * hidden=true (the count span is hidden whenever there is no - * "(n/m)" to show). Then the UNTRUNCATED full label — - * mainText + (countText ? " " + countText : "") — rides the button - * title + #sync-result (the aria-live announcer) exactly as before - * (A4 — the screen reader hears the whole thing; the visible span - * carries main + count, the space between them is the CSS gap). - * textContent only — the house rule, no HTML strings. Every label - * state goes through here (the running builders, the idle / settle - * labels, the 202/409 click, the re-attach). */ - function setSyncLabel(mainText, countText = "") { + /* Phase 128 (task 01) + Phase 129 (task 04): the label PARTS + * writer — the SINGLE point that writes #sync-label. mainText lands + * in the MAIN part (.sync-label-main — the shrinks-to-fit prefix + + * file/folder, the part CSS ellipsizes); countText lands in the + * COUNT part (.sync-label-count — the pinned "(n/m)" position, + * NEVER clipped); chunkText (phase 129) lands in the CHUNK part + * (.sync-label-chunk — the pinned "· chunk n/m" embed-batch + * position of the in-flight file). The COUNT and CHUNK parts share + * one contract: present → its textContent + hidden=false, absent → + * emptied + hidden=true (each span is hidden whenever there is + * nothing of its kind to show). Then the UNTRUNCATED full label — + * mainText + (countText ? " " + countText : "") + (chunkText ? + * " " + chunkText : "") — rides the button title + #sync-result + * (the aria-live announcer) exactly as before (A4 — the screen + * reader hears the whole "Syncing… (2/3) · chunk 14/38"; + * the visible spans carry main + count + chunk, the spaces between + * them are the CSS gap). textContent only — the house rule, no + * HTML strings. Every label state goes through here (the running + * builders, the idle / settle labels, the 202/409 click, the + * re-attach). */ + function setSyncLabel(mainText, countText = "", chunkText = "") { if (syncLabelMain) syncLabelMain.textContent = mainText; if (syncLabelCount) { if (countText) { @@ -691,7 +700,19 @@ export async function mount(root) { syncLabelCount.hidden = true; } } - const full = countText ? `${mainText} ${countText}` : mainText; + if (syncLabelChunk) { + if (chunkText) { + syncLabelChunk.textContent = chunkText; + syncLabelChunk.hidden = false; + } else { + syncLabelChunk.textContent = ""; + syncLabelChunk.hidden = true; + } + } + const full = + mainText + + (countText ? ` ${countText}` : "") + + (chunkText ? ` ${chunkText}` : ""); if (syncBtn) syncBtn.title = full; // A4: the untruncated label rides the title, always if (syncResult) syncResult.textContent = full; // A4: the announcer hears the whole thing } @@ -713,7 +734,11 @@ export async function mount(root) { * post-import phases (overview / summaries) take their own label * via fmtSyncPhaseLabel (below) — this builder keeps that * prelude/import fall-through byte-identical and serves the UPLOAD - * job unchanged (its status has no phase — D2). */ + * job unchanged (its status has no phase — D2). Phase 129 (task + * 04): this builder still returns NO chunk part — the pinned + * "· chunk n/m" part is added by fmtSyncPhaseLabel's import + * fall-through (the only site that sees the status's chunk + * fields), so the upload label and the 202/409 click stay bare. */ function fmtSyncLabel(kind, currentFile, done, total) { const prefix = kind === "upload" ? "Importing" : "Syncing…"; const main = currentFile ? `${prefix} ${currentFile}` : prefix; @@ -729,26 +754,42 @@ export async function mount(root) { * " + count "(n/m)" — the folder part is omitted * while current_summary is null (the phase's first poll) and the * COUNT IS ALWAYS EMITTED in this phase (the hook always carries a - * total); anything else (the null prelude, "import") → the - * phase-64 fmtSyncLabel parts, byte-identical (same builder, same - * fields, same order). The UPLOAD job never passes a status (its - * endpoint has no phase — D2) and keeps the bare fmtSyncLabel - * parts. The untruncated full label rides the button title + - * #sync-result like every running label (A4 — setSyncLabel). */ + * total) and the CHUNK part stays empty (the summaries phase's own + * (n/m) is its position — the status's chunk fields are cleared at + * this transition); anything else (the null prelude, "import") → + * the phase-64 fmtSyncLabel parts, byte-identical (same builder, + * same fields, same order) + the phase-129 CHUNK part: "· chunk + * n/m" ONLY while chunks_total > 0 (a file's embed batch is in + * flight — n the 1-based first chunk of the batch, m the file's + * total content chunks, batch granularity), empty before the first + * batch (0/0) so the part hides. The UPLOAD job never passes a + * status (its endpoint has no phase — D2) and keeps the bare + * fmtSyncLabel parts. The untruncated full label rides the button + * title + #sync-result like every running label (A4 — + * setSyncLabel). */ function fmtSyncPhaseLabel(status) { if (status.phase === "overview") { - return { main: "Writing KB overview…", count: "" }; + return { main: "Writing KB overview…", count: "", chunk: "" }; } if (status.phase === "summaries") { const folder = status.current_summary ? ` ${status.current_summary}` : ""; return { main: `Summarizing folders…${folder}`, count: `(${status.summaries_done}/${status.summaries_total})`, + chunk: "", // this phase's own (n/m) is its position — no chunk part }; } // null phase (the prelude) or "import" — today's byte-identical - // sync label (the phase-64 contract, unchanged). - return fmtSyncLabel("sync", status.current_file, status.files_done, status.files_total); + // sync label (the phase-64 contract, unchanged) + the phase-129 + // chunk part (hidden before the in-flight file's first batch). + const { main, count } = fmtSyncLabel( + "sync", status.current_file, status.files_done, status.files_total + ); + const chunk = + status.chunks_total > 0 + ? `· chunk ${status.current_chunk}/${status.chunks_total}` + : ""; + return { main, count, chunk }; } function fmtSyncResult(detail) { @@ -780,19 +821,23 @@ export async function mount(root) { * writer (setSyncLabel) lands them — the main part (CSS-ellipsized), * the pinned count part (hidden when absent), and the FULL * untruncated label on the button title + #sync-result (A4 — set - * for EVERY running label; CSS ellipsizes the main part only). Same - * never-stale mechanics as before (disabled, aria-busy, spinning - * icon, no is-error). + * for EVERY running label; CSS ellipsizes the main part only). + * Phase 129 (task 04): the third, pinned CHUNK part ("· chunk n/m" + * — the in-flight file's embed-batch position) is destructured the + * same way (absent from the bare builders — the upload job and the + * 202/409 click — so the "" default keeps the part hidden there). + * Same never-stale mechanics as before (disabled, aria-busy, + * spinning icon, no is-error). */ function enterSyncRunningState(kind, currentFile, done, total, status) { if (!syncBtn) return; syncBtn.disabled = true; syncBtn.setAttribute("aria-busy", "true"); - const { main, count } = + const { main, count, chunk = "" } = kind === "sync" && status ? fmtSyncPhaseLabel(status) : fmtSyncLabel(kind, currentFile, done, total); - setSyncLabel(main, count); // the parts + the A4 untruncated title / announcer + setSyncLabel(main, count, chunk); // the parts + the A4 untruncated title / announcer syncBtn.setAttribute("aria-label", "Sync sources"); syncBtn.classList.remove("is-error"); if (syncIcon) syncIcon.classList.add("is-spinning"); diff --git a/frontend/assets/styles.css b/frontend/assets/styles.css index 16cf747..cf49e5c 100644 --- a/frontend/assets/styles.css +++ b/frontend/assets/styles.css @@ -2064,10 +2064,14 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und (n/m)") — a long source/relative/path ellipsizes inside the pill; the full path lives in the button title + #sync-result (the aria-live announcer). Phase 128 (task 02): the label is a FLEX ROW - of two parts — .sync-label-main (the prefix + file/folder: min-width + of parts — .sync-label-main (the prefix + file/folder: min-width 0, the shrinkable item whose ellipsis engages) + .sync-label-count (the pinned "(n/m)": flex 0 0 auto, NEVER clipped — the - .sync-result mono language, a position reads as data). */ + .sync-result mono language, a position reads as data). Phase 129 + (task 04): a THIRD pinned part — .sync-label-chunk (the "· chunk + n/m" embed-batch position of the in-flight file, the same pinned + mono language; hidden in every state but the import phase after + the file's first batch). */ .sync-label { display: inline-flex; align-items: baseline; @@ -2086,6 +2090,11 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und white-space: nowrap; font-family: var(--mono); /* the .sync-result mono language (position reads as data) */ } +.sync-label-chunk { + flex: 0 0 auto; /* NEVER clipped — the · chunk n/m position is the point (phase 129) */ + white-space: nowrap; + font-family: var(--mono); /* the same pinned-part language as the count (position reads as data) */ +} /* Last-result announcer — soft ink, small mono */ .sync-result { display: block; @@ -5329,8 +5338,10 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und width; the History page-head already wraps the pill below its title block. The Sync label's min(16rem, 40vw) cap lifts and the label FILLS the pill (phase 128: flex 1 1 auto) — .sync-label-main - ellipsizes against the width left by the pinned .sync-label-count, - so the (n/m) stays on-screen at the right edge, never clipped; + ellipsizes against the width left by the pinned .sync-label-count + (+ the phase-129 .sync-label-chunk part while a file's embed batch + is in flight), so the position parts stay on-screen at the right + edge, never clipped; the refresh glyph joins its visible label. The aria-labels keep the accessible names. */ .page-head-row { flex-wrap: wrap; } diff --git a/frontend/index.html b/frontend/index.html index 8486772..90325ac 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -448,7 +448,7 @@

Knowledge base

@@ -460,9 +460,13 @@