Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 152_sync_progress_image_pass
Build and Push Containers / build-and-push-app (push) Successful in 3m34s
Details
Build and Push Containers / build-and-push-db (push) Successful in 23s
Details

Browse Source 
All criteria verified. The CLI path is hook-less (only `app/api/sync.py` passes the hook), and the byte-identity is pinned by `test_on_image_progress_none_run_is_byte_identical`.

## Final Report — Phase 152 final verification pass

- **Verified (no new code needed):** `SyncStatus.images_done/images_total` + reset/clear rules in `app/api/sync.py`; `on_image_progress` threaded through the importer's three PDF vision spans with the continuous per-file counter; the `_image_hook` closure; `sources.js` `· image n/m` label part (box + title + aria).
- **C1** status climbs 1→N + UI shows `· image n/m`: ✅ unit 3/15→15/15, integration 1/15→7/15, E2E 1/4→4/4 (title/aria ride the full string)
- **C2** never frozen: ✅ E2E asserts ≥3 distinct positions, starting at 1, non-decreasing, ≤ total, reaching it
- **C3** cleared at overview/summaries + all terminals; per-file reset via `_hook`; no stale position for image-less files: ✅ unit pins + E2E settle (image part gone in box/title/aria, endpoint 0/0)
- **C4** `/api/sync/status` additive-only: ✅ full-dict pin passes (all pre-existing keys unchanged + the two new 0/0 keys)
- **C5** hook-less caller byte-identical: ✅ CLI passes no hook; `test_on_image_progress_none_run_is_byte_identical` green
- **C6** `uv run pytest`: **3526 passed, exit 0** (95s); dedicated E2E in isolation: **2 passed** (95s); `uv run ruff check . && uv run pyright`: **clean**; coverage: **99% on `app/` (>90% gate)**
- Regression: all five neighbouring E2E suites green in isolation (progress_box 8, chunk_progress 2, label_mobile 4, pdf_support 6, image_documents 4)
- **C7** atomic commit: left to the harness (no git index/history touched per rules)
- **Docs:** README.md — sync-label paragraph already updated this phase, accurately; all `.env.example` vars documented; routes/commands verified — **no changes needed**. AGENTS.md — all references (PLAN, user_stories, test_smoke.py, .env.example, gpg config, .gitignore) verified — **no changes needed**
- **Defects found in prior work:** none (one prior env-leak defect was already fixed inside this phase's task 04, conftest)
- No live-infrastructure touched; no ntfy used. **Next pending phase:** none — `todo/` held only this phase
This commit is contained in:
ducoterra
2026-10-03 01:52:41 -04:00
parent c7cfcd26b5
commit 12fa69b7ea
39 changed files with 3971 additions and 133 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/152_sync_progress_image_pass/00_phase.md → .agents/phases/complete/152_sync_progress_image_pass/00_phase.md
View File
.agents/phases/todo/152_sync_progress_image_pass/01_status_image_fields.md → .agents/phases/complete/152_sync_progress_image_pass/01_status_image_fields.md
View File
.agents/phases/todo/152_sync_progress_image_pass/02_importer_image_hook.md → .agents/phases/complete/152_sync_progress_image_pass/02_importer_image_hook.md
View File
.agents/phases/todo/152_sync_progress_image_pass/03_label_and_wiring.md → .agents/phases/complete/152_sync_progress_image_pass/03_label_and_wiring.md
View File
.agents/phases/todo/152_sync_progress_image_pass/04_unit_tests.md → .agents/phases/complete/152_sync_progress_image_pass/04_unit_tests.md
View File
.agents/phases/todo/152_sync_progress_image_pass/05_e2e_image_progress.md → .agents/phases/complete/152_sync_progress_image_pass/05_e2e_image_progress.md
View File
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__00_phase.a1.err
View File
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__00_phase.a1.md
+16
View File
@@ -0,0 +1,16 @@
All criteria verified. The CLI path is hook-less (only `app/api/sync.py` passes the hook), and the byte-identity is pinned by `test_on_image_progress_none_run_is_byte_identical`.
## Final Report — Phase 152 final verification pass
- **Verified (no new code needed):** `SyncStatus.images_done/images_total` + reset/clear rules in `app/api/sync.py`; `on_image_progress` threaded through the importer's three PDF vision spans with the continuous per-file counter; the `_image_hook` closure; `sources.js` `· image n/m` label part (box + title + aria).
- **C1** status climbs 1→N + UI shows `· image n/m`: ✅ unit 3/15→15/15, integration 1/15→7/15, E2E 1/4→4/4 (title/aria ride the full string)
- **C2** never frozen: ✅ E2E asserts ≥3 distinct positions, starting at 1, non-decreasing, ≤ total, reaching it
- **C3** cleared at overview/summaries + all terminals; per-file reset via `_hook`; no stale position for image-less files: ✅ unit pins + E2E settle (image part gone in box/title/aria, endpoint 0/0)
- **C4** `/api/sync/status` additive-only: ✅ full-dict pin passes (all pre-existing keys unchanged + the two new 0/0 keys)
- **C5** hook-less caller byte-identical: ✅ CLI passes no hook; `test_on_image_progress_none_run_is_byte_identical` green
- **C6** `uv run pytest`: **3526 passed, exit 0** (95s); dedicated E2E in isolation: **2 passed** (95s); `uv run ruff check . && uv run pyright`: **clean**; coverage: **99% on `app/` (>90% gate)**
- Regression: all five neighbouring E2E suites green in isolation (progress_box 8, chunk_progress 2, label_mobile 4, pdf_support 6, image_documents 4)
- **C7** atomic commit: left to the harness (no git index/history touched per rules)
- **Docs:** README.md — sync-label paragraph already updated this phase, accurately; all `.env.example` vars documented; routes/commands verified — **no changes needed**. AGENTS.md — all references (PLAN, user_stories, test_smoke.py, .env.example, gpg config, .gitignore) verified — **no changes needed**
- **Defects found in prior work:** none (one prior env-leak defect was already fixed inside this phase's task 04, conftest)
- No live-infrastructure touched; no ntfy used. **Next pending phase:** none — `todo/` held only this phase
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__00_phase.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 98%]
...................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 764 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5719 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__01_status_image_fields.a1.err
View File
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__01_status_image_fields.a1.md
+19
View File
@@ -0,0 +1,19 @@
Task 01 is complete and everything is green. Final report:
**Implemented**
- `SyncStatus` gains `images_done`/`images_total` (0/0 defaults) with the phase-152 comment, a dataclass docstring paragraph, and a module-docstring paragraph after the phase-129 one; endpoint docstring extended (additive-only)
- Reset/clear discipline mirrors the chunk position: run-start reset, and `0` at all 7 `current_chunk` clear sites (overview transition, both summaries branches, success, cancelled, failed) — plus the per-file `_hook` closure now resets the image position to 0/0 (extended pairing rule: a `progress` call names the file and clears the image position)
- `GET /api/sync/status` payload gains `images_done`/`images_total` beside the chunk keys; every pre-existing key unchanged
- Tests: full idle-dict pins extended (unit + integration), new per-file-reset test driving the runner's own `_hook` (two-file fake, live 3/15 position cleared when file B is named), new transition+terminal clear test (overview/summaries/success), failed and cancelled terminal tests extended with live positions
**Results**
- `uv run pytest` → 3513 passed (full suite 100% green)
- `uv run pytest --cov=app --cov-report=term-missing` → 99% on `app/` (`app/api/sync.py` 100%)
- `uv run pytest tests/unit/test_sync_button.py -q` → 59 passed
- `uv run ruff check . && uv run pyright` → clean
**Docs:** no README/AGENTS.md impact — the change is an additive server-side API key; README documents the UI label (the `· image n/m` part lands in task 03), not the payload schema.
**Deviations:** none (ASSUMPTION A5 honored — no new `phase` value; importer/label wiring intentionally left to tasks 02/03).
**Next pending task:** `.agents/phases/todo/152_sync_progress_image_pass/02_importer_image_hook.md`
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__01_status_image_fields.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 201 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 748 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5700 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__02_importer_image_hook.a1.err
View File
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__02_importer_image_hook.a1.md
+16
View File
@@ -0,0 +1,16 @@
Task complete. Final report:
**Implemented** (phase 152, task 02 — importer's `on_image_progress` hook):
- `import_sources(..., on_image_progress=…)` threaded exactly like `on_chunk_progress`: → `_index_file(on_image=…)` → `_index_pdf_file(on_image=…)` → the three PDF vision spans; standalone image files get no sub-position (A4), counts only (A2)
- Each span fires the hook at the start of each image's slow step with `(base + i + 1, base + span_total)` — embedded pass total = post-dedupe/post-gate `len(images)` (repeat placement adds no position), pictorial = candidate count, fallback = rendered-page count; every call site `if on_image is not None:` (hook-less runs byte-identical)
- Continuous per-file counter: `_index_pdf_file`'s local `images_reported` — each span returns its reported count, the second span starts where the first ended; skipped spans report nothing
- Docstrings updated: module progress paragraph, `import_sources` (new paragraph + corrected the stale "embedded-image docs do not report either" sentence), `_index_file`, `_index_pdf_file`, and the three span docstrings
- 6 new unit tests in `tests/unit/test_importer.py` (PyMuPDF-built PDFs + `_PageVisionLLM`): embedded pass `(1,N)…(N,N)` + dedupe total + fast-path re-sync fires nothing; pictorial→embedded one counter; fallback→embedded continuity (`(1,2),(2,2),(3,3)`); text-only PDF never fires; toggle-off reports nothing; hook-less vs hooked run byte-identical (rows + all summary counters)
**Test / lint / coverage**: `uv run pytest --cov=app --cov-report=term-missing` → 3519 passed, 0 failed (exit 0), `app/` coverage **99%** (importer.py 99% — only 4 pre-existing lines missed); `uv run ruff check .` clean; `uv run pyright` 0 errors.
**Docs**: no doc impact (internal importer plumbing; the user-visible `· image n/m` label is task 03's scope).
**Notable decisions**: hook fires per image slot in the embedded pass (uniform with the other two spans, matches the task's `base + i + 1` formula and its placement in `_index_pdf_embedded_images`); span return shapes extended to `(content, reported)` tuples / `int` (minimal clean shape; both spans are importer-internal, callers updated in place).
**Next pending task**: `.agents/phases/todo/152_sync_progress_image_pass/03_label_and_wiring.md`
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__02_importer_image_hook.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
............................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 201 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 764 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5716 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__03_label_and_wiring.a1.err
View File
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__03_label_and_wiring.a1.md
+11
View File
@@ -0,0 +1,11 @@
All gates green. Final report:
**Task 03 — sync label wiring + `· image n/m` rendering: complete**
- `app/api/sync.py`: added the third paired closure `_image_hook(done, total)` beside `_hook`/`_chunk_hook` and passed `on_image_progress=_image_hook` into `import_sources` (established-voice pairing comment).
- `frontend/assets/sources.js`: `setSyncLabel` gains fifth param `imageText` appended last to the composed `full` (box + title + running aria-label, no new span — ASSUMPTION A5); `fmtSyncPhaseLabel` import fall-through renders `· image n/m` while `images_total > 0`, overview/summaries return `image: ""`; `enterSyncRunningState` destructures `image = ""` and passes it; all phase-128/129/149/150 label-part comments + `index.html` box comment updated to name the image part.
- Tests: source-level pins in `test_sync_button.py`/`test_frontend_sync_upload.py` (signature, `full` byte-for-byte, part present/absent, branches never read image fields, bare builder image-free) + new node-harness rows pinning the real path behaviorally: both parts at once (`… · chunk 125/126 · image 3/15`) and image-only (independent disappearance); all runner fakes accept the new kwarg; new integration test `test_image_position_climbs_between_polls_mid_run` (1/15 → 7/15 monotonic across two status polls via the runner's own closure, 0/0 at terminal, wiring pinned).
- Results: `uv run pytest --cov=app` → **3521 passed**, `app/api/sync.py` 100%, TOTAL **99%**; `uv run ruff check .` clean; `uv run pyright` 0 errors; sync-label E2Es green in isolation (`test_sync_chunk_progress.py` 2 passed, `test_sync_progress_box.py` 8 passed — pre-phase labels byte-identical).
- Docs: README.md sync-label section now documents `· image n/m`; no AGENTS.md impact.
- No deviations; no CSS/markup changes (position rides the composed string only, per the design).
- Next pending task: `04_unit_tests.md` (status-machine pins + importer hook pairing).
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__03_label_and_wiring.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 44%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
................................................................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 764 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5719 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__04_unit_tests.a1.err
View File
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__04_unit_tests.a1.md
+15
View File
@@ -0,0 +1,15 @@
All 5 new tests pass, full suite green. Work complete — final report:
## Phase 152, Task 04 — complete
**Implemented (unit + integration test pins, no app-code changes):**
- `tests/unit/test_sync_button.py`: mid-run pin via the runner's own `_image_hook` closure (`_GatedImport` now fires `image_pairs` — 3/15 → 15/15 climbing between ticks, three positions independent); per-file reset, overview/summaries transition clears, and failed + cancelled terminals all moved onto closure-driven pins (phase-64/129 idiom); new source-level pin of `fmtSyncPhaseLabel`'s image part (guard `images_total > 0`, exact template, `image: ""` in overview/summaries, composed `full` string, `· chunk 125/126 · image 3/15` example)
- `tests/unit/test_importer.py`: hook fires **before** each describe call (interleaving recorded in a fake LLM — `hook(1,N), vision(1), …`); blank-gate interaction (near-white image excluded from `total` by default, counted with `pdf_image_skip_blank=False`)
- `tests/integration/test_sync_api.py`: full sync over a fixture KB with one image-bearing PDF (real `import_sources` + gated mock LLM) — observed `images_done` sequence monotonic, never above `images_total`, ends at it; terminal clears; KB carries PDF + N image docs
- Defect fix (pre-existing failure): the owner's local `.env` `BOR_APP_NAME=Brain of Patrick` broke `test_config_returns_default_app_metadata` — added the `BOR_APP_NAME` default pin to `tests/conftest.py` (established leak-class idiom)
**Gates (all green):** `uv run pytest -q` exit 0 · `uv run pytest --cov=app --cov-report=term-missing` → TOTAL 99% (>90%; `app/api/sync.py` 100%, `app/rag/importer.py` 99%) · `uv run ruff check .` clean · `uv run pyright` 0 errors
**Docs:** no doc impact (README's `· image n/m` paragraph landed in task 03; AGENTS.md is protocol-only)
**Next pending task:** `05_e2e_image_progress.md` (dedicated Playwright suite)
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__04_unit_tests.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 98%]
...................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 764 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5719 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__04_unit_tests.a2.err
View File
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__04_unit_tests.a2.md
+1
View File
@@ -0,0 +1 @@
Now let me look at the existing test idioms and what's already covered.
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__04_unit_tests.a2.validate
+185
View File
@@ -0,0 +1,185 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 44%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
.......................................................F................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
................................................................. [100%]
=================================== FAILURES ===================================
___________________ test_config_returns_default_app_metadata ___________________
client = <starlette.testclient.TestClient object at 0x7f755c272890>
db = <sqlalchemy.orm.session.Session object at 0x7f755c2b7c50>
def test_config_returns_default_app_metadata(client, db: Session) -> None:
"""GET /api/config is public (anonymous) and returns exactly six
keys — the phase-39 app metadata, the phase-59 docs flag (inert
false while BOR_DOCS_REPO is empty — the "Save as doc" gating),
the phase-122 images flag (default false in the test env —
LOCKED A3: off by default), and the phase-62 UI customization
strings (composer placeholder, footer line). Phase 91: with an
empty ui_settings table the effective strings are the env defaults
(B1 — DB-over-env, the row absent here); the retired CSS-file
theming's ``theme`` key is gone (task 03 — the six keys are the
entire contract)."""
_clear_ui_settings(db)
r = client.get("/api/config")
assert r.status_code == 200
body = r.json()
assert set(body) == {
"app_name", "version", "docs_repo_configured",
"images", "input_placeholder", "footer_text",
}
> assert body["app_name"] == "Brain of Reese"
E AssertionError: assert 'Brain of Patrick' == 'Brain of Reese'
E
E - Brain of Reese
E + Brain of Patrick
tests/integration/test_api.py:69: AssertionError
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 764 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5719 17 99%
=========================== short test summary info ============================
FAILED tests/integration/test_api.py::test_config_returns_default_app_metadata
tests FAILED: unit + integration suite did not pass
All checks passed!
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
Traceback (most recent call last):
File "/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/bin/pyright", line 10, in <module>
sys.exit(entrypoint())
~~~~~~~~~~^^
File "/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/pyright/cli.py", line 31, in entrypoint
sys.exit(main(sys.argv[1:]))
~~~~^^^^^^^^^^^^^^
File "/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/pyright/cli.py", line 18, in main
return run(*args, **kwargs).returncode
~~~^^^^^^^^^^^^^^^^^
File "/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/pyright/cli.py", line 27, in run
return node.run('node', str(script), *args, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/pyright/node.py", line 144, in run
subprocess.run(node_args, **kwargs),
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "/var/home/ducoterra/.local/share/uv/python/cpython-3.13.13-linux-x86_64-gnu/lib/python3.13/subprocess.py", line 556, in run
stdout, stderr = process.communicate(input, timeout=timeout)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/home/ducoterra/.local/share/uv/python/cpython-3.13.13-linux-x86_64-gnu/lib/python3.13/subprocess.py", line 1214, in communicate
self.wait()
~~~~~~~~~^^
File "/var/home/ducoterra/.local/share/uv/python/cpython-3.13.13-linux-x86_64-gnu/lib/python3.13/subprocess.py", line 1280, in wait
return self._wait(timeout=timeout)
~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/var/home/ducoterra/.local/share/uv/python/cpython-3.13.13-linux-x86_64-gnu/lib/python3.13/subprocess.py", line 2085, in _wait
(pid, sts) = self._try_wait(0)
~~~~~~~~~~~~~~^^^
File "/var/home/ducoterra/.local/share/uv/python/cpython-3.13.13-linux-x86_64-gnu/lib/python3.13/subprocess.py", line 2043, in _try_wait
(pid, sts) = os.waitpid(self.pid, wait_flags)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
validation FAILED (see output above)
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__05_e2e_image_progress.a1.err
View File
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__05_e2e_image_progress.a1.md
+19
View File
@@ -0,0 +1,19 @@
All green. Final report:
## Task 05 complete — dedicated E2E for the sync image-progress label
**Implemented**
- `tests/e2e/test_sync_image_progress.py` (new): module app on its own port (8171) + `slow_llm.py` delay proxy (8909, `SLOW_DELAY_S = 1.5 s`) in front of the mock; local-dir source with one PyMuPDF-built PDF (10 × ~4 KB text pages → 20 chunks / 10 embed batches, 4 distinct non-blank images, all surviving the phase-151 gates); ground truth computed in-test via `extract_pdf_text` + `chunk_document` with the importer's exact args, sizing knobs pinned to code defaults (leak-guard pattern)
- `test_label_advances_through_the_image_span` (anti-freeze proof): dense (~300 ms) page sampling of `#sync-progress-text` first-seen order → asserts ≥3 distinct `· image n/4` positions starting at 1, non-decreasing, never above the total, reaching it; the chunk part (`· chunk n/20`, total = the file's TRUE chunk count) appears earlier and coexists on every image-span label; A4 title/aria == box on every running sample; settle = `5 added` counts line, terminal label, parts cleared, image part gone + endpoint 0/0
- `test_mid_run_reload_re_attaches_with_image_position`: UI-click start, first observed position `1/4`, mid-run `page.reload()` → load-time re-attach renders the current `· image n/4` with A4 title/aria, then settles to the counts line
**Results**
- `uv run pytest tests/e2e/test_sync_image_progress.py -v --no-cov` (isolated): **2 passed**
- Neighbours (each in isolation): `test_sync_progress_box` 8 ✓, `test_sync_chunk_progress` 2 ✓, `test_sync_label_mobile` 4 ✓, `test_pdf_support` 6 ✓, `test_image_documents` 4 ✓
- `uv run pytest --cov=app --cov-report=term-missing`: 3526 passed, **TOTAL 99%** (>90%); `uv run ruff check .` clean; `uv run pyright` 0 errors
**Docs** — no doc impact (test-only task; no new setup/config/conventions). Phase dir move + atomic commit are the harness's, not made here.
**Decisions** — test 2 starts the sync via the UI click (the page's 2 s poll starts only on click/re-attach — an API-only start left the page unpolled); fixture timing verified against real logs (image pipeline = 1 describe + 2 embed batches = 4.5 s hold > 2 s poll; run ≈ 40 s).
**Next pending task** — none in `152_sync_progress_image_pass/` (task 05 was the last); phase 152 is ready for the harness to move to `complete/`.
.agents/reports/152_sync_progress_image_pass/152_sync_progress_image_pass__05_e2e_image_progress.a1.validate
+120
View File
@@ -0,0 +1,120 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 14%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 98%]
...................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 270 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 135 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 431 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 764 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 155 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 443 0 100%
--------------------------------------------------
TOTAL 5719 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
README.md
+7 -2
View File
@@ -334,8 +334,13 @@ whole git-source refresh in one click:
phase (the live file, its `(n/m)` file position, and — while a file's
embed batch is in flight — the pinned `· chunk n/m` embed position,
batch granularity: the first chunk of the in-flight batch over the
file's total content chunks) — shows in a text box under the button
row, wrapping (never clipped); the same untruncated label rides the
file's total content chunks; and — while a PDF's images are being
described — the `· image n/m` vision position: the 1-based image
about to be described over the file's total images, ONE continuous
counter across the PDF's three vision spans, so a long image pass
visibly climbs instead of sitting frozen on the last chunk
position) — shows in a text box under the button row,
wrapping (never clipped); the same untruncated label rides the
button title and the box, which is the SOLE `aria-live` region for
the sync area (`role="status"`, polite — phase 150 folded the
pre-phase result-line announcer into the box). At settle the box
TODO.md
+2
View File
@@ -1 +1,3 @@
# TODO
- [ ] image extraction from a PDF is throwing a 404 `https://brain.reeseapps.com/api/documents/679071cb-99f7-4c21-af46-2fa8ff916730/pages/1`
app/api/sync.py
+124 -1
View File
@@ -188,6 +188,23 @@ counts convention does NOT extend to it: the file counts survive the
terminal for the result line's context, the chunk position clears
with the file.
Phase 152 (task 01) adds the in-flight file's VISION-image position —
``images_done`` (the 1-based index of the image about to be described
and embedded) + ``images_total`` (the file's total images for this
run — the three PDF vision spans of ONE file share ONE continuous
counter per file; task 02 wires the importer's
``on_image_progress`` hook, task 03 the runner's closure that feeds
these fields). They are 0/0 before the first image, during the
``"overview"`` / ``"summaries"`` phases, and in terminal states — the
same spots ``current_chunk`` clears. Unlike the chunk position (every
file embeds chunks), a file may have NO images at all, so the per-file
``progress`` hook ALSO resets them to 0/0: the pairing rule, extended
— a ``progress`` call names the file and clears the image position,
and every image position until the next ``progress`` call belongs to
it (a file with no images must not inherit the previous PDF's
position). ASSUMPTION A5: no new ``phase`` value — the image spans
live inside ``phase: "import"``.
The ``failed`` state's ``error`` string is masked by the shared
sanitizer — the ``user:pass@`` masker now lives in :mod:`app.core.errors`
(imported here under the private name ``_sanitize_error``).
@@ -272,6 +289,21 @@ class SyncStatus:
convention covers the FILE counts only — the chunk position is a
sub-position of a file that's done, so it clears in the terminal
while ``files_done`` / ``files_total`` keep their final values.
Phase 152 (task 01) image-position fields: ``images_done`` is
the 1-based index of the vision image about to be described and
``images_total`` the in-flight file's total images for this run
(the importer's ``on_image_progress`` hook — the three PDF vision
spans of one file share ONE continuous counter, the module
docstring's contract). Like the chunk position, the image
position is a SUB-position of a file that's done: the
keep-final-counts convention covers the FILE counts only, so it
clears with the file — 0/0 before the first image, during the
``"overview"`` / ``"summaries"`` phases, and in terminal states
(the same spots the chunk position clears) — and the per-file
``progress`` hook resets it to 0/0 (a ``progress`` call names the
file and clears the image position; a file with no images must
not inherit the previous PDF's position).
"""
state: Literal["idle", "running", "success", "failed", "cancelled"] = "idle"
@@ -298,6 +330,14 @@ class SyncStatus:
# prelude/overview/summaries phases, and in terminal states.
current_chunk: int = 0
chunks_total: int = 0
# Phase 152 (task 01): the in-flight file's VISION-image position —
# the 1-based index of the image about to be described and the
# file's total images for this run (the importer's
# on_image_progress hook; the three PDF vision spans share ONE
# continuous counter per file). 0/0 before the first image, during
# the prelude/overview/summaries phases, and in terminal states.
images_done: int = 0
images_total: int = 0
_status = SyncStatus()
@@ -334,6 +374,15 @@ def sync_status() -> dict[str, Any]:
user-cancelled run — a terminal state with ``error`` null and
``detail`` empty; the run's final file + summary counts are kept,
the chunk position cleared.
Phase 152 (task 01): ``images_done`` / ``images_total`` are the
in-flight file's vision-image position (the importer's
``on_image_progress`` hook — the three PDF vision spans of one
file share ONE continuous counter): 0/0 before the first image, in
the prelude/overview/summaries phases, and in terminal states; the
per-file ``progress`` hook clears it at every file boundary (a
file with no images never shows a stale position). Additive only —
every pre-existing key keeps its name, type, and meaning.
"""
return {
"state": _status.state,
@@ -350,6 +399,15 @@ def sync_status() -> dict[str, Any]:
# (overview/summaries transitions + every terminal).
"current_chunk": _status.current_chunk,
"chunks_total": _status.chunks_total,
# Phase 152 (task 01): the in-flight file's VISION-image
# position (the importer's on_image_progress hook — the three
# PDF vision spans share ONE continuous counter per file):
# 0/0 before the first image, during the
# prelude/overview/summaries phases, and in terminal states
# (cleared at every point the import phase ends; the per-file
# progress hook re-clears it at every file boundary).
"images_done": _status.images_done,
"images_total": _status.images_total,
# Phase 98 (task 01): the phase machine — null/0/0 idle (the
# dataclass defaults) and in terminal states (which clear
# phase + current_summary but keep the final summary counts).
@@ -461,6 +519,11 @@ async def _run_sync() -> None:
# run — 0/0 until the import's first embed batch reports.
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image-position fields reset with the
# run — 0/0 until the import's first vision span reports (the
# per-file progress hook re-clears them at every file boundary).
_status.images_done = 0
_status.images_total = 0
try:
settings = get_settings()
# Step 1 (phase 41): fail fast — verify both models the sync
@@ -556,6 +619,15 @@ async def _run_sync() -> None:
_status.current_file = f"{source}/{rel}"
_status.files_done = done
_status.files_total = total
# Phase 152 (task 01): the pairing rule, extended — a
# progress call names the file AND clears the image
# position: every image position until the next progress
# call belongs to this file. A file with no images must
# not inherit the previous PDF's position (unlike the
# chunk position, not every file embeds chunks, so the
# run-start reset alone cannot cover it).
_status.images_done = 0
_status.images_total = 0
# Phase 129 (task 03): the in-flight file's embed-chunk
# position — the LLMClient.embed hook (task 01) fires before
@@ -567,6 +639,21 @@ async def _run_sync() -> None:
_status.current_chunk = chunk
_status.chunks_total = total
# Phase 152 (task 03): the in-flight file's VISION-image
# position — the importer's ``on_image_progress`` hook (task
# 02) fires before each of the PDF's vision describe calls
# with (position_1based, total), the file's ONE continuous
# counter across its three vision spans (the importer owns
# the continuity — this closure only lands the pair); the
# file's IDENTITY is the sibling ``progress`` hook's, which
# ALSO resets this position (the three hooks are paired:
# ``progress`` names the file and clears the image position,
# ``on_chunk_progress`` reports the embed-batch position,
# ``on_image_progress`` reports the vision-image position).
def _image_hook(done: int, total: int) -> None:
_status.images_done = done
_status.images_total = total
# Phase 98 (task 01): the phase machine — the import phase
# starts NOW (the model-check + clone/pull prelude above
# reported ``phase: null``).
@@ -577,7 +664,12 @@ async def _run_sync() -> None:
# imported.
_check_cancelled()
summary: ImportSummary = await import_sources(
sources, llm, prune=True, progress=_hook, on_chunk_progress=_chunk_hook,
sources,
llm,
prune=True,
progress=_hook,
on_chunk_progress=_chunk_hook,
on_image_progress=_image_hook, # phase 152 (task 03)
ignore_by_root=ignore_by_root,
include_hidden_by_root=include_hidden_by_root,
doc_dates_by_root=doc_dates_by_root,
@@ -596,6 +688,11 @@ async def _run_sync() -> None:
# values (the phase-64 convention covers files only).
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image position clears at the
# same point (a sub-position of a done file, like the
# chunk position).
_status.images_done = 0
_status.images_total = 0
overview = await regenerate_overview(llm)
# Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03):
# the folder summaries — the drill-down ls's per-level
@@ -649,6 +746,10 @@ async def _run_sync() -> None:
# chunk-position clear as the overview transition.
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image position clears at
# the same point (a sub-position of a done file).
_status.images_done = 0
_status.images_total = 0
folder_stats = await generate_folder_summaries(
fs_db, llm, changed_docs=summary.changed_paths,
on_progress=_summary_hook,
@@ -670,6 +771,11 @@ async def _run_sync() -> None:
# chunk-position clear.
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image position clears
# at the same point (a sub-position of a done
# file).
_status.images_done = 0
_status.images_total = 0
folder_stats = await generate_folder_summaries(
fs_db, llm, only_missing=True, on_progress=_summary_hook
)
@@ -716,6 +822,11 @@ async def _run_sync() -> None:
# line's context).
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image position is a sub-position of
# a file that's done — it clears in the terminal too (the FILE
# counts keep their final values, the sub-position does not).
_status.images_done = 0
_status.images_total = 0
_status.detail = {
"files": summary.files,
"added": summary.added,
@@ -754,6 +865,12 @@ async def _run_sync() -> None:
# success and failed (the convention split, pinned).
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image position clears in the
# CANCELLED terminal too — every terminal clears it (a
# sub-position of a file that's done/abandoned, the FILE
# counts above keep their final values).
_status.images_done = 0
_status.images_total = 0
logger.info(
"sync: cancelled by user state=cancelled files_done=%d files_total=%d",
_status.files_done, _status.files_total,
@@ -773,3 +890,9 @@ async def _run_sync() -> None:
# of a file that's done/abandoned does not).
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image position clears in the FAILED
# terminal too — the convention split (a sub-position of a
# file that's done/abandoned; the FILE counts keep their final
# values next to the error).
_status.images_done = 0
_status.images_total = 0
app/rag/importer.py
+180 -25
View File
@@ -285,10 +285,14 @@ log-only (the sync ``detail`` shape is untouched — the stats log-only
convention, phase 133).
``import_sources`` accepts an optional per-file ``progress`` callback
(phase 64, task 01) reporting the file being processed right now, and
an optional ``on_chunk_progress`` callback (phase 129, task 02)
(phase 64, task 01) reporting the file being processed right now, an
optional ``on_chunk_progress`` callback (phase 129, task 02)
reporting the in-flight file's embedding-chunk position at batch
granularity (see the function docstring).
granularity, and an optional ``on_image_progress`` callback (phase
152, task 02) reporting the in-flight PDF's vision-image position —
immediately before each of the PDF's vision describe calls, with the
image's 1-based position in the file's ONE continuous vision counter
and the span's count (see the function docstring).
"""
from __future__ import annotations
@@ -663,6 +667,7 @@ async def import_sources(
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None,
on_image_progress: Callable[[int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
@@ -708,9 +713,41 @@ async def import_sources(
``1/1``, which IS its only chunk): the document's summary-chunk
embed (:func:`_store_summary`) NEVER receives the hook, so the
counter counts content chunks only (no ``1/1`` noise after the
real progress ends), and the post-index enrichment pass / the
PDF's embedded-image docs (separate documents, not the walked
file) do not report either.
real progress ends), and the post-index passes (the phase-125
enrichment, the phase-137 document-context backfill) do not report
either — the PDF's vision describe calls report through the
separate ``on_image_progress`` hook instead (phase 152, below).
``on_image_progress`` (phase 152, task 02) is an optional
vision-image-position hook for the PDF file currently being
indexed: while the PDF's vision spans run — the phase-133
page-screenshot fallback, the phase-140 pictorial sparse-page
pass, and the phase-126 embedded-image pass — it is called
immediately before each of their vision describe calls with
``(position_1based, total)``: the 1-based position of the image
about to be described and the count for that span — the extracted
image count (post-dedupe, post-blank-gate, phase 151: the work
actually about to happen) for the embedded pass, the candidate-
page count for the pictorial pass, the rendered-page count for the
screenshot fallback. The spans of ONE file are reported as ONE
continuous counter: each span is called with the count already
reported for this file (*base*) and returns the number of images
it reported, and the caller adds them — the pass that runs second
starts where the first ended, so the position climbs across the
file's whole vision span instead of restarting (the fallback and
the pictorial pass are mutually exclusive — the phase-140
precedence over the garbage gate — and both precede the embedded
pass, which runs after the document commit). A skipped span (the
``images`` toggle off, zero candidates, a clean text layer) reports
nothing and advances the counter not at all. Standalone image
files get NO sub-position (phase 152 A4 — one image per file,
``1/1`` per file would be noise; the file count already covers
them), and the hook carries counts only, never the image identity
(A2 — the per-file ``progress`` hook names the file). Same "only
when provided" rule as ``progress`` and ``on_chunk_progress``:
``None`` (every existing caller — the CLI, the chat path) is a
zero-cost guarded no-op, and the run is byte-identical to a
hook-less run.
``ignore_by_root`` (phase 89, A1/A2) maps ``str(root)`` — the root path
string exactly as passed in *sources* — to that source's RAW ignore-path
@@ -861,7 +898,7 @@ async def import_sources(
await _index_file(
session, source=source, rel=rel, full_path=path, llm=llm,
summary=summary, raw_date=dates_map.get(rel),
on_chunk=on_chunk_progress,
on_chunk=on_chunk_progress, on_image=on_image_progress,
)
except EmbeddingError as e:
# A pathological file (e.g. content the embedding endpoint
@@ -1078,6 +1115,7 @@ async def _index_file(
summary: ImportSummary,
raw_date: datetime | None = None,
on_chunk: Callable[[int, int], None] | None = None,
on_image: Callable[[int, int], None] | None = None,
) -> None:
"""Upsert one file: doc row + chunk rows + embeddings, one transaction.
@@ -1112,6 +1150,15 @@ async def _index_file(
:func:`_index_pdf_file` (its content embed is the primary target).
``None`` (the default) keeps every branch byte-identical.
``on_image`` (phase 152, task 02) is the in-flight file's
vision-image-position hook (see :func:`import_sources`) — passed
through to the PDF branch only (:func:`_index_pdf_file` — its
three vision spans share ONE continuous counter per file): the
image branch receives NO sub-position (phase 152 A4 — one image
per file, ``1/1`` would be noise; the file count already covers
it) and the text branch has no vision span. ``None`` (the
default) keeps every branch byte-identical.
Image files (phase 122, toggle on) delegate to
:func:`_index_image_file` — the binary path (bytes digest,
persistent copy, ``content`` = the vision description) — BEFORE
@@ -1165,7 +1212,7 @@ async def _index_file(
return await _index_pdf_file(
session, source=source, rel=rel, full_path=full_path, llm=llm,
summary=summary, raw_date=raw_date, file_digest=file_digest,
on_chunk=on_chunk,
on_chunk=on_chunk, on_image=on_image,
)
content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
@@ -1401,6 +1448,7 @@ async def _index_pdf_file(
file_digest: str,
raw_date: datetime,
on_chunk: Callable[[int, int], None] | None = None,
on_image: Callable[[int, int], None] | None = None,
) -> None:
"""The phase-126 PDF branch of :func:`_index_file` — a PDF is
indexed from its EXTRACTED TEXT (LOCKED A1/A2/A4), one transaction
@@ -1488,6 +1536,26 @@ async def _index_pdf_file(
documents — their ``1/1`` positions would be noise after the
file's real progress ends, the A6 exclusion). ``None`` keeps the
branch byte-identical.
``on_image`` (phase 152, task 02) is the in-flight file's
vision-image-position hook (see :func:`import_sources`): it is
forwarded to the three vision spans — the phase-133 fallback
(:func:`_fallback_page_screenshots`), the phase-140 pictorial pass
(:func:`_pictorial_pdf_pass`), and the embedded-image pass
(:func:`_index_pdf_embedded_images`) — each firing it immediately
before its vision describe calls. The spans of this file share
ONE continuous counter: the local *images_reported* starts at 0,
each span is called with the count reported so far as its *base*
(the pass that runs second starts where the first ended), and the
returned reported count is added — the position climbs across the
file's whole vision span instead of restarting. The fallback and
the pictorial pass are mutually exclusive (the phase-140
precedence over the garbage gate — an unusable layer takes the
fallback, a usable one the pass), both run BEFORE the document
commit, and the embedded pass runs after it; a skipped span
(toggle off, zero candidates, a clean text layer) reports nothing
and advances the counter not at all. ``None`` keeps the branch
byte-identical.
"""
# Extract the text (fail-soft: an unopenable PDF yields "" — the
# document may still carry embedded images worth indexing).
@@ -1504,6 +1572,14 @@ async def _index_pdf_file(
# — the pre-phase-148 viewer).
fallback_fired = False
pictorial_described = False
# Phase 152 (task 02): the file's ONE continuous vision-image
# counter — 0 at the top of the file (the per-file pairing rule:
# each file's spans start fresh; the caller's status fields reset
# at the file boundary). Each span below is called with the count
# reported so far as its *base* and returns the number of images
# it reported, which is added — the pass that runs second starts
# where the first ended. Stays 0 while the hook is None.
images_reported = 0
# Phase 133 (task 03): the garbage gate — BEFORE the digest/chunk/
# summary pipeline, so it runs on the FINAL content exactly once.
# A clean layer never reaches the fallback (the gate is the only
@@ -1511,9 +1587,11 @@ async def _index_pdf_file(
if not is_usable_pdf_text(content):
summary.pdfs_screenshotted += 1
fallback_fired = True
content = await _fallback_page_screenshots(
session, source=source, rel=rel, full_path=full_path, llm=llm
content, reported = await _fallback_page_screenshots(
session, source=source, rel=rel, full_path=full_path, llm=llm,
on_image=on_image, base=images_reported,
)
images_reported += reported
elif llm.settings.pdf_pictorial:
# Phase 140 (task 03, LOCKED A1–A4): the pictorial pass —
# USABLE layers only (the precedence is locked: the phase-133
@@ -1523,9 +1601,11 @@ async def _index_pdf_file(
# pre-phase-140 path); otherwise the FINAL content carries the
# described candidates' illustration blocks (A4: the digest
# below runs over it, the phase-126/133 convention).
passed = await _pictorial_pdf_pass(
source=source, rel=rel, full_path=full_path, llm=llm, summary=summary
passed, reported = await _pictorial_pdf_pass(
source=source, rel=rel, full_path=full_path, llm=llm, summary=summary,
on_image=on_image, base=images_reported,
)
images_reported += reported
if passed is not None:
pictorial_described = True
content = passed
@@ -1650,6 +1730,7 @@ async def _index_pdf_file(
await _index_pdf_embedded_images(
session, source=source, rel=rel, full_path=full_path,
llm=llm, summary=summary, raw_date=raw_date,
on_image=on_image, base=images_reported,
)
@@ -1660,7 +1741,9 @@ async def _fallback_page_screenshots(
rel: str,
full_path: Path,
llm: Embedder,
) -> str:
on_image: Callable[[int, int], None] | None = None,
base: int = 0,
) -> tuple[str, int]:
"""The phase-133 (task 03) page-screenshot fallback — the owner's
specified path ("screenshotting the pages and reading each one as
an image") for a PDF whose text layer failed
@@ -1700,6 +1783,18 @@ async def _fallback_page_screenshots(
unchanged row's content digest (the row stays put until the
bytes move; the doc is still correct either way).
* phase 152 (task 02): the ``on_image`` hook (the file's
vision-image position, see :func:`import_sources`) is called
immediately before EACH rendered page's vision call with
``(base + i + 1, base + len(pages))`` — the page's 1-based slot
in the rendered-page list and the span's total (the rendered-
page count), at the START of the page's slow call, never after.
The return is the ``(content, reported)`` pair — *reported* the
number of images this span reported (0 while the hook is
``None``), the count the caller adds to the file's continuous
counter. A hook-less run is byte-identical (every call site is
guarded).
*session* is part of the signature for symmetry with the other
per-file index helpers (the fallback itself needs no DB access —
the caller's upsert pipeline owns the row).
@@ -1711,14 +1806,22 @@ async def _fallback_page_screenshots(
source, rel, len(pages),
)
if not pages:
return ""
return "", 0
blocks: list[str] = []
got_any = False
reported = 0
for i, png in enumerate(pages):
page = i + 1
if not png:
blocks.append(f"page {page} — description unavailable")
continue
# Phase 152 (task 02): the file's vision-image position —
# this page's 1-based slot and the span's total (the rendered-
# page count), at the START of the page's slow call, never
# after.
if on_image is not None:
on_image(base + i + 1, base + len(pages))
reported += 1
desc = await describe_image(llm, data=png, mime="image/png")
if desc is None or not desc.strip():
blocks.append(f"page {page} — description unavailable")
@@ -1726,10 +1829,13 @@ async def _fallback_page_screenshots(
blocks.append(desc.strip())
got_any = True
if not got_any:
return ""
return "\n\n".join(
f"{PAGE_SEPARATOR.format(page=page)}\n{block}"
for page, block in enumerate(blocks, start=1)
return "", reported
return (
"\n\n".join(
f"{PAGE_SEPARATOR.format(page=page)}\n{block}"
for page, block in enumerate(blocks, start=1)
),
reported,
)
@@ -1740,7 +1846,9 @@ async def _pictorial_pdf_pass(
full_path: Path,
llm: Embedder,
summary: ImportSummary,
) -> str | None:
on_image: Callable[[int, int], None] | None = None,
base: int = 0,
) -> tuple[str | None, int]:
"""The phase-140 (task 03) pictorial pass (LOCKED A1–A4) — the
IKEA assembly-instruction class (TODO L5): a PDF whose text layer
is USABLE but SPARSE (mostly pictorial pages — 4–73 chars of
@@ -1791,6 +1899,19 @@ async def _pictorial_pdf_pass(
(+the described page count), plus the ONE loud log line (PLAN
§9): ``pdf: pictorial pass source=… path=… — N candidate pages,
M described (min_chars=…)``.
* phase 152 (task 02): the ``on_image`` hook (the file's
vision-image position, see :func:`import_sources`) is called
immediately before EACH candidate's vision call with
``(base + i + 1, base + len(candidates))`` — the candidate's
1-based slot in the deterministic gate's list and the span's
total (the candidate-page count), at the START of the page's
slow call, never after (an unreadable page — the zero-byte
render placeholder — burns no vision call and reports nothing).
The return is the ``(content-or-None, reported)`` pair —
*reported* the number of images this span reported (0 while the
hook is ``None`` or there are zero candidates), the count the
caller adds to the file's continuous counter. A hook-less run
is byte-identical (every call site is guarded).
"""
settings = llm.settings
page_texts = extract_pdf_page_texts(full_path)
@@ -1798,17 +1919,18 @@ async def _pictorial_pdf_pass(
page_texts, min_chars=settings.pdf_pictorial_min_chars
)
if not candidates:
return None
return None, 0
# ONE render for the whole candidate set (the phase-140 selector —
# a 500-page PDF with 30 sparse pages renders 30 PNGs, not 500);
# the result is aligned with *candidates* (ascending, one slot
# each — a zero-byte placeholder never shifts the positions).
pngs = render_pdf_pages(full_path, pages=candidates)
descriptions: dict[int, str] = {}
reported = 0
# strict: the render's result is aligned with the candidate list by
# contract (one slot per named page, ascending) — a length drift is
# a parser regression, fail loud.
for page, png in zip(candidates, pngs, strict=True):
for i, (page, png) in enumerate(zip(candidates, pngs, strict=True)):
if not png:
# The phase-133 zero-byte placeholder: an unreadable page.
logger.warning(
@@ -1817,6 +1939,13 @@ async def _pictorial_pdf_pass(
page, source, rel,
)
continue
# Phase 152 (task 02): the file's vision-image position —
# this candidate's 1-based slot and the span's total (the
# candidate-page count), at the START of the page's slow
# call, never after.
if on_image is not None:
on_image(base + i + 1, base + len(candidates))
reported += 1
desc = await describe_pdf_page(
llm,
data=png,
@@ -1854,7 +1983,7 @@ async def _pictorial_pdf_pass(
if page in descriptions:
block += f"\n\n[Page {page} illustration: {descriptions[page]}]"
parts.append(f"{PAGE_SEPARATOR.format(page=page)}\n{block}")
return "\n\n".join(parts)
return "\n\n".join(parts), reported
def _persist_pdf_pages(
@@ -1951,7 +2080,9 @@ async def _index_pdf_embedded_images(
llm: Embedder,
summary: ImportSummary,
raw_date: datetime,
) -> None:
on_image: Callable[[int, int], None] | None = None,
base: int = 0,
) -> int:
"""The phase-126 (A3) embedded-image pass — one call per (re)indexed
PDF, after its document commit.
@@ -2034,6 +2165,19 @@ async def _index_pdf_embedded_images(
layer: origin-line-only blocks, fail-soft like the rest of the
parser.
* phase 152 (task 02): the ``on_image`` hook (the file's
vision-image position, see :func:`import_sources`) is called at
the START of EACH image's pipeline step — before the slow
:func:`_describe_or_skip` seam inside
:func:`_index_one_embedded_image` — with
``(base + i + 1, base + len(images))``: the image's 1-based slot
in the pass's list (post-dedupe, post-blank-gate, phase 151:
the work actually about to happen) and the span's total. Returns
the number of images reported (0 while the hook is ``None`` or
the toggle is off) — the count the caller adds to the file's
continuous counter. A hook-less run is byte-identical (every
call site is guarded).
``raw_date`` is the PDF's resolved source date (the pass runs only
for (re)indexed PDFs, so it is always concrete) — the image docs
inherit the PDF's date (they are part of it).
@@ -2048,7 +2192,7 @@ async def _index_pdf_embedded_images(
"source=%s path=%s",
source, rel,
)
return
return 0
# Phase 137 (task 02/03): the document context's raw material —
# the SHARED probe (task 01's per-page text map extracted ONCE per
# PDF — the pass reads it per image — plus the phase-133 garbage
@@ -2069,12 +2213,23 @@ async def _index_pdf_embedded_images(
)
summary.pdf_images_deduped += stats.duplicates
summary.pdf_images_blank_skipped += stats.blank_skipped
for img in images:
reported = 0
for i, img in enumerate(images):
# Phase 152 (task 02): the file's vision-image position —
# this image's 1-based slot in the pass's list (post-dedupe,
# post-blank-gate: the work actually about to happen) and the
# span's total, at the START of the image's pipeline step
# (the slow :func:`_describe_or_skip` seam inside
# :func:`_index_one_embedded_image`), never after.
if on_image is not None:
on_image(base + i + 1, base + len(images))
reported += 1
await _index_one_embedded_image(
session, img, source=source, rel=rel, full_path=full_path,
llm=llm, summary=summary, raw_date=raw_date,
page_texts=page_texts, text_usable=text_usable,
)
return reported
async def _index_one_embedded_image(
frontend/assets/sources.js
+93 -49
View File
@@ -364,7 +364,8 @@
*
* Phase 149 (task 01) — the sync detail box + the icon-only running
* button: the owner's report was that the full live label (a long
* source/relative/path + the counts + the chunk part) is clipped in
* source/relative/path + the counts + the chunk part + the phase-152
* image part) is clipped in
* the pill even on desktop — the mobile solution (stuffing the parts
* in the button) had become the permanent surface. The fix: while a
* sync/upload run is in flight, the button shows ONLY the spinning
@@ -399,7 +400,8 @@
* global [hidden] rule does the display). Two content modes, decided
* at the callsites (the phase-149 "the callsites are the only
* decision sites" rule):
* • running mode — setSyncLabel(main, count, chunk, running) writes
* • running mode — setSyncLabel(main, count, chunk, running, image)
* writes
* the SAME untruncated full label through setSyncProgress, ONLY
* while running (the idle / settle button labels — "Sync sources",
* "Synced HH:MM" — must not fill the panel; they stay on the
@@ -749,7 +751,12 @@ export async function mount(root) {
* pinned "(n/m)" position (hidden when absent); phase 129 (task
* 04): + .sync-label-chunk, the pinned "· chunk n/m" embed-batch
* position of the in-flight file (hidden in every state but the
* import phase after the file's first batch)), #sync-cancel-btn
* import phase after the file's first batch); phase 152 (task 03):
* the in-flight PDF's "· image n/m" vision-image position gets NO
* span of its own — the running button is the spinner (its parts
* are hidden, phase 149), so the position rides only the composed
* full string (the box / title / running aria-label)),
* #sync-cancel-btn
* (the Cancel control — phase 134 task 02, ship-hidden, revealed
* only while a sync run is in flight), #sync-icon (the
* spinner icon), #sync-progress (Phase 149 task 01 + Phase 150:
@@ -811,29 +818,38 @@ export async function mount(root) {
return `${pad(d.getHours())}:${pad(d.getMinutes())}`;
}
/* Phase 128 (task 01) + Phase 129 (task 04): the label PARTS
* writer — the SINGLE point that writes #sync-label. mainText lands
* in the MAIN part (.sync-label-main — the shrinks-to-fit prefix +
* file/folder, the part CSS ellipsizes); countText lands in the
* COUNT part (.sync-label-count — the pinned "(n/m)" position,
* NEVER clipped); chunkText (phase 129) lands in the CHUNK part
* (.sync-label-chunk — the pinned "· chunk n/m" embed-batch
* position of the in-flight file). The COUNT and CHUNK parts share
* one contract: present → its textContent + hidden=false, absent →
* emptied + hidden=true (each span is hidden whenever there is
* nothing of its kind to show). Then the UNTRUNCATED full label —
* mainText + (countText ? " " + countText : "") + (chunkText ?
* " " + chunkText : "") — rides the button title (A4 — the screen
* reader hears the whole "Syncing… <file> (2/3) · chunk 14/38"
* there, and in the #sync-progress box — the sole aria-live
* region, phase 150 — while a run is in flight); the visible
* spans carry main + count + chunk, the spaces between them are
* the CSS gap. textContent only — the house rule, no HTML strings.
* Every label state goes through here (the running builders, the
* idle / settle labels, the 202/409 click, the re-attach).
/* Phase 128 (task 01) + Phase 129 (task 04) + Phase 152 (task 03):
* the label PARTS writer — the SINGLE point that writes #sync-label.
* mainText lands in the MAIN part (.sync-label-main — the
* shrinks-to-fit prefix + file/folder, the part CSS ellipsizes);
* countText lands in the COUNT part (.sync-label-count — the pinned
* "(n/m)" position, NEVER clipped); chunkText (phase 129) lands in
* the CHUNK part (.sync-label-chunk — the pinned "· chunk n/m"
* embed-batch position of the in-flight file); imageText (phase
* 152) is the in-flight PDF's "· image n/m" vision-image position —
* it gets NO span of its own (the running button is the spinner —
* its parts are hidden, phase 149), it rides ONLY the composed full
* string. The COUNT and CHUNK parts share one contract: present →
* its textContent + hidden=false, absent → emptied + hidden=true
* (each span is hidden whenever there is nothing of its kind to
* show). Then the UNTRUNCATED full label — mainText + (countText ?
* " " + countText : "") + (chunkText ? " " + chunkText : "") +
* (imageText ? " " + imageText : "") — rides the button title (A4 —
* the screen reader hears the whole "Syncing… <file> (2/3) · chunk
* 14/38 · image 3/15" there, and in the #sync-progress box — the
* sole aria-live region, phase 150 — while a run is in flight); the
* visible spans carry main + count + chunk, the spaces between them
* are the CSS gap. textContent only — the house rule, no HTML
* strings. Every label state goes through here (the running
* builders, the idle / settle labels, the 202/409 click, the
* re-attach).
* Phase 149 (task 01): the fourth parameter — `running` (the
* in-flight flag; the callsites are the ONLY decision sites: the
* running entry passes true, the settle passes the default false).
* Phase 152 (task 03): the fifth parameter — `imageText` (the "·
* image n/m" part; the chunk and image parts are distinct and both
* may show, and each disappears independently — the bare builders
* and the non-running calls pass nothing, so both default "").
* Phase 150 (task 02): while running, the SAME untruncated `full`
* string goes into the box through the single box writer
* (setSyncProgress — textContent only), so the box can never
@@ -846,7 +862,7 @@ export async function mount(root) {
* non-running call restores the ship value "Sync sources" (the
* static HTML attribute is the idle truth — the JS only
* overwrites/restores it, never removes it). */
function setSyncLabel(mainText, countText = "", chunkText = "", running = false) {
function setSyncLabel(mainText, countText = "", chunkText = "", running = false, imageText = "") {
if (syncLabelMain) syncLabelMain.textContent = mainText;
if (syncLabelCount) {
if (countText) {
@@ -866,10 +882,16 @@ export async function mount(root) {
syncLabelChunk.hidden = true;
}
}
// Phase 152 (task 03): the image part is the LAST piece of the
// composed full string (after the chunk part) — the box, the
// button title, and the running aria-label all carry it; the
// button's label spans stay as-is (no new span — see the
// docstring).
const full =
mainText +
(countText ? ` ${countText}` : "") +
(chunkText ? ` ${chunkText}` : "");
(chunkText ? ` ${chunkText}` : "") +
(imageText ? ` ${imageText}` : "");
if (syncBtn) syncBtn.title = full; // A4: the untruncated label rides the title, always
// Phase 150 (task 02): the box — the sole sync text surface and
// the sole aria-live region — receives the running label through
@@ -924,6 +946,8 @@ export async function mount(root) {
* 04): this builder still returns NO chunk part — the pinned
* "· chunk n/m" part is added by fmtSyncPhaseLabel's import
* fall-through (the only site that sees the status's chunk
* fields), and phase 152 (task 03) likewise adds the "· image
* n/m" part there (the only site that sees the status's image
* fields), so the upload label and the 202/409 click stay bare. */
function fmtSyncLabel(kind, currentFile, done, total) {
const prefix = kind === "upload" ? "Importing" : "Syncing…";
@@ -932,30 +956,38 @@ export async function mount(root) {
return { main, count };
}
/* Phase 98 (task 02, D2) + Phase 128 (task 01, the PARTS rework):
* the SYNC job's phase-aware running label — the post-import span
* where the file count sits still (the owner's "the number pauses
* for a really long time"): "overview" → "Writing KB overview…"
* (no count part); "summaries" → main "Summarizing folders…
* <source/folder>" + count "(n/m)" — the folder part is omitted
* while current_summary is null (the phase's first poll) and the
* COUNT IS ALWAYS EMITTED in this phase (the hook always carries a
* total) and the CHUNK part stays empty (the summaries phase's own
* (n/m) is its position — the status's chunk fields are cleared at
* this transition); anything else (the null prelude, "import") →
* the phase-64 fmtSyncLabel parts, byte-identical (same builder,
* same fields, same order) + the phase-129 CHUNK part: "· chunk
* n/m" ONLY while chunks_total > 0 (a file's embed batch is in
* flight — n the 1-based first chunk of the batch, m the file's
* total content chunks, batch granularity), empty before the first
* batch (0/0) so the part hides. The UPLOAD job never passes a
/* Phase 98 (task 02, D2) + Phase 128 (task 01, the PARTS rework)
* + Phase 152 (task 03): the SYNC job's phase-aware running label —
* the post-import span where the file count sits still (the owner's
* "the number pauses for a really long time"): "overview" →
* "Writing KB overview…" (no count part); "summaries" → main
* "Summarizing folders… <source/folder>" + count "(n/m)" — the
* folder part is omitted while current_summary is null (the phase's
* first poll) and the COUNT IS ALWAYS EMITTED in this phase (the
* hook always carries a total) and the CHUNK and IMAGE parts stay
* empty (the summaries phase's own (n/m) is its position — the
* status's chunk + image fields are cleared at this transition);
* anything else (the null prelude, "import") → the phase-64
* fmtSyncLabel parts, byte-identical (same builder, same fields,
* same order) + the phase-129 CHUNK part: "· chunk n/m" ONLY while
* chunks_total > 0 (a file's embed batch is in flight — n the
* 1-based first chunk of the batch, m the file's total content
* chunks, batch granularity), empty before the first batch (0/0) so
* the part hides + the phase-152 IMAGE part: "· image n/m" ONLY
* while images_total > 0 (a PDF's vision span is in flight — n the
* 1-based image about to be described, m the file's total images —
* ONE continuous counter across the PDF's three vision spans, the
* importer owns the continuity), empty before the first image
* (0/0) so the part hides; the two positions are distinct and both
* may show ("… · chunk 125/126 · image 3/15"), each disappearing
* independently. The UPLOAD job never passes a
* status (its endpoint has no phase — D2) and keeps the bare
* fmtSyncLabel parts. The untruncated full label rides the button
* title + #sync-progress like every running label (A4 —
* setSyncLabel). */
function fmtSyncPhaseLabel(status) {
if (status.phase === "overview") {
return { main: "Writing KB overview…", count: "", chunk: "" };
return { main: "Writing KB overview…", count: "", chunk: "", image: "" };
}
if (status.phase === "summaries") {
const folder = status.current_summary ? ` ${status.current_summary}` : "";
@@ -963,11 +995,14 @@ export async function mount(root) {
main: `Summarizing folders…${folder}`,
count: `(${status.summaries_done}/${status.summaries_total})`,
chunk: "", // this phase's own (n/m) is its position — no chunk part
image: "", // the image position belongs to the import phase only
};
}
// null phase (the prelude) or "import" — today's byte-identical
// sync label (the phase-64 contract, unchanged) + the phase-129
// chunk part (hidden before the in-flight file's first batch).
// chunk part (hidden before the in-flight file's first batch) +
// the phase-152 image part (hidden before the in-flight PDF's
// first vision describe call).
const { main, count } = fmtSyncLabel(
"sync", status.current_file, status.files_done, status.files_total
);
@@ -975,7 +1010,11 @@ export async function mount(root) {
status.chunks_total > 0
? `· chunk ${status.current_chunk}/${status.chunks_total}`
: "";
return { main, count, chunk };
const image =
status.images_total > 0
? `· image ${status.images_done}/${status.images_total}`
: "";
return { main, count, chunk, image };
}
function fmtSyncResult(detail) {
@@ -1011,7 +1050,10 @@ export async function mount(root) {
* Phase 129 (task 04): the third, pinned CHUNK part ("· chunk n/m"
* — the in-flight file's embed-batch position) is destructured the
* same way (absent from the bare builders — the upload job and the
* 202/409 click — so the "" default keeps the part hidden there).
* 202/409 click — so the "" default keeps the part hidden there);
* phase 152 (task 03) destructures the fourth, IMAGE part ("·
* image n/m" — the in-flight PDF's vision-image position) the same
* way (it too rides only the composed full string — no span).
* Same never-stale mechanics as before (disabled, aria-busy,
* spinning icon, no is-error). Phase 149 (task 01): the writer call
* passes `running = true` — the detail box reveals with the full
@@ -1025,13 +1067,15 @@ export async function mount(root) {
if (!syncBtn) return;
syncBtn.disabled = true;
syncBtn.setAttribute("aria-busy", "true");
const { main, count, chunk = "" } =
const { main, count, chunk = "", image = "" } =
kind === "sync" && status
? fmtSyncPhaseLabel(status)
: fmtSyncLabel(kind, currentFile, done, total);
// the parts + the A4 untruncated title / announcer + the phase-149
// detail box (revealed, full label) + the running aria-label
setSyncLabel(main, count, chunk, true);
// detail box (revealed, full label — phase 152: with the image
// part when a PDF's vision span is in flight) + the running
// aria-label
setSyncLabel(main, count, chunk, true, image);
syncBtn.classList.remove("is-error");
if (syncIcon) syncIcon.classList.add("is-spinning");
// Phase 149 (task 01): the icon-only running button — the class
frontend/index.html
+3 -2
View File
@@ -483,8 +483,9 @@
</p>
</div>
<!-- Phase 149 (task 01): the sync detail box — the full UNTRUNCATED live
label ("Syncing… <source/rel/path> (n/m) · chunk n/m") in a bordered
panel under the button row. Phase 150 (task 01) folded the
label ("Syncing… <source/rel/path> (n/m) · chunk n/m" — phase 152
appends "· image n/m" while a PDF's vision span is in flight)
in a bordered panel under the button row. Phase 150 (task 01) folded the
announcer in: the previous live-region span beside it is retired,
and THIS box is now the SOLE sync text surface AND the
SOLE aria-live region for the sync area
tests/conftest.py
+6
View File
@@ -48,6 +48,12 @@ from app.config import Settings as _Settings # noqa: E402
os.environ["BOR_DOCS_REPO"] = ""
os.environ["BOR_SUGGESTIONS"] = json.dumps(_Settings.model_fields["suggestions"].default)
# Phase 152 (task 04, defect fix): the same leak class for the display
# name — an operator's local ``.env`` may legitimately carry
# ``BOR_APP_NAME`` (the dev environment runs a renamed build), and the
# default-metadata pins (``/api/config`` ``app_name``, the page titles
# + header brand) must see the code default.
os.environ["BOR_APP_NAME"] = _Settings.model_fields["app_name"].default
# Phase 62: the same leak class for the new UI customization settings —
# an operator's local ``.env`` may legitimately carry
tests/e2e/test_sync_image_progress.py
+885
View File
@@ -0,0 +1,885 @@
"""Phase 152 task 05 E2E (Playwright, mock-only): during a PDF sync whose
images take time to describe, the live label shows the image position
MOVING — the owner's "stuck for several minutes" (TODO.md:3: the label
sat on ``Syncing… … (2540/2540) · chunk 125/126`` through the PDF's
vision spans) is impossible on the fixture.
The dedicated story suite for ``152_sync_progress_image_pass`` (A16 —
one Playwright file per phase, run in isolation): the ``on_image_progress
hook (phase 152, tasks 02/03) fires immediately before each vision
describe call with ``(position, total)``, the runner's ``_image_hook``
closure lands it on ``SyncStatus.images_done/images_total``, and
``fmtSyncPhaseLabel`` renders ``· image n/m`` — the phase-149 box
(``#sync-progress``) is the surface that shows it, the button
``title``/``aria-label`` carry the UNTRUNCATED composed string (A4).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_sync_image_progress.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic mock plus the phase-64 ``tests/e2e/slow_llm.py`` delay
proxy in front of it (``SLOW_DELAY_S = 1.5 s``) that stretches EVERY
LLM call (vision describes included) so the fixture's vision span
outlives the UI's 2 s status poll and each position outlives the
recorder's ~300 ms page-sampling cadence by ~10× — the observed
positions are dense samples of a real climb, not races.
Timing fixture (the phase-129/98 rationale, re-sized for vision spans):
the fixture's ONE-PDF run is ≈ 27 LLM calls ≈ 40 s: 2 model probes +
10 in-flight embed batches (the ~4 KB-per-page text packs into 20
content chunks at the app's code-default, pinned sizing — the token
budget groups them into 10 requests of 2) + 4 image pipelines (one
vision describe + 2 content-embed batches each — the image doc's
context + description content packs into 3 chunks) + 1 PDF summary +
1 overview + 1 folder summary. The image span itself is 4 × (1.5 s
describe + 2 × 1.5 s embed) = 18 s: each of the 4 positions holds
for 4.5 s — that outlives the UI's 2 s poll (at least one tick lands
in every 4.5 s window — every position is rendered, not a race) and
the test's ~300 ms page sampling sees every rendered value.
KB fixture (the ``test_local_directory_sources.py`` / phase-94 API
registration pattern — a host temp dir, the app runs on the same host;
no git anywhere): ``imgpass/`` with EXACTLY ONE file, a PyMuPDF-built
``imgpass.pdf`` (LOCKED A1) — 10 text pages (each ~4 KB of
deterministic filler, well over the 100-char bar — the phase-140
pictorial gate never fires, and the layer is clean — the phase-133
fallback never fires: the EMBEDDED-IMAGE pass is the file's one and
only vision span) + 4 GENUINELY DISTINCT non-blank images on pages
1–4 (different sizes + seeds → different stored bytes — the phase-151
dedupe keeps all four; colorful noise, sample mean ~127 — well under
the 235 near-white floor, so the blank gate keeps all four: the
span's total is the post-dedupe/post-gate count, the work actually
about to happen). The expected chunk count is computed IN THE TEST by
calling ``extract_pdf_text`` (the SAME parser the importer uses) and
``chunk_document`` (the SAME function + arguments — the importer's
``max(400, target)`` floor + the overlap setting — as
``_replace_content_chunks``), read back from the PDF on disk — the
ground truth never drifts from the app under test. The sizing knobs
are pinned to the code defaults in the app-under-test env (the
leak-guard pattern — an operator's gitignored ``.env`` tuning
``BOR_CHUNK_TARGET_CHARS`` must not change the suite's ground truth).
The suite runs ONE module app (it binds its own port — the conftest
session app is never started in an isolated run) over the shared E2E
Postgres: the SLOW app (``BOR_LLM_BASE_URL`` = the slow proxy in front
of the mock), ``BOR_IMAGES=true`` + the ``pdf`` import token pinned
EXPLICITLY (an operator's local gitignored ``.env`` cannot leak the
scope in either direction) + ``BOR_PDF_PICTORIAL`` pinned off (the
house E2E convention — this suite's span is the embedded pass; the
fixture's ≥100-char pages make the pictorial pass a no-op even if the
knob leaked on).
Autouse cleanup (the phase-96 pattern): before each test wait for no
running sync on the module app (the run state lives in its memory),
then truncate the shared registry + KB tables (documents/chunks/
query_log/steering_notes/kb_overview/git_sources/sources_meta/
folder_summaries) — each test seeds its own source row and runs its
own sync; after each test the same truncate.
Test → observable mapping (Playwright Mapping Rule):
1. ``test_label_advances_through_the_image_span`` (the anti-freeze
proof — the UI's own 2 s poll, densely sampled at ~300 ms): after
clicking **Sync sources**, the test collects the DISTINCT
``#sync-progress-text`` values seen while the run is in flight
(first-seen order) — and asserts: (a) at least THREE distinct
labels containing ``· image `` are observed (the freeze the owner
reported is a single repeated label — "≥3 distinct positions" IS
the bug's negation), the positions START at 1, are
NON-DECREASING in observation order, never EXCEED the total (4 —
the post-gate count), and REACH it; (b) the CHUNK part appeared
EARLIER in the run (the first ``· chunk n/20`` label precedes the
first ``· image `` label — the two positions are distinct) and
every image-span label carries the chunk part WITH IT (the last
chunk position persists through the file's image span —
"… (1/1) · chunk 19/20 · image 3/4"), its total == the file's
TRUE chunk count; (c) the button ``title`` + ``aria-label`` carry
the SAME full string as the box on every running sample (A4 —
asserted per sample, atomically); (d) at settle the box shows the
counts line (``5 added`` — the PDF + its 4 image docs), the label
reads the terminal ``Synced HH:MM``, the count + chunk parts are
hidden + empty, and the image part is GONE (no stale position in a
terminal state — the endpoint's terminal carries 0/0 + the cleared
chunk position while the file counts keep their final values).
2. ``test_mid_run_reload_re_attaches_with_image_position`` (the
never-stale contract): the API starts the sync, the page's own
poll renders the image span (the FIRST observed position is
``1/4`` — the whole span is ahead), the test reloads the page
mid-run, and the load-time re-attach (``initSyncButton`` — a
running sync re-enters with the WHOLE status, phase 98) renders
the CURRENT image position with the A4 title/aria — then the run
settles to the counts line.
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import httpx
import pymupdf as fitz # PyMuPDF — the fixture-PDF builder (LOCKED A1)
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.rag.chunker import chunk_document
from app.rag.llm import LLMClient
from app.rag.pdf import extract_pdf_text, is_usable_pdf_text
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
MOCK_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module's app binds its own port
# instead (a same-port second uvicorn dies on bind and would drive the
# wrong server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_IMGPROG", "8171"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
#: The slow-LLM proxy's port (the conftest's mock LLM stays on
#: MOCK_PORT — the proxy fronts it).
SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT_IMGPROG", "8909"))
SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}"
#: Per-LLM-request delay on the proxy — sized so each in-flight VISION
#: PIPELINE's position (the describe + the content embed) outlives the
#: UI's 2 s poll (see the module docstring's timing fixture paragraph).
SLOW_DELAY_S = "1.5"
# --------------------------------------------------------------------------
# Fixture constants (deterministic, token-controlled)
# --------------------------------------------------------------------------
#: The local source — the temp directory's basename (``kind=local`` →
#: the directory's basename is the source name, phase 38).
SOURCE = "imgpass"
#: The single fixture file (the source is one file — the count part
#: reads "(1/1)" throughout the import phase).
PDF_NAME = "imgpass.pdf"
N_FILES = 1
#: The fixture PDF's shape — 10 text pages (each ~4 KB of filler —
#: 20 content chunks at the code-default sizing, 10 embed batches) +
#: 4 distinct non-blank images on pages 1–4 (the span's total —
#: post-dedupe, post-blank-gate: the work actually about to happen).
N_PAGES = 10
N_IMAGES = 4
#: The settled counts line (phase 150 — the box is the sole sync text
#: surface): the PDF + its 4 embedded-image docs, a fresh KB (nothing
#: updated/unchanged/pruned).
SETTLE_TEXT = f"{1 + N_IMAGES} added"
#: The three sizing knobs, pinned to the CODE DEFAULTS in both the
#: app-under-test env (the leak guards below) and the test process's
#: expected-value computation — an operator's gitignored ``.env``
#: (which may legitimately tune ``BOR_CHUNK_TARGET_CHARS``) cannot
#: change the suite's ground truth. Derived from the class fields, so
#: they never drift from app/config.py (the conftest leak-guard
#: pattern).
CHUNK_TARGET = _Settings.model_fields["chunk_target_chars"].default
CHUNK_OVERLAP = _Settings.model_fields["chunk_overlap_chars"].default
EMBED_BATCH_SIZE = _Settings.model_fields["embed_batch_size"].default
#: The test process's mirror of the app-under-test sizing settings
#: (explicit kwargs beat any ``.env`` for the fields the expected
#: values depend on).
_EXPECTED_SETTINGS = _Settings(
chunk_target_chars=CHUNK_TARGET,
chunk_overlap_chars=CHUNK_OVERLAP,
embed_batch_size=EMBED_BATCH_SIZE,
)
#: "Synced HH:MM" — the local-time last-result label (sources.js's
#: fmtSyncTime), any hour/minute (the test_sync_button.py pattern).
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}$")
#: The settled boot label — idle ("Sync sources") or the RE-ATTACHED
#: terminal from the previous test's sync (the module app is shared,
#: its in-memory sync state survives the test boundary, and the
#: phase-32 boot re-attach renders the last result).
IDLE_OR_SETTLED = re.compile(r"^(Sync sources|Synced \d{1,2}:\d{2})$")
#: The chunk part (the phase-129 unit-pinned shape): "· chunk n/m" —
#: ``m`` is asserted against the expected chunk count by the test.
CHUNK_PART_RE = re.compile(r"· chunk (\d+)/(\d+)")
#: The image part (the phase-152 unit-pinned shape): "· image n/m".
IMAGE_PART_RE = re.compile(r"· image (\d+)/(\d+)")
#: Generous settle budgets: a changed sync against the slowed LLM is
#: ≈ 40 s; the UI's 2 s poll settles at most one tick after the
#: terminal state lands.
SETTLE_TIMEOUT_MS = 120_000
# ---------------------------------------------------------------------------
# The fixture PDF (built with PyMuPDF — LOCKED A1)
# ---------------------------------------------------------------------------
def _noise_png(w: int, h: int, seed: int) -> bytes:
"""A deterministic colorful-noise PNG (the "real image" stand-in).
Different ``(w, h, seed)`` → different stored bytes (the phase-151
byte-dedupe keeps all four); the formula's sample mean is ~127 —
well under the phase-151 blank gate's 235 near-white floor, and
the samples scatter far outside the ±17 band — every image
survives the gate with room to spare (see the module docstring's
KB fixture paragraph)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, w, h))
for y in range(h):
for x in range(w):
pix.set_pixel(
x,
y,
((x * 7 + y * 11 + seed * 13) % 256,
(x * 3 + y * 17 + seed * 29) % 256,
(x * 5 + y * 23 + seed * 41) % 256),
)
return pix.tobytes("png")
#: The four fixture images' shapes (distinct sizes — the distinct-
#: bytes guarantee even if two seeds collided) — one per page 1–4.
_IMAGE_SHAPES = [(24, 18), (32, 24), (40, 20), (28, 28)]
def _page_text(k: int) -> str:
"""One page's deterministic ~4 KB filler (no randomness — the
expected chunk count is stable run to run)."""
base = (
f"Page {k} notes for the imgpass fixture document carry "
f"deterministic filler number {k} of the ten-page pdf, written "
f"out to size so the chunker has real content to work with "
f"across the page. "
)
return base * 11
def _build_pdf(pdf_path: Path) -> None:
"""The fixture PDF: ``N_PAGES`` text pages (``insert_textbox`` —
the return value is checked — an overfull rect would silently
drop text and skew the ground truth) + the 4 images on pages 1–4
(distinct ``insert_image`` streams — distinct xrefs, distinct
bytes). The metadata title is the stem (no title — the stem
fallback, phase 126 A2)."""
doc = fitz.open()
for k in range(1, N_PAGES + 1):
page = doc.new_page()
rc = page.insert_textbox(
fitz.Rect(48, 90, 548, 812), _page_text(k), fontsize=10
)
assert rc >= 0, f"the page {k} text did not fit the rect (rc={rc})"
if k <= N_IMAGES:
w, h = _IMAGE_SHAPES[k - 1]
page.insert_image(
fitz.Rect(72, 820 - h * 2, 72 + w * 2, 860),
stream=_noise_png(w, h, k),
)
doc.save(pdf_path)
doc.close()
def _expected_chunks(pdf_path: Path) -> list[str]:
"""The PDF's expected content chunks — the SAME function and
arguments the importer's ``_replace_content_chunks`` uses
(``chunk_document(content, rel, max(400, target), overlap)`` — the
importer's token-cap floor rule; ``rel`` = the source-relative
path) over the SAME parser the importer reads
(``extract_pdf_text`` — the ``---PAGE-N---`` page markers): the
endpoint's ``chunks_total`` must equal
``len(chunk_document(...))`` — the file's TRUE chunk count, not a
batch count."""
content = extract_pdf_text(pdf_path)
return chunk_document(
content, PDF_NAME, max(400, CHUNK_TARGET), CHUNK_OVERLAP
)
def _expected_batch_count(chunks: list[str]) -> int:
"""The expected embed-batch count — the SAME
``LLMClient._batch_texts`` partition the app's embed loop walks
(the phase-129 unit pattern — call it directly for the boundaries).
The task's sizing bar: ≥ 2 batches (the chunk part's climb is a
real sequence, not a single point)."""
return len(LLMClient(_EXPECTED_SETTINGS)._batch_texts(chunks))
# ---------------------------------------------------------------------------
# The in-browser snapshot functions (Playwright Mapping Rule — the
# rendered DOM is the observable).
#
# ``_JS_SAMPLE`` — test 1's dense sampler: the box text + the button
# title/aria read in the SAME in-browser evaluation (a 2 s poll tick
# updates the parts + the title together in one JS task, so the atomic
# check can never straddle one — the phase-128 pattern); ``running``
# marks the in-flight labels (the A4 title/aria contract applies to
# them).
#
# ``_JS_IMAGE_SNAP`` — test 2's wait: null until the box carries the
# ``· image n/<arg>`` part, then the texts (read atomically with the
# title/aria).
# ---------------------------------------------------------------------------
_JS_SAMPLE = """
() => {
const boxTextEl = document.querySelector('#sync-progress-text');
const btn = document.querySelector('#sync-btn');
if (!boxTextEl || !btn) return null;
const box = boxTextEl.textContent;
return {
box: box,
title: btn.getAttribute('title') || '',
aria: btn.getAttribute('aria-label') || '',
running: box.startsWith('Syncing…'),
};
}
"""
_JS_IMAGE_SNAP = """
(n) => {
const boxTextEl = document.querySelector('#sync-progress-text');
const btn = document.querySelector('#sync-btn');
if (!boxTextEl || !btn) return null;
const box = boxTextEl.textContent;
const m = box.match(/^Syncing… .*· image (\\d+)\\/(\\d+)$/);
if (!m || Number(m[2]) !== n) return null;
return {
box: box,
posN: Number(m[1]),
title: btn.getAttribute('title') || '',
aria: btn.getAttribute('aria-label') || '',
};
}
"""
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def imgpass_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The one-source temp tree (see the module docstring): the app
server runs on the same host, so the paths are visible to it.
The directory NAME is the source name (``kind=local``, phase 38).
EXACTLY ONE file: the PyMuPDF-built ``imgpass.pdf`` (10 text
pages + 4 distinct non-blank images — see :func:`_build_pdf`)."""
root = tmp_path_factory.mktemp("bor_imgprog")
src = root / SOURCE
src.mkdir()
_build_pdf(src / PDF_NAME)
assert len(list(src.rglob("*"))) == N_FILES
return src
def _app_env(base_url: str, tmp_path_factory: pytest.TempPathFactory) -> dict[str, str]:
"""The per-module app env (the conftest pattern, cf.
``test_oneshot_llm_retry.py`` / ``test_kb_tree.py``): the
``base_url`` picks the LLM (the slow proxy in front of the mock),
``BOR_GIT_SOURCES`` forced empty (the sync source is this suite's
own DB-registered local row), the sizing knobs pinned to the code
defaults (the leak guards — see the module constants), the
phase-126 knobs pinned EXPLICITLY (``BOR_IMPORT_EXTENSIONS`` with
the ``pdf`` token — without it the walk is blind to the fixture —
and ``BOR_IMAGES=true`` — the embedded pass is the span under
test), the E2E ``BOR_PDF_PICTORIAL`` off-pin (the house
convention — the suite's span is the embedded pass; the fixture's
≥100-char pages keep the pictorial pass a no-op even if the knob
leaked on), and the other leak-guarded code defaults."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url
)
# Mock-calibrated threshold (conftest pattern): this suite never
# asks the chat model anything — the gate is never on a path.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
# Phase 67: instant retry waits + the code-default budget (the
# conftest leak-guard pattern).
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
# reads it from cwd) — override it with an EMPTY value (the env var
# beats the .env file): the registry must hold EXACTLY the local
# directory this suite registers.
env["BOR_GIT_SOURCES"] = ""
# Leak guards (conftest pattern): an operator's local (gitignored)
# .env cannot leak corpus-specific settings into the app under
# test — including the chunking / batching knobs this suite's
# ground truth is computed from (see the module constants).
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
_Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
env["BOR_CHUNK_TARGET_CHARS"] = str(CHUNK_TARGET)
env["BOR_CHUNK_OVERLAP_CHARS"] = str(CHUNK_OVERLAP)
env["BOR_EMBED_BATCH_SIZE"] = str(EMBED_BATCH_SIZE)
# Phase 126/122: the import scope WITH the ``pdf`` token + the
# embedded-image toggle ON (the span under test) + the E2E
# pictorial off-pin + the suite-private image home (the app under
# test must not write copies into the owner's real ~/bor-sources).
env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf"
env["BOR_IMAGES"] = "true"
env["BOR_PDF_PICTORIAL"] = "0"
env["BOR_IMAGE_DIR"] = str(tmp_path_factory.mktemp("bor_imgprog_images"))
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
return env
@pytest.fixture(scope="module")
def slow_llm(mock_llm: int) -> Iterator[int]:
"""The delay-injecting reverse proxy in front of the mock LLM
(``tests/e2e/slow_llm.py``) — the timing fixture for the whole
suite: it delays EVERY LLM call, vision describes included, so
each in-flight image pipeline's position outlives the UI's 2 s
poll (see ``SLOW_DELAY_S`` + the module docstring's timing
fixture paragraph)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S
env["E2E_MOCK_PORT"] = str(MOCK_PORT)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app",
"--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{SLOW_URL}/v1/models")
yield SLOW_PORT
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_server(
mock_llm: int,
slow_llm: int,
imgpass_dir: Path,
tmp_path_factory: pytest.TempPathFactory,
) -> Iterator[str]:
"""The SLOW app under test (the whole suite):
``BOR_LLM_BASE_URL`` is the slow proxy in front of the mock (the
timing fixture — see the module docstring). ``imgpass_dir`` is a
dependency only for the fixture ordering (the temp tree exists
before the first test). Skips the module (clear instructions)
when Postgres is down — the sync needs the KB tables."""
env = _app_env(f"{SLOW_URL}/v1", tmp_path_factory)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
body = httpx.get(f"{APP_URL}/api/health", timeout=5).json()
if body["db"] != "up":
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
share one Postgres, so a leftover git_sources row or document would
pollute the file counts + the counts line the syncs and the
assertions pin exactly. ``sources_meta`` (the KB generation counter)
resets with the KB — the sync re-creates the row."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources, sources_meta, folder_summaries"
)
)
db.commit() # without the commit the TRUNCATE rolls back (the house pattern)
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
"""No background sync may leak across tests (the run state lives in
the app's memory, and a still-running run would keep importing into
the NEXT test's truncated KB): wait for the app's sync status to
be non-running BEFORE the truncate. Own admin session (the
endpoint is admin-only) — usually a no-op: the tests settle only
after their run's terminal state."""
with httpx.Client(base_url=base_url, timeout=5.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
if r.json()["state"] != "running":
return
time.sleep(0.2)
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
@pytest.fixture(autouse=True)
def _clean(app_url: str) -> Iterator[None]:
"""Per-test KB isolation (the phase-96 cleanup pattern): the
shared E2E Postgres is truncated before AND after every test —
each test seeds its own source row and runs its own sync, so the
KB rows + folder_summaries rows of the temp source are exactly
this test's own. The pre-wait keeps a leaked run from the previous
test (the module app is shared) from racing the truncate."""
_wait_sync_settled(app_url)
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _register_source(base_url: str, path: str) -> None:
"""Register the temp directory through the authenticated API (the
``test_local_directory_sources.py`` / phase-94 pattern — a plain
directory is an API/DB-only operation)."""
with httpx.Client(base_url=base_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.post(
"/api/git-sources", json={"kind": "local", "path": path}
)
assert r.status_code == 201, r.text
def _wait_terminal(app_url: str, timeout_s: float = 180.0) -> dict[str, Any]:
"""Poll the status endpoint (own admin session) until the
UI-/API-triggered run reaches a terminal state — the source of
truth for the settle (the UI's 2 s poll renders it at most one
tick later). A FAILED terminal fails the test with the terminal
body (the ``detail`` / error text) — never an opaque label
timeout."""
with httpx.Client(base_url=app_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
body = r.json()
if body["state"] in ("success", "failed"):
assert body["state"] == "success", body
return body
time.sleep(0.2)
raise AssertionError(f"the run never reached a terminal state: {body}")
def _start_sync_click(page: Page, app_url: str, sync_dir: Path) -> None:
"""Register the source, land the admin on the sources view, wait
for the settled boot label (the phase-64 pattern — also proves the
phase-32 boot re-attach completed, so no later re-render can
clobber the running state), then click **Sync sources** and pin
the click's immediate 202 state (the bare "Syncing…" — the count
AND the chunk parts hidden: no file yet, no batch in flight — and
the image part is absent too: no PDF's vision span has started,
``images_total == 0``)."""
_register_source(app_url, str(sync_dir))
login(page, app_url) # lands on /sources.html (the button's home)
btn = page.locator("#sync-btn")
expect(btn).to_be_visible(timeout=30_000)
expect(page.locator("#sync-label")).to_have_text(IDLE_OR_SETTLED)
expect(btn).to_be_enabled()
btn.click()
# The click's immediate state (the 202 moment — the bare prefix,
# the phase-64 contract): disabled, aria-busy, spinning icon, and
# the label's MAIN part alone ("Syncing…" — the 202 click carries
# no status, so the pinned parts hide, phase 128/129 writers).
# Phase 149 (task 01): the running detail rides the box — the 202's
# bare label is in #sync-progress (the button is the spinner).
expect(btn).to_be_disabled()
expect(btn).to_have_attribute("aria-busy", "true")
expect(btn.locator(".sync-icon")).to_have_class(re.compile(r"\bis-spinning\b"))
expect(page.locator("#sync-progress")).to_have_text("Syncing…")
expect(page.locator(".sync-label-count")).to_be_hidden()
expect(page.locator(".sync-label-chunk")).to_be_hidden()
expect(page.locator("#sync-error-banner")).to_be_hidden()
# ---------------------------------------------------------------------------
# 1. The anti-freeze proof — the label advances through the PDF's
# image span (≥3 distinct positions, monotonic, chunk part earlier,
# the A4 title/aria, the settled clearing)
# ---------------------------------------------------------------------------
def test_label_advances_through_the_image_span(
page: Page, app_url: str, imgpass_dir: Path
) -> None:
"""A fresh admin page on ``/sources.html`` clicks **Sync sources**;
the test then densely samples (≈ every 300 ms) the rendered
``#sync-progress-text`` — NOT the API (the rendered DOM is the
observable; the UI's own 2 s poll is the writer) — collecting the
DISTINCT values in first-seen order until the settled counts line,
and asserts the phase-152 contract end to end (see the module
docstring's test mapping): the image part climbs (≥3 distinct
positions, starting at 1, non-decreasing, never above the total,
reaching it), the chunk part appeared earlier (and persists
through the image span with the file's TRUE chunk total), the
A4 title/aria ride the full string on every running sample, and
the settle clears the image position (the counts line, the
terminal label, the cleared endpoint position — no stale
position in a terminal state)."""
page.set_default_timeout(30_000)
# The ground truth — computed from the PDF ON DISK with the SAME
# parser + function + arguments the importer uses (see the helper
# docstrings): the TRUE chunk count + the batch count (the task's
# sizing bar — ≥2 embed batches, so the chunk part's climb is a
# real sequence).
pdf_path = imgpass_dir / PDF_NAME
chunks = _expected_chunks(pdf_path)
n_chunks = len(chunks)
assert n_chunks >= EMBED_BATCH_SIZE + 1, n_chunks # 2+ embed batches
assert _expected_batch_count(chunks) >= 2 # the sizing bar
# The happy path's premise: a clean text layer (the phase-133
# fallback never fires) — the embedded-image pass is the file's
# one and only vision span (the ≥100-char pages keep the phase-140
# gate candidate-free even under the app's pinned-off knob).
assert is_usable_pdf_text(extract_pdf_text(pdf_path))
_start_sync_click(page, app_url, imgpass_dir)
# The observation loop (the task's "poll the page, not the API"):
# dense samples of the rendered box text (≈ 300 ms — the evaluate
# round-trip + the sleep) in first-seen order, each sample read
# atomically with the button's title/aria (A4 checked per running
# sample), until the settled counts line lands (or the generous
# deadline — a failed run then fails LOUDLY in _wait_terminal
# with the terminal body, never an opaque timeout).
seen: list[str] = []
last: str | None = None
deadline = time.monotonic() + 150
while time.monotonic() < deadline:
snap = page.evaluate(_JS_SAMPLE)
assert snap is not None
if snap["running"]:
# A4 — while a run is in flight, the title + the
# aria-label carry the SAME untruncated full string as the
# box (the "· image" part included).
assert snap["title"] == snap["box"], snap
assert snap["aria"] == snap["box"], snap
if snap["box"] != last:
seen.append(snap["box"])
last = snap["box"]
if snap["box"] == SETTLE_TEXT:
break
time.sleep(0.2)
# --- the settle (the UI's 2 s poll renders the terminal at most one
# tick after it lands; the endpoint is the source of truth) ---
assert last == SETTLE_TEXT, f"the run never settled to the counts line: {seen[-5:]}"
terminal = _wait_terminal(app_url)
# The counts line's ground truth: the PDF + its 4 embedded-image
# docs — one file, 5 added.
assert terminal["detail"]["added"] == 1 + N_IMAGES, terminal
# The keep-final-file-counts split (the phase-64/129 convention,
# restated for the phase-152 sub-position): the FILE counts keep
# their final values, the sub-file chunk + image positions clear
# with the file (0/0 in the terminal — no stale position), the
# phase clears.
assert terminal["files_done"] == N_FILES and terminal["files_total"] == N_FILES, terminal
assert terminal["current_file"] is None, terminal
assert terminal["current_chunk"] == 0 and terminal["chunks_total"] == 0, terminal
assert terminal["images_done"] == 0 and terminal["images_total"] == 0, terminal
assert terminal["phase"] is None, terminal
# --- the run was real (the prelude + the import span were
# sampled) ---
assert any(t == "Syncing…" for t in seen), f"no bare prelude label: {seen[:6]}"
import_labels = [t for t in seen if t.startswith("Syncing… ")]
assert any(f"({N_FILES}/{N_FILES})" in t for t in import_labels), (
f"no file count part: {import_labels[:6]}"
)
# --- the chunk part appeared EARLIER in the run (the two
# positions are distinct) ---
chunk_idx = next(
(i for i, t in enumerate(seen) if CHUNK_PART_RE.search(t)), None
)
assert chunk_idx is not None, f"no chunk part observed: {seen}"
first_chunk = CHUNK_PART_RE.search(seen[chunk_idx])
assert first_chunk is not None, seen[chunk_idx]
# The chunk part's total is the file's TRUE chunk count (not a
# batch count — the phase-129 A6 assumption, the E2E lens).
assert int(first_chunk.group(2)) == n_chunks, (first_chunk.group(0), n_chunks)
# --- the image part: ≥3 distinct positions (the anti-freeze
# proof — the owner's freeze is a single repeated label), the
# positions START at 1, are NON-DECREASING in observation order,
# never EXCEED the total (the post-dedupe/post-gate count — the
# work actually about to happen), and REACH it ---
def _image_part(t: str) -> re.Match[str]:
m = IMAGE_PART_RE.search(t)
assert m is not None, t # the list comprehension's filter guarantees it
return m
image_items = [
(i, _image_part(t))
for i, t in enumerate(seen)
if IMAGE_PART_RE.search(t)
]
assert len(image_items) >= 3, (
f"fewer than 3 distinct image positions observed: "
f"{[m.group(0) for _, m in image_items]}"
)
first_image_idx = image_items[0][0]
assert chunk_idx < first_image_idx, (chunk_idx, first_image_idx)
positions = [(int(m.group(1)), int(m.group(2))) for _, m in image_items]
assert positions[0][0] == 1, f"the climb does not start at 1: {positions}"
assert all(m == N_IMAGES for _, m in positions), positions
assert all(n <= N_IMAGES for n, _ in positions), positions
ns = [n for n, _ in positions]
assert ns == sorted(ns), f"the climb is backwards somewhere: {positions}"
assert N_IMAGES in ns, f"the climb never reached the total: {positions}"
# The two positions COEXIST (distinct and both may show — "… (1/1)
# · chunk 19/20 · image 3/4"): every image-span label carries the
# chunk part with it (the last chunk position persists through the
# file's image span — the per-file reset + the phase transitions
# clear it, not the end of the chunk span).
for i, _ in image_items:
cm = CHUNK_PART_RE.search(seen[i])
assert cm is not None, f"an image-span label lost the chunk part: {seen[i]}"
assert int(cm.group(2)) == n_chunks, (cm.group(0), n_chunks)
# --- the settle: the counts line, the terminal label, the parts
# cleared + empty, and the IMAGE part GONE (no stale position in a
# terminal state — the endpoint pin above is the server-side
# half) ---
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
btn = page.locator("#sync-btn")
expect(btn).to_be_enabled()
expect(btn).not_to_have_attribute("aria-busy")
expect(page.locator(".sync-label-count")).to_be_hidden()
expect(page.locator(".sync-label-count")).to_have_text("")
expect(page.locator(".sync-label-chunk")).to_be_hidden()
expect(page.locator(".sync-label-chunk")).to_have_text("")
# The counts line rides the box (phase 150 — the sole sync text
# surface): the image part is not in it, not in the title, not in
# the aria (the settle restores the ship aria value).
box = page.locator("#sync-progress-text").inner_text()
assert box == SETTLE_TEXT, box
assert "· image" not in box and "· image" not in (btn.get_attribute("title") or "")
assert btn.get_attribute("aria-label") == "Sync sources", btn.get_attribute("aria-label")
expect(page.locator("#sync-error-banner")).to_be_hidden()
# ---------------------------------------------------------------------------
# 2. The never-stale contract — a mid-run page reload re-attaches with
# the current image position visible
# ---------------------------------------------------------------------------
def test_mid_run_reload_re_attaches_with_image_position(
page: Page, app_url: str, imgpass_dir: Path
) -> None:
"""The admin clicks **Sync sources** (the run is started the way
the owner starts it — the click is what starts the page's 2 s
poll; the reload is the contract under test), the page's own 2 s
poll renders the image span (the FIRST observed position is
``1/4`` — the whole span is ahead, so the reload + the load-time
re-attach happen with the span in flight), the test reloads the
page mid-run, and the load-time re-attach (``initSyncButton`` —
a running sync re-enters with the WHOLE status, phase 98's
never-stale contract) renders the CURRENT ``· image n/4`` with
the A4 title/aria; the run then settles to the counts line (the
image part gone — no stale position)."""
page.set_default_timeout(30_000)
_start_sync_click(page, app_url, imgpass_dir)
# The image span, rendered by the page's own 2 s poll (the span
# starts after the prelude + the file's embed batches — ≈ 21 s
# under the slow proxy; each position holds 4.5 s — the describe
# + the image doc's 2 embed batches).
snap = page.wait_for_function(
_JS_IMAGE_SNAP, arg=N_IMAGES, timeout=60_000
).json_value()
# The FIRST observed position is 1 — the whole span is ahead (the
# reload's re-attach is guaranteed to catch the span in flight:
# ≈ 18 s remain, the local reload + the load-time status fetch
# land well inside it).
assert snap["posN"] == 1, snap
# A4 — the pre-reload running state is untruncated on the title +
# the aria (the "· image" part included).
assert snap["title"] == snap["box"] and "· image" in snap["title"], snap
assert snap["aria"] == snap["box"], snap
# The mid-run reload (the session cookie survives — the re-attach
# is the admin's, phase 98).
page.reload()
# The load-time re-attach: a running sync re-enters with the WHOLE
# status — the CURRENT image position is visible (the never-stale
# contract), the A4 title/aria ride it.
snap2 = page.wait_for_function(
_JS_IMAGE_SNAP, arg=N_IMAGES, timeout=30_000
).json_value()
assert snap2["title"] == snap2["box"] and "· image" in snap2["title"], snap2
assert snap2["aria"] == snap2["box"], snap2
# Then the settle: the counts line (the image part gone — the
# terminal clears the position, test 1 pins the endpoint half),
# the terminal label, the parts cleared + empty.
expect(page.locator("#sync-progress-text")).to_have_text(
SETTLE_TEXT, timeout=SETTLE_TIMEOUT_MS
)
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
btn = page.locator("#sync-btn")
expect(btn).to_be_enabled()
expect(btn).not_to_have_attribute("aria-busy")
expect(page.locator(".sync-label-count")).to_be_hidden()
expect(page.locator(".sync-label-chunk")).to_be_hidden()
assert "· image" not in (btn.get_attribute("title") or "")
expect(page.locator("#sync-error-banner")).to_be_hidden()
terminal = _wait_terminal(app_url)
assert terminal["detail"]["added"] == 1 + N_IMAGES, terminal
assert terminal["images_done"] == 0 and terminal["images_total"] == 0, terminal
tests/integration/test_sync_api.py
+352 -1
View File
@@ -93,6 +93,7 @@ import asyncio
import logging
import os
import subprocess
import textwrap
import threading
import time
from collections.abc import Callable, Iterator, Sequence
@@ -100,6 +101,7 @@ from datetime import UTC, datetime
from pathlib import Path
from typing import Any
import pymupdf as fitz
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import select, text
@@ -309,6 +311,12 @@ class FakeImportSources:
# regresses) — the canned fake never fires it (the runner's
# status chunk fields stay 0/0, like a no-batch import).
self.chunk_hooks: list[object] = []
# Phase 152 (task 03): the image-position hook the runner
# passes (a live closure while wired, None if the wiring
# regresses) — the canned fake never fires it (the status
# image fields stay 0/0, like the chunk fields of a no-batch
# import).
self.image_hooks: list[object] = []
async def __call__(
self,
@@ -320,6 +328,7 @@ class FakeImportSources:
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -329,6 +338,7 @@ class FakeImportSources:
self.prune_flags.append(prune)
self.progress_hooks.append(progress)
self.chunk_hooks.append(on_chunk_progress)
self.image_hooks.append(on_image_progress)
self.ignore_maps.append(ignore_by_root or {})
self.include_hidden_maps.append(include_hidden_by_root or {})
self.doc_dates_maps.append(doc_dates_by_root or {})
@@ -500,6 +510,10 @@ def test_admin_sync_success_reports_full_detail(
# (the dataclass defaults).
"current_chunk": 0,
"chunks_total": 0,
# Phase 152 (task 01): the image-position keys — 0/0 idle
# (the dataclass defaults).
"images_done": 0,
"images_total": 0,
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
"phase": None,
"current_summary": None,
@@ -656,13 +670,23 @@ def test_double_trigger_while_running_returns_409(
class _GatedImport:
"""An import that fires the runner's progress hook once, then
parks on a threading gate — the test reads the status mid-import
(the phase ``"import"`` + the file hook)."""
(the phase ``"import"`` + the file hook).
Phase 152 (task 03): optionally ALSO fires the in-flight file's
image-position pairs (``image_pairs``) through the runner's OWN
``on_image_progress`` closure: pair one lands before the main park
(readable mid-run), every later pair after the main park + its own
park (``image_release``), so each position is readable on the
status between polls (the phase-129 chunk idiom, mirrored for the
vision-image position)."""
def __init__(
self,
summary: ImportSummary,
started: threading.Event,
release: threading.Event,
image_pairs: Sequence[tuple[int, int]] = (),
image_release: threading.Event | None = None,
) -> None:
self.summary = summary
self.started = started
@@ -672,6 +696,13 @@ class _GatedImport:
# passes (recorded for the wiring pin — the gated fake never
# fires it, so the status chunk fields stay 0/0).
self.chunk_hook: Callable[[int, int], None] | None = None
# Phase 152 (task 03): the image-position hook the runner
# passes (recorded for the wiring pin) + the optional image
# pairs (see the class docstring).
self.image_hook: Callable[[int, int], None] | None = None
self.image_pairs = list(image_pairs)
self.image_release = image_release
self.image_hook_calls: list[tuple[int, int]] = []
async def __call__(
self,
@@ -683,6 +714,7 @@ class _GatedImport:
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -691,6 +723,25 @@ class _GatedImport:
progress("repo", "notes/deep.md", 1, 3)
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
self.chunk_hook = on_chunk_progress
self.image_hook = on_image_progress
if on_image_progress is not None and self.image_pairs:
# Phase 152 (task 03): the in-flight PDF's image-position
# pairs, fired through the runner's OWN hook closure (the
# closure under test) — pair one before the main park, each
# later pair after the main park + its own park (each park
# lets the test read the status between polls). A wiring
# regression to None skips this branch and the pair test
# times out.
on_image_progress(*self.image_pairs[0])
self.image_hook_calls.append(self.image_pairs[0])
self.started.set()
await asyncio.to_thread(self.release.wait)
for pair in self.image_pairs[1:]:
on_image_progress(*pair)
self.image_hook_calls.append(pair)
if self.image_release is not None:
await asyncio.to_thread(self.image_release.wait)
return self.summary
self.started.set()
await asyncio.to_thread(self.release.wait)
return self.summary
@@ -867,6 +918,305 @@ def test_status_reports_the_phase_machine_across_the_run(
assert gated_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
def test_image_position_climbs_between_polls_mid_run(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""Phase 152 (task 03): while a PDF's vision span is in flight,
the status carries the in-flight image position — the pairs are
fired through the runner's OWN ``_image_hook`` closure (task 03's
wiring: ``import_sources(..., on_image_progress=_image_hook)``),
and the position CLIMBS between two polls of
``GET /api/sync/status`` (1/15 → 7/15 — monotonic, never above
the total, no backwards jumps — the owner's stuck label now
moves), paired with the file the ``progress`` hook named (the
hooks are paired — the position belongs to that file); the
success terminal clears the position to 0/0 while the FILE
counts keep their final values (the convention split — same as
the chunk position)."""
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
image_release = threading.Event()
gated_import = _GatedImport(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
threading.Event(),
image_pairs=[(1, 15), (7, 15)], # two describe calls of a 15-image span
image_release=image_release,
)
monkeypatch.setattr(sync_api, "import_sources", gated_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
# Idle: the image keys ride along as 0/0 (the dataclass defaults).
body = sync_client.get("/api/sync/status").json()
assert body["state"] == "idle"
assert body["images_done"] == 0 and body["images_total"] == 0
assert sync_client.post("/api/sync").status_code == 202
try:
assert gated_import.started.wait(5.0), "the run never reached the first image pair"
# Poll 1 — the import phase, pair one live: the position is
# 1/15, paired with the file the progress hook named; the
# chunk position is 0/0 (this canned file has no embed
# batches — the two positions are independent).
s = sync_client.get("/api/sync/status").json()
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 1 and s["images_total"] == 15
# Release pair one: the import fires pair two (7/15 — the
# NEXT describe call of the same 15-image span) and parks
# again.
gated_import.release.set()
deadline = time.monotonic() + 5.0
while time.monotonic() < deadline:
s = sync_client.get("/api/sync/status").json()
if s["images_done"] == 7:
break
assert s["state"] == "running", (
f"unexpected state {s['state']!r} while waiting for pair two: {s}"
)
time.sleep(0.05)
else:
pytest.fail("the second image pair never landed on the status")
# Poll 2 — the position CLIMBED (1 → 7, monotonic — no
# backwards jump, never above the total): the label is
# moving, not frozen.
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/notes/deep.md"
assert s["images_done"] == 7 and s["images_total"] == 15
assert 0 < s["images_done"] <= s["images_total"]
image_release.set()
finally:
gated_import.release.set()
image_release.set()
body = _poll(sync_client, "success")
# Terminal: the image position clears (0/0) ... the FILE counts
# KEEP their final values (the convention split — the position is
# a sub-position of a done file, like the chunk position).
assert body["images_done"] == 0 and body["images_total"] == 0
assert body["current_file"] is None
assert body["files_done"] == 1 and body["files_total"] == 3
# Wiring: BOTH pairs went through the runner's OWN live closure
# (the status reads above came through it — a wiring regression
# to None would have skipped the firing branch and timed out),
# and the file hook fired once.
assert gated_import.image_hook is not None
assert gated_import.image_hook_calls == [(1, 15), (7, 15)]
assert gated_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
# --- phase 152 (task 04): the full-run image-position monotonicity -------
def _kb_solid_png(value: int) -> bytes:
"""A 40x30 solid-color PNG (channel *value*): every value used
below stays under the phase-151 blank gate's 235 mean floor
(kept, never blank-skipped), and distinct values are distinct
images (the parser's dedupe keeps every one)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 40, 30), 0)
pix.clear_with(value)
return pix.tobytes("png")
def _kb_image_pdf(path: Path, values: Sequence[int]) -> None:
"""A REAL one-page PDF (the fixture KB's image-bearing file): a
text block comfortably over the phase-140 100-char sparseness
bar (a USABLE layer — the phase-133 gate stays closed, zero
pictorial candidates even with the toggle on — the "no other
vision span" class) carrying *values* the solid-color embedded
images (one placement each, distinct)."""
doc = fitz.open()
page = doc.new_page()
text = (
"The quadlet unit file defines the container interface, the "
"image, the network, and the restart policy. Every key maps "
"to a docker flag in the generated service file, and comments "
"use the usual semicolon form throughout the whole document."
)
for i, line in enumerate(textwrap.wrap(text, 60)):
page.insert_text((72, 72 + i * 14), line)
y = 300
for value in values:
page.insert_image(fitz.Rect(72, y, 112, y + 30), stream=_kb_solid_png(value))
y += 40
doc.save(path)
doc.close()
class _GatedVisionLLM(FakeEmbedder):
"""The phase-152 (task 04) mock LLM — the deterministic
in-process ``FakeEmbedder`` whose VISION calls (the multimodal
describe shape ``describe_image`` sends) PARK on a threading
gate: the run parks inside the PDF's embedded-image span, so the
test can read the status between the describe calls
(deterministic — no sleeps, no wall-clock dependence; the gate
holds the run until the test releases it). The hook fired BEFORE
the parked call (the task-02 contract), so the position is live
while the run is parked. Text (``lite``) calls keep the
``FakeEmbedder`` behaviour; the embeds are the deterministic
vectors."""
def __init__(self) -> None:
super().__init__()
self.vision_parks = 0 # the vision calls that parked (and released)
self.vision_release = threading.Event()
async def chat(
self, messages: list[dict[str, Any]], model: str | None = None
) -> str:
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
# One slow vision describe call — the gate (clear before
# the wait — a latching set would otherwise leak across
# calls): the test releases after reading the position.
self.vision_release.clear()
self.vision_parks += 1
await asyncio.to_thread(self.vision_release.wait)
return "A solid color swatch described." # deterministic, non-empty
return await super().chat(messages, model)
def test_image_position_monotonic_across_polls_over_a_real_pdf(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
clean_documents: None,
) -> None:
"""Phase 152 (task 04): a FULL sync run over a fixture KB with
ONE image-bearing PDF (the REAL ``import_sources`` — the
runner's ``_image_hook`` is driven by the importer's own
embedded-image span; the LLM is the deterministic mock whose
vision calls park on a gate) — polling ``GET /api/sync/status``
from the test while the run is parked between the describe
calls, the observed ``images_done`` sequence is MONOTONICALLY
NON-DECREASING, never above ``images_total``, and ends at
``images_total`` (no backwards jumps, no overshoot — the
owner's stuck label climbs one describe call at a time); the
position is paired with the file the ``progress`` hook named
(the hooks are paired), the success terminal clears it to 0/0
while the FILE counts keep their final values, and the KB
carries the PDF + its N image docs (the span really ran)."""
kb = tmp_path / "PictKB"
kb.mkdir()
(kb / "notes.md").write_text("# Notes\nthe local fixture\n", encoding="utf-8")
_kb_image_pdf(kb / "report.pdf", (90, 130, 170)) # 3 distinct images
n = 3
_seed_local(db, kb)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
clone_calls, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
llm = _GatedVisionLLM()
llm.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
import_extensions="md,txt,pdf",
images=True,
image_dir=str(tmp_path / "images"),
pdf_page_dir=str(tmp_path / "pages"),
)
monkeypatch.setattr(sync_api, "LLMClient", lambda: llm)
_stub_probe(monkeypatch) # the probe passes on the mock (no network)
_login(sync_client)
def status() -> dict:
return sync_client.get("/api/sync/status").json()
def wait_for(predicate, what: str, timeout: float = 10.0) -> None:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if predicate():
return
time.sleep(0.02)
raise AssertionError(f"timed out waiting for {what}")
# Idle: the image keys ride along as 0/0 (the dataclass
# defaults).
body = status()
assert body["state"] == "idle"
assert body["images_done"] == 0 and body["images_total"] == 0
assert sync_client.post("/api/sync").status_code == 202
try:
# The run reaches the PDF's vision span: the first hook fired
# (total = the span's N) and the first describe call parked
# the run (the gate).
wait_for(
lambda: status()["state"] == "running"
and status()["phase"] == "import"
and status()["images_total"] == n,
"the run to reach the PDF's vision span",
)
# Each describe call: the hook fired BEFORE the call, so the
# position is live while the run is parked in the call —
# observe it, release the call, let the next position climb.
observed: list[tuple[int, int]] = []
for expected in range(1, n + 1):
def position_ready(target: int = expected) -> bool:
# the default argument binds the loop variable (B023)
return status()["images_done"] == target
wait_for(position_ready, f"images_done to climb to {expected}")
s = status()
assert s["state"] == "running" and s["phase"] == "import"
# The position is paired with the file the progress hook
# named (the two hooks are paired — the position belongs
# to that file).
assert s["current_file"] == "PictKB/report.pdf"
observed.append((s["images_done"], s["images_total"]))
llm.vision_release.set()
# The run finishes the rest of the pipeline (the span is
# done — no more vision calls) and settles.
assert llm.vision_parks == n, "one park per describe call"
finally:
llm.vision_release.set()
body = _poll(sync_client, "success")
# The observed sequence: monotonically non-decreasing, never
# above the total, ending at the total (the gate made it exact —
# 1..N, no backwards jumps, no overshoot).
dones = [d for d, _ in observed]
assert dones == list(range(1, n + 1)), f"the sequence: {observed}"
assert all(d <= t for d, t in observed), "no overshoot"
assert dones[-1] == n, "ends at images_total"
assert {t for _, t in observed} == {n}, "one total for the whole span"
# Terminal: the image position clears (0/0) ... the FILE counts
# KEEP their final values (the convention split — the position
# is a sub-position of a done file, like the chunk position).
assert body["images_done"] == 0 and body["images_total"] == 0
assert body["current_file"] is None
assert body["files_done"] == 2 and body["files_total"] == 2
assert clone_calls == [] # nothing to clone — local is walked directly
# The KB carries the fixture md + the PDF + its N image docs —
# the span really ran (N new image documents, no failures).
assert body["detail"]["added"] == 2 + n
assert body["detail"]["errors"] == 0
docs = {(d["source"], d["path"])
for d in sync_client.get("/api/docs").json()["documents"]}
assert ("PictKB", "report.pdf") in docs
assert sum(1 for _, p in docs if "#page=" in p) == n
def test_failed_terminal_after_the_summary_hook_clears_phase_keeps_counts(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
@@ -1497,6 +1847,7 @@ def test_import_error_is_reported_with_credentials_masked(
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
tests/unit/test_frontend_sync_upload.py
+242 -37
View File
@@ -127,6 +127,28 @@ setSyncProgress and pins the result mode behaviorally (a settle
result line reveals the box, "" empties + hides it); the bare
settle's row now shows the box LEFT ALONE (the settle label rides
the button only — A2).
Phase 152 (task 03) adds the label's FOURTH part — the vision-image
position: fmtSyncPhaseLabel's import fall-through gains "· image
n/m" (rendered while images_total > 0 — a PDF's vision span is in
flight; n the 1-based image about to be described, m the file's
total images, ONE continuous counter across the PDF's three vision
spans), the overview/summaries branches return image: "" (the
position belongs to the import phase only — the status's image
fields are cleared at those transitions, so the branches never read
them), and the bare builder stays image-free (the upload label and
the 202/409 click stay bare). setSyncLabel(mainText, countText =
"", chunkText = "", running = false, imageText = "") appends the
image part LAST to the composed full string — so the box, the
button title, and the running aria-label all carry
"Syncing… <file> (2/3) · chunk 14/38 · image 3/15". The image part
gets NO span of its own (the running button is the spinner — its
parts are hidden, phase 149): it rides ONLY the composed string,
while the chunk part keeps its .sync-label-chunk span; the two
positions are distinct and each disappears independently. The node
harness gains the two in-flight image rows (both parts at once —
the owner's stuck label — and the image-only pictorial PDF) and
pins the composed full string byte-for-byte.
"""
from __future__ import annotations
@@ -298,8 +320,9 @@ def test_fmt_sync_label_returns_the_parts_contract() -> None:
def test_set_sync_label_is_the_single_parts_writer() -> None:
"""setSyncLabel(mainText, countText = "", chunkText = "",
running = false) — Phase 128 (task 01) + Phase 129 (task 04):
the SINGLE point that writes #sync-label. The MAIN part gets
running = false, imageText = "") — Phase 128 (task 01) + Phase
129 (task 04) + Phase 152 (task 03): the SINGLE point that writes
#sync-label. The MAIN part gets
mainText (textContent only — the house rule, no HTML strings);
the COUNT part gets countText when present (textContent +
hidden=false — the "(n/m)" position is visible, NEVER clipped)
@@ -309,11 +332,18 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
hidden=true — the part renders only while a file's embed batch
is in flight). Then the UNTRUNCATED full label — mainText +
(countText ? " " + countText : "") + (chunkText ? " " + chunkText
: "") — rides the button title (A4 — the screen reader hears the
whole "Syncing… <file> (2/3) · chunk 14/38" there, and in the
box — see below). Phase 149 (task 01): the fourth parameter
: "") + (imageText ? " " + imageText : "") — rides the button
title (A4 — the screen reader hears the
whole "Syncing… <file> (2/3) · chunk 14/38 · image 3/15" there,
and in the box — see below). The image part (phase 152) gets NO
span of its own (the running button is the spinner — its parts
are hidden, phase 149): it rides only the composed full string.
Phase 149 (task 01): the fourth parameter
`running` (the in-flight flag — the running entry passes true,
the settle the default false). Phase 150 (task 02): while
the settle the default false); phase 152 (task 03) adds the
fifth — `imageText` (the bare builders and the non-running calls
pass nothing, so it defaults "" and the part stays out of the
string). Phase 150 (task 02): while
running, the SAME full label goes into the detail box
(#sync-progress — the SOLE aria-live region, the announcer the
screen reader hears) through the single box writer
@@ -324,10 +354,20 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
running / the ship value "Sync sources" otherwise
(overwrites/restores only — never removes)."""
body = _fn(_js(), "setSyncLabel")
assert (
'function setSyncLabel(mainText, countText = "", chunkText = "", running = false)'
in body
), "phase 149: the in-flight flag is the fourth parameter"
# the exact signature, byte-for-byte (concatenated — the literal
# is 105 columns, over the line cap): main, count, chunk, the
# phase-149 `running` flag (fourth), the phase-152 `imageText`
# (fifth — the bare builders + non-running calls pass nothing,
# so it defaults to the empty string).
expected_sig = (
'function setSyncLabel(mainText, countText = "", chunkText = "",'
' running = false, imageText = "")'
)
assert expected_sig in body, (
"phase 149: the in-flight flag is the fourth parameter; phase"
" 152: the image part is the fifth, defaulting to the empty"
" string (the part stays out of the composed full string)"
)
# the parts: textContent only (the house rule), null-guarded refs
assert "syncLabelMain.textContent = mainText" in body
assert "syncLabelCount.textContent = countText" in body
@@ -353,8 +393,12 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
"const full =\n"
" mainText +\n"
" (countText ? ` ${countText}` : \"\") +\n"
" (chunkText ? ` ${chunkText}` : \"\");"
) in body, ("the full label is main + ' ' + count + ' ' + chunk (spaces = the CSS gaps)")
" (chunkText ? ` ${chunkText}` : \"\") +\n"
" (imageText ? ` ${imageText}` : \"\");"
) in body, (
"the full label is main + ' ' + count + ' ' + chunk + ' ' +"
" image — the image part LAST (spaces = the CSS gaps)"
)
i_full = body.find("const full")
i_title = body.find("syncBtn.title = full")
i_box = body.find("if (running) setSyncProgress(full);")
@@ -375,7 +419,8 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
"""fmtSyncPhaseLabel(status) — the SYNC job's phase-aware label
(phase 98 task 02, D2; phase 128 task 01: the PARTS). "overview"
(phase 98 task 02, D2; phase 128 task 01: the PARTS; phase 152
task 03: the IMAGE part). "overview"
→ main exactly "Writing KB overview…" (no folder, no count part);
"summaries" → main "Summarizing folders… <current_summary>" +
count "(n/m)" — the folder part (one leading space +
@@ -385,7 +430,14 @@ def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
a total); anything else (the null prelude, "import") → the
phase-64 fmtSyncLabel parts, byte-identical (same builder, same
fields, same order). The UPLOAD job never enters here — its status
has no phase (D2)."""
has no phase (D2). Phase 152 (task 03): the import fall-through
is the ONLY branch that reads the status's image fields — "·
image n/m" renders while images_total > 0 (a PDF's vision span is
in flight — n the 1-based image about to be described, m the
file's total images) and hides at 0/0; the overview/summaries
branches return image: "" and never read the fields (the position
belongs to the import phase only — the runner clears it at those
transitions)."""
body = _fn(_js(), "fmtSyncPhaseLabel")
assert "function fmtSyncPhaseLabel(status)" in body
# "overview": the exact copy, no folder, no counts.
@@ -421,14 +473,30 @@ def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
)
assert "chunks_total" not in ov, "the overview branch never reads the chunk fields"
assert "chunks_total" not in sum_, "the summaries branch never reads the chunk fields"
# phase 152 (task 03): the overview/summaries labels carry NO
# image part either (the position belongs to the import phase
# only) and never read the status's image fields (the runner
# clears them at those transitions).
assert 'image: ""' in ov, "the overview label carries no image part"
assert 'image: ""' in sum_, (
"the summaries label carries no image part (its (n/m) is its position)"
)
assert "images_total" not in ov, "the overview branch never reads the image fields"
assert "images_total" not in sum_, "the summaries branch never reads the image fields"
# anything else (null prelude, "import"): byte-identical to today's
# sync label — same builder, same fields, same order — plus the
# phase-129 chunk part: "· chunk n/m" ONLY while chunks_total > 0
# (a file's embed batch is in flight — n the 1-based first chunk
# of the batch, m the file's total content chunks, batch
# granularity), "" otherwise (0/0 before the first batch — the
# part hides). The fall-through is the ONLY branch that reads the
# status's chunk fields.
# part hides) + the phase-152 image part: "· image n/m" ONLY
# while images_total > 0 (a PDF's vision span is in flight — n
# the 1-based image about to be described, m the file's total
# images — ONE continuous counter across the PDF's three vision
# spans), "" otherwise (0/0 — the part hides). The fall-through
# is the ONLY branch that reads the status's chunk + image
# fields, and the two parts are independent (both may show,
# each disappears on its own 0/0).
i_destr = body.find("const { main, count } = fmtSyncLabel(")
assert -1 < i_destr < i_fall, "the fall-through destructures the phase-64 parts"
fall = body[i_destr:]
@@ -444,11 +512,22 @@ def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
i_guard = fall.find("status.chunks_total > 0")
i_else = fall.find(": \"\";", i_guard)
assert -1 < i_guard < i_else, "0/0 (before the first batch) → the empty chunk part"
assert "return { main, count, chunk }" in fall
# phase 152 (task 03): the image part — "· image n/m" ONLY while
# images_total > 0, the exact template + the 0/0 → "" guard, and
# the return gains the fourth part.
assert "status.images_total > 0" in fall, (
"the image part renders only while a vision span is in flight"
)
assert "`· image ${status.images_done}/${status.images_total}`" in fall
i_iguard = fall.find("status.images_total > 0")
i_ielse = fall.find(": \"\";", i_iguard)
assert -1 < i_iguard < i_ielse, "0/0 (before the first image) → the empty image part"
assert "return { main, count, chunk, image }" in fall
# the bare builder (the upload job + the 202/409 click) returns NO
# chunk part — those labels stay bare.
# chunk OR image part — those labels stay bare.
bare = _fn(_js(), "fmtSyncLabel")
assert "chunk" not in bare, "fmtSyncLabel stays chunk-free (upload + the click stay bare)"
assert "image" not in bare, "fmtSyncLabel stays image-free (upload + the click stay bare)"
def test_enter_running_state_is_phase_aware_and_writes_the_parts() -> None:
@@ -460,13 +539,16 @@ def test_enter_running_state_is_phase_aware_and_writes_the_parts() -> None:
endpoint has no phase) and the 202/409 click keep the bare
fmtSyncLabel parts, byte-identical. Phase 128 (task 01): the entry
itself no longer writes the label — it hands the parts to the
SINGLE writer (setSyncLabel(main, count, chunk) — phase 129:
the chunk part destructures with a "" default), which lands the
SINGLE writer (setSyncLabel(main, count, chunk, running, image) —
phase 129: the chunk part destructures with a "" default, phase
152: the image part does too), which lands the
main part, the pinned count part (hidden when absent), the pinned
chunk part (phase 129 — hidden when absent), and the FULL
untruncated label on the button title + #sync-progress (A4 — set
for EVERY running label; CSS ellipsizes the main part only).
Phase 150 (task 02): the box write rides in the writer too
for EVERY running label; CSS ellipsizes the main part only —
phase 152: with the image part appended last when a PDF's vision
span is in flight; it gets no span of its own). Phase 150 (task
02): the box write rides in the writer too
(setSyncLabel → setSyncProgress, running mode)."""
body = _fn(_js(), "enterSyncRunningState")
assert (
@@ -487,13 +569,16 @@ def test_enter_running_state_is_phase_aware_and_writes_the_parts() -> None:
# the parts are handed to the single writer (A4 lives in it) —
# phase 129: the chunk part destructures with a "" default (the
# bare builders — the upload job, the 202/409 click — return no
# chunk part, so the part stays hidden there); phase 149: the
# chunk part, so the part stays hidden there); phase 152: the
# image part does too (it rides only the composed full string —
# no span); phase 149: the
# call passes `running = true` (the detail box reveals + the
# full-label aria-label ride in the writer).
assert "const { main, count, chunk = \"\" } =" in body, (
"the chunk part destructures with the '' default (bare builders return none)"
assert 'const { main, count, chunk = "", image = "" } =' in body, (
"the chunk + image parts destructure with the '' defaults"
" (bare builders return none)"
)
i_set = body.find("setSyncLabel(main, count, chunk, true)")
i_set = body.find("setSyncLabel(main, count, chunk, true, image)")
assert -1 < i_fall < i_set, "the parts are written after the split (running = true)"
assert body.count("setSyncLabel(") == 1, "one writer call per entry"
assert "syncBtn.title =" not in body, ("the title write lives in setSyncLabel (A4)")
@@ -713,6 +798,25 @@ enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
current_chunk: 14, chunks_total: 38,
});
show("import-inflight");
// 1b. Phase 152 (task 03): import, a PDF's vision span in flight
// (the owner's stuck label — the last chunk position 125/126 is
// STILL on the status + the climbing image position 3/15): the
// composed full string carries BOTH parts (chunk keeps its span,
// the image part rides the string only — no span):
enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 3,
current_chunk: 125, chunks_total: 126, images_done: 3, images_total: 15,
});
show("import-image-span");
// 1c. Phase 152 (task 03): import, a purely pictorial PDF (no text
// chunks — the chunk position never left 0/0) in its vision
// span: the image part shows ALONE (the chunk part stays hidden
// + empty — the two parts are independent):
enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 3,
current_chunk: 0, chunks_total: 0, images_done: 7, images_total: 7,
});
show("import-image-only");
// 2. import before the first batch (0/0) — the chunk part hides:
enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 3,
@@ -893,6 +997,96 @@ test-document-viewer pattern). The pinned "· chunk n/m" part (the
)
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_image_part_renders_only_in_the_import_phase_under_node() -> None:
"""Phase 152 (task 03): behavioral check (node) of the REAL label
path — the extracted fmtSyncPhaseLabel / setSyncLabel /
enterSyncRunningState run against the stub element refs. The
pinned "· image n/m" part (the task's fixture values):
* import, a PDF's vision span in flight (images_total 15 > 0)
WITH the last chunk position still on the status (125/126):
the composed full string carries BOTH parts — the box, the
button title, and the running aria-label all read exactly
"Syncing… src/a/b.md (2/3) · chunk 125/126 · image 3/15"
(the owner's stuck label, now moving); the chunk part KEEPS
its span ("· chunk 125/126" visible) and the image part gets
NO span of its own (it rides the composed string only — the
running button is the spinner, phase 149);
* import, a purely pictorial PDF (the chunk position never left
0/0) in its vision span: the image part shows ALONE —
"Syncing… src/a/b.md (2/3) · image 7/7" — and the chunk part
stays hidden + empty (the two parts are distinct and each
disappears independently on its own 0/0);
* every OTHER running state (the pre-batch import, the prelude,
the 202/409 click, the upload run, the overview, the
summaries — the status's image fields are 0/0 or cleared at
those transitions) leaves the composed string image-free.
The box / Cancel-control / is-running contracts of the running
state apply to the new rows exactly as to every other running
state (the harness's other pins cover them per tag)."""
rows = _chunk_harness_rows()
by_tag = {row[0]: row for row in rows}
# row = [tag, title, announcer (the box's text — the sole live
# region, phase 150), main, count, countHidden, chunk,
# chunkHidden, cancelHidden, cancelDisabled, progressText,
# progressHidden, btnRunning, iconSpinning, ariaLabel]
# 1b. both parts at once — the composed full string, byte-for-byte
# (the image part LAST, after the chunk part), on the box, the
# title, and the running aria-label (A4) — the chunk span
# keeps its own part.
full = "Syncing… src/a/b.md (2/3) · chunk 125/126 · image 3/15"
row = by_tag["import-image-span"]
assert row[1] == full, row[1]
assert row[2] == full, "the announcer (the box) hears the whole thing (A4)"
assert row[14] == full, "the running button's accessible name is the full label"
assert row[10] == full, "the detail box carries the EXACT full label"
assert row[6] == "· chunk 125/126" and row[7] is False, (
"the chunk part keeps its own span alongside the image part"
)
# 1c. the image part alone (the chunk position never left 0/0 —
# the two parts are independent).
row = by_tag["import-image-only"]
assert row[1] == "Syncing… src/a/b.md (2/3) · image 7/7", row[1]
assert row[2] == row[1] and row[14] == row[1], row
assert row[6] == "" and row[7] is True, (
"the chunk part stays hidden + empty (its own 0/0)"
)
# The pre-batch import (0/0 on BOTH positions) and every other
# running state: the composed string is image-free (the part
# hides on its own 0/0 — the overview/summaries transitions
# clear the position server-side, the bare builders carry none).
for tag, expected_title in (
("import-inflight", "Syncing… src/a/b.md (2/3) · chunk 14/38"),
("import-prebatch", "Syncing… src/a/b.md (2/3)"),
("prelude", "Syncing…"),
("click", "Syncing…"),
("upload", "Importing"),
("overview", "Writing KB overview…"),
("summaries", "Summarizing folders… src/alpha (2/3)"),
):
row = by_tag[tag]
assert row[1] == expected_title, (tag, row)
assert "· image" not in row[1], (
f"{tag}: the image part shows only while images_total > 0"
)
# The running-state contracts apply to the new rows too: box
# visible + is-running + spinning, the Cancel control shown +
# enabled (a sync run in flight).
for tag in ("import-image-span", "import-image-only"):
row = by_tag[tag]
assert row[11] is False, f"{tag}: the box must be visible while in flight"
assert row[12] is True and row[13] is True, f"{tag}: is-running + spinning"
assert row[8] is False and row[9] is False, (
f"{tag}: the Cancel control is shown + enabled while a sync runs"
)
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_progress_box_and_running_button_contract_under_node() -> None:
"""Phase 149 (task 01) — the detail box + the icon-only running
@@ -995,14 +1189,16 @@ def test_progress_box_and_running_button_contract_under_node() -> None:
def test_every_other_state_passes_no_chunk_text() -> None:
"""Source-level pin of the render-only-while-chunks_total>0
contract: the import fall-through is the ONLY place in sources.js
that reads the status's chunk fields (the summaries/overview
branches and the bare builders never touch them), and every
NON-running label emission passes no chunk text (the part hides):
the settle/idle labels go through setSyncLabel(label) — the only
running-flagged call site (phase 149: `running = true`) is the
running entry."""
"""Source-level pin of the render-only-while-chunks_total>0 +
render-only-while-images_total>0 contracts: the import
fall-through is the ONLY place in sources.js that reads the
status's chunk OR image fields (the summaries/overview branches
and the bare builders never touch them), and every
NON-running label emission passes no chunk/image text (both
parts hide): the settle/idle labels go through setSyncLabel(label)
— the only running-flagged call site (phase 149: `running = true`)
is the running entry (phase 152: passing the image part as its
fifth argument)."""
js = _js()
body = _fn(js, "fmtSyncPhaseLabel")
i_fall = body.find("fmtSyncLabel(")
@@ -1010,12 +1206,21 @@ def test_every_other_state_passes_no_chunk_text() -> None:
assert "chunks_total" not in pre and "current_chunk" not in pre, (
"the overview/summaries branches never read the chunk fields"
)
assert "images_total" not in pre and "images_done" not in pre, (
"the overview/summaries branches never read the image fields (phase 152)"
)
fall = body[i_fall:]
assert "status.chunks_total" in fall and "status.current_chunk" in fall
assert js.count("setSyncLabel(main, count, chunk, true)") == 1, (
"the ONLY running-flagged call site is the running entry (phase 149)"
assert "status.images_total" in fall and "status.images_done" in fall, (
"the import fall-through is the ONLY reader of the image fields (phase 152)"
)
assert js.count("setSyncLabel(main, count, chunk, true, image)") == 1, (
"the ONLY running-flagged call site is the running entry (phase 149"
" — phase 152: with the image part)"
)
assert "setSyncLabel(label)" in js, (
"the settle/idle labels stay two-arg (no chunk/image)"
)
assert "setSyncLabel(label)" in js, "the settle/idle labels stay two-arg (no chunk)"
# ---------- the two-job tick (startSyncPolling) ----------
tests/unit/test_importer.py
+508 -1
View File
@@ -18,7 +18,8 @@ import base64
import hashlib
import logging
import os
from collections.abc import Callable
import textwrap
from collections.abc import Callable, Sequence
from datetime import UTC, datetime
from pathlib import Path
@@ -41,6 +42,11 @@ from app.rag.importer import (
match_extension,
)
from app.rag.llm import EmbeddingError, LLMError
from app.rag.pdf import (
extract_pdf_images_with_stats,
extract_pdf_page_texts,
pictorial_pages,
)
from tests.fakes import FakeEmbedder
#: The original seven A9 formats as dotted suffixes (pre-phase-47 default
@@ -2088,3 +2094,504 @@ def test_fallback_embedded_images_still_join_the_pipeline(
assert image_doc.content != d1 and image_doc.content != d2
finally:
_cleanup_source(db, "both")
# ---------- phase 152 (task 02): on_image_progress plumbing ----------
#: A text block comfortably over the phase-140 100-char sparseness bar
#: (a USABLE layer, zero pictorial candidates even with the toggle on
#: — the "no vision span" class). Inserted wrapped at 60 chars: a
#: single long line would be CLIPPED at the page edge on extraction,
#: while the wrapped multi-line read round-trips (the phase-140
#: fixture's class property).
_P152_DENSE_TEXT = (
"The quadlet unit file defines the container interface, the image, "
"the network, and the restart policy. Every key maps to a docker "
"flag in the generated service file, and comments use the usual "
"semicolon form throughout the whole document."
)
assert len(_P152_DENSE_TEXT.strip()) > 100 # the class property, not a typo
def _p152_solid_png(value: int) -> bytes:
"""A 40x30 solid-color PNG (channel *value*): every value used
below stays under the phase-151 blank gate's 235 mean floor (kept,
never blank-skipped), and distinct values are distinct images (the
dedupe keeps every one)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 40, 30), 0)
pix.clear_with(value)
return pix.tobytes("png")
def _build_image_pdf(
path: Path,
values: Sequence[int],
dense: bool = True,
repeat_first: bool = False,
) -> None:
"""A REAL one-page PDF: *dense* a text block over the 100-char bar
(a USABLE layer — the fallback never fires; zero pictorial
candidates even with the toggle on) carrying *values* the
solid-color embedded images (one placement each, distinct).
``dense=False`` inserts the sparse ``BESTA`` marker instead (a
pictorial candidate under the default 100-char bar).
``repeat_first`` adds a SECOND placement of the first image (the
same stream — one xref, the parser's repeated-placement class,
phase 151: the dedupe drops it, the position count is
post-dedupe)."""
doc = fitz.open()
page = doc.new_page()
text = _P152_DENSE_TEXT if dense else "BESTA"
for i, line in enumerate(textwrap.wrap(text, 60)):
page.insert_text((72, 72 + i * 14), line)
y = 300
for value in values:
page.insert_image(fitz.Rect(72, y, 112, y + 30), stream=_p152_solid_png(value))
y += 40
if repeat_first:
page.insert_image(
fitz.Rect(150, 300, 190, 330), stream=_p152_solid_png(values[0])
)
doc.save(path)
doc.close()
def _build_pictorial_image_pdf(path: Path, values: Sequence[int]) -> None:
"""A REAL two-page PDF — the ONE continuous counter's two-span
case: page 1 is SPARSE (``BESTA`` — a pictorial candidate under
the default 100-char bar), page 2 is dense (never a candidate) and
carries *values* the solid-color embedded images. Both the
phase-140 pictorial pass (before the document commit) and the
phase-126 embedded-image pass (after it) fire on it."""
doc = fitz.open()
doc.new_page().insert_text((72, 72), "BESTA")
page = doc.new_page()
for i, line in enumerate(textwrap.wrap(_P152_DENSE_TEXT, 60)):
page.insert_text((72, 72 + i * 14), line)
y = 300
for value in values:
page.insert_image(fitz.Rect(72, y, 112, y + 30), stream=_p152_solid_png(value))
y += 40
doc.save(path)
doc.close()
def _build_garbage_image_pdf(path: Path, values: Sequence[int]) -> None:
"""A REAL two-page garbage-text PDF (every page XML-ish — the
phase-133 gate fires, the page-screenshot fallback runs) with
*values* the solid-color embedded images on page 2 — the fallback
span + the embedded span, ONE continuous counter."""
doc = fitz.open()
for p in range(1, 3):
page = doc.new_page()
for i in range(10):
page.insert_text((72, 72 + i * 14), f"<foo:p{p}>{i}</foo:p{p}>")
y = 400
for value in values:
doc[1].insert_image(
fitz.Rect(72, y, 112, y + 30), stream=_p152_solid_png(value)
)
y += 40
doc.save(path)
doc.close()
def _p152_summary_fields(s: ImportSummary) -> tuple:
"""One run's observable outcome (the byte-identity pin's left
side): every count a hook could plausibly move + the changed
paths' RELS (the source names are equal by construction — the two
roots carry the same directory name)."""
return (
s.files, s.added, s.updated, s.unchanged, s.pruned, s.errors,
s.chunks, s.embed_batches, s.summaries, s.summary_errors,
s.images_failed, s.pdfs_screenshotted, s.pdfs_pictorial,
s.pdf_pictorial_pages, s.pdf_images_deduped, s.pdf_images_blank_skipped,
tuple(rel for _, rel in s.changed_paths),
)
def _p152_rows(db, source: str) -> list[tuple]:
"""A source's document rows (the byte-identity pin's right side),
identity-blind: the fields a hook could plausibly move."""
docs = db.scalars(select(Document).where(Document.source == source)).all()
return sorted(
(d.path, d.title, d.content, d.content_hash, d.is_image, d.summary,
d.image_context)
for d in docs
)
def test_on_image_progress_embedded_pass_climbs_one_per_image(
db, tmp_path: Path
) -> None:
"""The phase-152 core pin: a PDF with N DISTINCT embedded images
(one placement each — post-dedupe, post-blank-gate: the work
actually about to happen) fires the hook ONCE per image, 1-based,
with the span's total, at the START of each image's pipeline step:
``(1, N) … (N, N)``. A repeated placement of the first image (the
parser's dedupe, phase 151) adds NO position (the total is the
post-dedupe count), a clean usable text layer burns no fallback or
pictorial span (the embedded pass's totals stand alone), and the
phase-147 fast path on a re-sync (raw bytes unchanged) burns no
vision call and fires the hook never."""
root = tmp_path / "imgprog"
root.mkdir()
values = (90, 130, 170, 210) # 4 distinct images, all under the blank floor
_build_image_pdf(root / "report.pdf", values, repeat_first=True)
n = len(values)
llm = _PageVisionLLM()
llm.settings = _pdf_settings(tmp_path, images=True)
events: list[tuple[int, int]] = []
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
# The PDF + its N image docs; the repeat placement deduped.
assert (summary.files, summary.added) == (1, 1 + n)
assert summary.pdf_images_deduped == 1
assert (summary.images_failed, summary.pdfs_screenshotted,
summary.pdfs_pictorial) == (0, 0, 0)
# The embedded pass is the ONLY vision span: N slow calls.
assert llm.vision_calls == n
# The core pin: once per image, 1-based, the span's total.
assert events == [(i, n) for i in range(1, n + 1)]
# The phase-147 fast path: a re-sync (bytes unchanged) burns no
# vision call and fires the hook never — the events list grows
# not at all.
summary2 = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
assert summary2.unchanged == 1
assert llm.vision_calls == n
assert len(events) == n
finally:
_cleanup_source(db, root.name)
def test_on_image_progress_pictorial_then_embedded_is_one_counter(
db, tmp_path: Path
) -> None:
"""ASSUMPTION A3: the spans of ONE file are ONE continuous counter
— the embedded pass starts where the pictorial pass ended (``base
= images_done_so_far``): the positions climb ``1…C+N`` with NO
restart and never above the combined total (the pictorial span's
total is its candidate-page count, the embedded span's total the
combined count)."""
root = tmp_path / "imgprog-two"
root.mkdir()
values = (90, 130, 170)
_build_pictorial_image_pdf(root / "ikea.pdf", values)
# The span inputs, computed IN the test (no hardcoded lists — the
# phase-140 idiom): the deterministic gate's candidates and the
# post-dedupe/post-gate image count.
pdf = root / "ikea.pdf"
page_texts = extract_pdf_page_texts(pdf)
n_cand = len(pictorial_pages(page_texts, min_chars=100))
images, _ = extract_pdf_images_with_stats(pdf)
n_img = len(images)
assert (n_cand, n_img) == (1, len(values))
llm = _PageVisionLLM()
llm.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
import_extensions="md,txt,pdf",
images=True,
image_dir=str(tmp_path / "images"),
pdf_pictorial=True, # default on — the pass runs on the sparse page
)
events: list[tuple[int, int]] = []
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
assert (summary.files, summary.added) == (1, 1 + n_img)
assert (summary.pdfs_pictorial, summary.pdf_pictorial_pages) == (1, 1)
# The continuous counter: 1…C (total C) then C+1…C+N (total
# C+N) — no restart, never above the combined total.
assert events == (
[(i, n_cand) for i in range(1, n_cand + 1)]
+ [(n_cand + i, n_cand + n_img) for i in range(1, n_img + 1)]
)
finally:
_cleanup_source(db, root.name)
def test_on_image_progress_fallback_span_climbs_into_embedded(
db, tmp_path: Path
) -> None:
"""The owner's stuck-label class: a garbage-text PDF's per-page
screenshot describes (the phase-133 fallback span — the span that
froze the label for minutes) report the rendered-page count
(``1…M`` with total ``M``) and hand their reported count to the
embedded pass, which continues the SAME counter (``M+1`` with
total ``M+N``) — every slow call has a position before it."""
root = tmp_path / "imgprog-fb"
root.mkdir()
_build_garbage_image_pdf(root / "garbage.pdf", (90,))
llm = _PageVisionLLM()
llm.settings = _pdf_settings(tmp_path, images=True)
events: list[tuple[int, int]] = []
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
assert summary.pdfs_screenshotted == 1
assert (summary.files, summary.added, summary.images_failed) == (1, 2, 0)
# 2 fallback pages (total 2) + 1 embedded image continuing at
# 3 (total 3) — ONE counter across the two spans.
assert events == [(1, 2), (2, 2), (3, 3)]
# Every slow call (2 pages + 1 image) was preceded by a
# position — the label never sits frozen on a busy run.
assert llm.vision_calls == 3
finally:
_cleanup_source(db, root.name)
def test_on_image_progress_text_only_pdf_never_fires(
db, tmp_path: Path
) -> None:
"""A clean-text PDF with NO embedded images fires the hook NEVER,
with BOTH default-on toggles: the garbage gate stays closed (no
fallback span), the pictorial pass's deterministic gate finds zero
candidates (a no-op — the clean-text byte-identity class), and the
embedded pass's list is empty. The skipped spans report nothing
and advance the counter not at all (there is no vision call at
all)."""
root = tmp_path / "imgprog-none"
root.mkdir()
_build_image_pdf(root / "clean.pdf", ()) # dense text, zero images
llm = _PageVisionLLM()
llm.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
import_extensions="md,txt,pdf",
images=True,
image_dir=str(tmp_path / "images"),
pdf_pictorial=True, # default on — the pass runs and no-ops
)
events: list[tuple[int, int]] = []
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
assert (summary.files, summary.added) == (1, 1)
assert (summary.pdfs_screenshotted, summary.pdfs_pictorial) == (0, 0)
assert llm.vision_calls == 0 # no span ran — zero slow calls
assert events == []
finally:
_cleanup_source(db, root.name)
def test_on_image_progress_toggle_off_reports_nothing(
db, tmp_path: Path
) -> None:
"""The ``images`` toggle OFF skips the embedded span entirely (the
prune guard keeps any existing image docs) — a PDF that WOULD
carry N embedded images reports nothing (a skipped span advances
the counter not at all) and indexes as text only."""
root = tmp_path / "imgprog-off"
root.mkdir()
_build_image_pdf(root / "off.pdf", (90, 130))
llm = _PageVisionLLM()
llm.settings = _pdf_settings(tmp_path, images=False)
events: list[tuple[int, int]] = []
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
assert (summary.files, summary.added) == (1, 1)
assert llm.vision_calls == 0
assert events == []
finally:
_cleanup_source(db, root.name)
def test_on_image_progress_none_run_is_byte_identical(
db, tmp_path: Path
) -> None:
"""The regression pin: a hook-less run (the CLI/chat default —
``on_image_progress=None``, every call site a guarded no-op) lands
byte-identical to a run WITH the hook: the same documents (path,
title, content, digest, image flag, summary, context) and the same
summary counters. The hook observes; it never changes the run."""
# Both roots carry the SAME directory name — the source (the
# walk's root.name) is the document's provenance and part of the
# stored image context, so the comparison needs it equal.
root_a = tmp_path / "a" / "same_src"
root_a.mkdir(parents=True)
root_b = tmp_path / "b" / "same_src"
root_b.mkdir(parents=True)
values = (90, 130, 170, 210)
_build_image_pdf(root_a / "same.pdf", values)
_build_image_pdf(root_b / "same.pdf", values)
llm_a, llm_b = _PageVisionLLM(), _PageVisionLLM()
llm_a.settings = _pdf_settings(tmp_path, images=True)
llm_b.settings = _pdf_settings(tmp_path, images=True)
events: list[tuple[int, int]] = []
try:
# Run A (hook-less) first, snapshot its rows, clear the shared
# source name, then run B (hooked) — the rows line up 1:1.
summary_a = asyncio.run(import_sources([root_a], llm_a, session=db))
rows_a = _p152_rows(db, "same_src")
_cleanup_source(db, "same_src")
summary_b = asyncio.run(
import_sources(
[root_b], llm_b, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
rows_b = _p152_rows(db, "same_src")
# The hook ran in run B (N image positions, the span's total)
# — and changed nothing observable.
assert events == [(i, len(values)) for i in range(1, len(values) + 1)]
assert _p152_summary_fields(summary_a) == _p152_summary_fields(summary_b)
assert rows_a == rows_b
finally:
_cleanup_source(db, "same_src")
class _InterleavingVisionLLM(_PageVisionLLM):
"""A ``_PageVisionLLM`` variant that records each vision call in
a SHARED event sequence (phase 152, task 04): the importer fires
the ``on_image_progress`` hook BEFORE the
``_index_one_embedded_image`` describe seam, so the sequence reads
hook, vision, hook, vision, … — the interleaving pin (the label
moves at the START of each slow call, never after it)."""
def __init__(self, events: list[tuple]) -> None:
super().__init__()
self.events = events
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
reply = await super().chat(messages, model)
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
self.events.append(("vision", self.vision_calls))
return reply
def test_on_image_progress_hook_fires_before_each_describe_call(
db, tmp_path: Path
) -> None:
"""Phase 152 (task 04): the hook fires BEFORE the corresponding
describe call (the contract: at the START of each slow call, never
after it) — the interleaving pinned by recording events in the
fake: for a PDF with N embedded images the sequence reads
hook(1,N), vision(1), hook(2,N), vision(2), …, hook(N,N),
vision(N) (the fake's vision call IS the describe seam the hook
precedes — the label moves while the describe is about to start,
so a busy run never reads frozen)."""
root = tmp_path / "imgprog-order"
root.mkdir()
values = (90, 130, 170)
_build_image_pdf(root / "order.pdf", values)
n = len(values)
events: list[tuple] = []
llm = _InterleavingVisionLLM(events)
llm.settings = _pdf_settings(tmp_path, images=True)
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append(("image", pos, total)),
)
)
assert (summary.files, summary.added) == (1, 1 + n)
assert llm.vision_calls == n # one describe per image
# The interleaving: hook BEFORE each describe call, in order.
expected: list[tuple] = []
for i in range(1, n + 1):
expected.append(("image", i, n))
expected.append(("vision", i))
assert events == expected
finally:
_cleanup_source(db, root.name)
def test_on_image_progress_blank_gate_excludes_skipped_images_from_the_total(
db, tmp_path: Path
) -> None:
"""Phase 152 (task 04, the phase-151 interaction): images skipped
by the dedupe/blank gate are NOT counted in the hook's total —
the total is the work actually about to happen (the pass's
post-dedupe, post-gate list). A PDF with 2 real + 1 near-white
image (250 — the blank gate's class, mean ≥ 235): the gate
(default ON) drops the near-white one BEFORE the pass — the hook
fires for the 2 kept with total 2 (the skipped image burns no
vision call, no position); with the kill switch OFF
(``pdf_image_skip_blank=False`` — the pre-phase-151 behavior) the
near-white image is indexed too — the hook fires for all 3 with
total 3 (the total is the work about to happen under the knob's
state)."""
values_a = (90, 130, 250) # 250 = near-white (the blank gate's class)
values_b = (110, 150, 250) # distinct bytes — a different document set
# Both roots carry the SAME directory name — the source is part of
# the stored image context, so the runs line up 1:1 (the house
# cleanup idiom separates them).
root_a = tmp_path / "a" / "same_src"
root_a.mkdir(parents=True)
root_b = tmp_path / "b" / "same_src"
root_b.mkdir(parents=True)
_build_image_pdf(root_a / "blank.pdf", values_a)
_build_image_pdf(root_b / "blank.pdf", values_b)
llm_a = _PageVisionLLM()
llm_a.settings = _pdf_settings(tmp_path, images=True) # skip_blank default ON
events_a: list[tuple[int, int]] = []
try:
summary_a = asyncio.run(
import_sources(
[root_a], llm_a, session=db,
on_image_progress=lambda pos, total: events_a.append((pos, total)),
)
)
# The blank gate dropped the near-white image BEFORE the
# pass — the total is the 2 actually about to happen.
assert summary_a.pdf_images_blank_skipped == 1
assert (summary_a.files, summary_a.added) == (1, 3)
assert llm_a.vision_calls == 2
assert events_a == [(1, 2), (2, 2)]
# The knob OFF: the near-white image is indexed too — the
# total is all 3 (the pre-phase behavior, the work about to
# happen under the knob's state). Fresh source (the house
# cleanup idiom — run B's rows would otherwise be updates of
# run A's rows, same synthetic image paths).
_cleanup_source(db, "same_src")
llm_b = _PageVisionLLM()
llm_b.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
import_extensions="md,txt,pdf",
images=True,
image_dir=str(tmp_path / "images"),
pdf_pictorial=False,
pdf_image_skip_blank=False,
)
events_b: list[tuple[int, int]] = []
summary_b = asyncio.run(
import_sources(
[root_b], llm_b, session=db,
on_image_progress=lambda pos, total: events_b.append((pos, total)),
)
)
assert summary_b.pdf_images_blank_skipped == 0
assert (summary_b.files, summary_b.added) == (1, 4)
assert llm_b.vision_calls == 3
assert events_b == [(1, 3), (2, 3), (3, 3)]
finally:
_cleanup_source(db, "same_src")
tests/unit/test_sync_button.py
+567 -15
View File
@@ -110,6 +110,56 @@ poll tree's new 5th branch (after failed, before the upload branches
— the 7-branch tree is now 8), the neutral ``applySyncCancelled``
(no modal, no banner, no error styling), and the
``initSyncButton`` re-attach of a ``cancelled`` run.
Phase 152 (task 01): the status contract gains the in-flight file's
VISION-image position — ``images_done`` (the 1-based index of the
image about to be described) + ``images_total`` (the file's total
images for this run — the three PDF vision spans share ONE continuous
counter per file, the importer's ``on_image_progress`` hook that
task 02/03 wire). The idle dict pins them 0/0; the per-file reset is
the pairing rule, extended — a ``progress`` call names the file AND
clears the image position, pinned through the runner's own ``_hook``
closure (the two-file fake below: the position fired through the
runner's own ``_image_hook`` closure while file A is in flight is
0/0 when file B is named — a file with no images must not inherit
the previous PDF's position); the clear rule mirrors the chunk
position — 0/0 at the overview/summaries transitions and in EVERY
terminal (success, failed, and cancelled — each pinned with a live
position in flight; task 04 moves every live position below onto the
runner's OWN closures, the phase-64/129 idiom), while the FILE
counts keep their final values (the convention split: a
sub-position of a file that's done clears with the file).
Phase 152 (task 03): the runner's THIRD paired closure —
``_image_hook`` (beside ``_hook`` and ``_chunk_hook``) — lands the
importer's ``on_image_progress`` pairs on the status, and the import
call now passes ``on_image_progress=_image_hook`` (the wiring pin:
the fakes receive the kwarg and record the live closure). The
frontend gains the "· image n/m" label PART: the running entry
destructures ``image = ""`` and the writer call becomes
``setSyncLabel(main, count, chunk, true, image)`` — the image part
rides ONLY the composed full string (box + title + running
aria-label — no new button span; the running button is the
spinner), the chunk part keeps its own span, and the bare builders
+ non-running calls pass nothing (both parts default "").
Phase 152 (task 04): the position pins move onto the runner's OWN
closures (the phase-64/129 idiom — the fakes fire the image pairs
through the ``on_image_progress`` kwarg, the ``_image_hook`` under
test, instead of standing in on the status object): a mid-run pair
(3/15, then 15/15) climbs on the status BETWEEN ticks while
``current_file`` and the chunk position keep their own values (the
THREE positions are independent sub-positions of the file), the
per-file reset, the overview/summaries transition clears, and the
FAILED + CANCELLED terminals all fire their live positions through
the closure too. The frontend source pins the ``fmtSyncPhaseLabel``
image part (present iff ``images_total > 0`` — the exact guard +
template), the ``overview`` / ``summaries`` branches' ``image: ""``
(unchanged labels), and ``setSyncLabel``'s composed ``full`` string
byte-for-byte (a status carrying both positions reads "… (2540/2540)
· chunk 125/126 · image 3/15" on the box + title + running
aria-label — the runtime string is behaviorally pinned in
tests/unit/test_frontend_sync_upload.py's node check).
"""
from __future__ import annotations
@@ -393,13 +443,16 @@ def test_sources_js_running_state_is_never_stale() -> None:
Phase 128 (task 01): the entry hands the label PARTS (the
phase-aware split — sync + status → fmtSyncPhaseLabel, the
fall-through → the byte-identical fmtSyncLabel) to the SINGLE
writer, setSyncLabel(main, count, chunk — phase 129: the chunk
part destructures with a "" default), which lands the main part,
writer, setSyncLabel(main, count, chunk, running, image — phase
129: the chunk part destructures with a "" default, phase 152:
the image part does too), which lands the main part,
the pinned count part (hidden when absent), the pinned chunk part
(phase 129 — hidden when absent), and the FULL untruncated label
on the button title + #sync-progress (A4) — the direct span /
title / announcer writes are GONE from the entry (they live in
setSyncLabel + setSyncProgress). Phase 149 (task 01): the writer call passes
on the button title + #sync-progress (A4 — phase 152: with the
image part appended after the chunk part when a PDF's vision span
is in flight; the image part gets no span of its own) — the
direct span / title / announcer writes are GONE from the entry
(they live in setSyncLabel + setSyncProgress). Phase 149 (task 01): the writer call passes
`running = true` (the detail box reveals with the full label),
and the button's accessible name rides in the writer too (the
full live label — the icon-only running button keeps a name);
@@ -414,8 +467,10 @@ def test_sources_js_running_state_is_never_stale() -> None:
assert "fmtSyncLabel(kind, currentFile, done, total)" in body
# …and the single writer lands the parts (A4 in it — the title is
# set for EVERY running label: the file-only rule is adjusted
# away) with the phase-149 `running` flag (the box reveals).
assert "setSyncLabel(main, count, chunk, true)" in body
# away) with the phase-149 `running` flag (the box reveals) +
# the phase-152 image part (rides only the composed full string —
# no span).
assert "setSyncLabel(main, count, chunk, true, image)" in body
assert body.count("setSyncLabel(") == 1, "the entry writes ONLY through the parts writer"
assert "syncBtn.title =" not in body, ("the title write lives in setSyncLabel (A4)")
assert "syncLabel.textContent" not in body, ("no direct span write — the parts")
@@ -433,6 +488,70 @@ def test_sources_js_running_state_is_never_stale() -> None:
)
def test_fmt_sync_phase_label_image_part_is_pinned_source_level() -> None:
"""Phase 152 (task 04): the frontend SOURCE pins the "· image
n/m" label part (the browser behavior is pinned in
tests/unit/test_frontend_sync_upload.py's node check — the REAL
extracted functions run against stub refs; this pin keeps the
template itself from drifting without a browser):
* the import fall-through is the ONLY reader of the status's
image fields, guarded on ``images_total > 0`` (the part is
present IFF ``images_total > 0`` — 0/0 hides it, the exact
guard + template + empty false-branch);
* the ``overview`` / ``summaries`` branches return ``image: ""``
(the position belongs to the import phase only — those labels
stay unchanged by the phase);
* the running entry destructures the part with a `""` default and
hands it to the single writer as the FIFTH argument (after the
`running` flag);
* ``setSyncLabel``'s composed ``full`` string appends the image
part LAST (after the chunk part) — so a status carrying both
positions reads exactly "Syncing… <file> (2540/2540) · chunk
125/126 · image 3/15" on the box, the title, and the running
aria-label (byte-for-byte; the docstring's worked example
carries the shape).
"""
js = _text(SOURCES_JS)
builder = _body(js, "fmtSyncPhaseLabel")
# The import fall-through: present IFF images_total > 0 — the
# exact guard + template (the phase-129 chunk part's sibling,
# same shape).
assert (
"status.images_total > 0\n"
" ? `· image ${status.images_done}/${status.images_total}`\n"
" : \"\";"
) in builder
assert "return { main, count, chunk, image };" in builder
# The overview branch: the unchanged label + the part absent.
assert (
'return { main: "Writing KB overview…", count: "", chunk: "", image: "" };'
) in builder
# The summaries branch: the unchanged label + the part absent
# (the phase's own (n/m) is its position).
assert "image: \"\", // the image position belongs to the import phase only" in builder
assert "`Summarizing folders…${folder}`" in builder
# The running entry: the part destructures with a `""` default
# and rides the writer call as the fifth argument.
entry = _body(js, "enterSyncRunningState")
assert 'const { main, count, chunk = "", image = "" }' in entry
assert "setSyncLabel(main, count, chunk, true, image)" in entry
# The writer: the composed full string — the image part LAST,
# after the chunk part (the "… · chunk 125/126 · image 3/15"
# shape the docstring's worked example carries).
writer = _body(js, "setSyncLabel")
assert (
"const full =\n"
" mainText +\n"
" (countText ? ` ${countText}` : \"\") +\n"
" (chunkText ? ` ${chunkText}` : \"\") +\n"
" (imageText ? ` ${imageText}` : \"\");"
) in writer
assert "· chunk 125/126 · image 3/15" in js, (
"the worked example (both positions at once) is documented"
)
def test_sources_js_terminal_states() -> None:
"""Terminal rendering: success → enabled + 'Synced HH:MM' (local
time of finished_at); failed → enabled + retry-ready 'Sync sources'
@@ -927,7 +1046,20 @@ class _GatedImport:
``_chunk_hook`` closure: pair one lands before the main park
(readable mid-run), and every later pair lands after the main
park + its own park (``chunk_release``), so each position is
readable on the status between ticks."""
readable on the status between ticks.
Phase 152 (task 03): the runner now ALWAYS passes the
image-position closure (task 03's wiring) — recorded for the
wiring pins.
Phase 152 (task 04): optionally ALSO fires the in-flight file's
image-position pairs (``image_pairs``) through the runner's OWN
``_image_hook`` closure (pair one before the main park —
readable mid-run; every later pair after the main park + its own
park ``image_release`` — the phase-129 chunk idiom, mirrored for
the vision-image position). With no pairs the fake never fires
it (the status image fields stay 0/0, like the chunk fields of a
no-batch import)."""
def __init__(
self,
@@ -937,6 +1069,8 @@ class _GatedImport:
fail: BaseException | None = None,
chunk_pairs: Sequence[tuple[int, int]] = (),
chunk_release: threading.Event | None = None,
image_pairs: Sequence[tuple[int, int]] = (), # phase 152 (task 04)
image_release: threading.Event | None = None, # phase 152 (task 04)
) -> None:
self.summary = summary
self.started = started
@@ -947,6 +1081,12 @@ class _GatedImport:
self.chunk_pairs = list(chunk_pairs)
self.chunk_release = chunk_release
self.chunk_hook_calls: list[tuple[int, int]] = []
# Phase 152 (task 03): the image-position hook the runner
# passes (recorded for the wiring pins).
self.image_hook: Callable[[int, int], None] | None = None
self.image_pairs = list(image_pairs)
self.image_release = image_release
self.image_hook_calls: list[tuple[int, int]] = []
async def __call__(
self,
@@ -958,6 +1098,7 @@ class _GatedImport:
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -966,6 +1107,16 @@ class _GatedImport:
if progress is not None:
progress("repo", "notes/deep.md", 1, 3)
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
# Phase 152 (task 03): the runner now ALWAYS passes the
# image-position closure (task 03's wiring) — recorded for the
# wiring pins.
self.image_hook = on_image_progress
# Phase 152 (task 04): image pair one lands BEFORE the main
# park (readable mid-run — the phase-129 chunk-pair-one
# idiom, mirrored for the vision-image position).
if on_image_progress is not None and self.image_pairs:
on_image_progress(*self.image_pairs[0])
self.image_hook_calls.append(self.image_pairs[0])
if on_chunk_progress is not None and self.chunk_pairs:
# Phase 129 (task 03): the file's chunk-position pairs,
# fired through the runner's OWN hook closure (the closure
@@ -986,6 +1137,17 @@ class _GatedImport:
else:
self.started.set()
await asyncio.to_thread(self.release.wait) # park without freezing the worker loop
# Phase 152 (task 04): every later image pair lands after the
# main park + its own park (``image_release``), so each
# position is readable on the status between ticks. A wiring
# regression to None skips this branch and the pair tests
# time out.
if on_image_progress is not None:
for pair in self.image_pairs[1:]:
on_image_progress(*pair)
self.image_hook_calls.append(pair)
if self.image_release is not None:
await asyncio.to_thread(self.image_release.wait)
if self.fail is not None:
raise self.fail
return self.summary
@@ -1088,10 +1250,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
fresh_sync_status: None,
) -> None:
"""Idle: the three phase-64 progress keys ride along as null/0/0,
the two phase-129 chunk keys as 0/0, the four phase-98 phase keys
as null/0/0/0, and EVERY pre-existing key is unchanged — the full
response dict is pinned, so the current UI and every existing
consumer keep working."""
the two phase-129 chunk keys as 0/0, the two phase-152 image keys
as 0/0, the four phase-98 phase keys as null/0/0/0, and EVERY
pre-existing key is unchanged — the full response dict is pinned,
so the current UI and every existing consumer keep working."""
assert sync_api.sync_status() == {
"state": "idle",
"started_at": None,
@@ -1105,6 +1267,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
# (the dataclass defaults, no special-casing).
"current_chunk": 0,
"chunks_total": 0,
# Phase 152 (task 01): the image-position keys — 0/0 idle
# (the dataclass defaults, no special-casing).
"images_done": 0,
"images_total": 0,
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
"phase": None,
"current_summary": None,
@@ -1145,13 +1311,19 @@ def test_mid_run_status_reports_current_file(
assert s["files_done"] == 0 and s["files_total"] == 0
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
# Phase 152 (task 01): the image position is 0/0 in the
# prelude too (no vision span yet).
assert s["images_done"] == 0 and s["images_total"] == 0
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Import phase: the hook's file is live on the status, and the
# phase machine names the phase (the summary span has not
# started — its counters are still 0/0); no embed batch has
# fired the chunk hook (this canned import passes no pairs),
# so the chunk position stays 0/0.
# so the chunk position stays 0/0; the image position stays
# 0/0 too (this canned import fires no image hook — task 03
# wires the runner's image closure, the importer's span is
# what would feed it).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
@@ -1161,6 +1333,7 @@ def test_mid_run_status_reports_current_file(
assert s["current_summary"] is None
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 0 and s["images_total"] == 0
fake_import.release.set()
finally:
clone.release.set()
@@ -1178,7 +1351,8 @@ def test_mid_run_status_reports_current_file(
# folder step never fired the summary hook, so the summary counts
# stay 0/0 (the keep-final-counts convention keeps what fired);
# the chunk position clears with the file (phase 129 — it is a
# sub-position of a done file, never a keep-final counter).
# sub-position of a done file, never a keep-final counter), and
# so does the image position (phase 152 — same convention).
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
@@ -1187,6 +1361,7 @@ def test_mid_run_status_reports_current_file(
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["error"] is None
assert s["started_at"] is not None and s["finished_at"] is not None
@@ -1215,6 +1390,7 @@ def test_failed_terminal_clears_current_file_keeps_counts(
"failed: connection refused"
),
chunk_pairs=[(1, 38)], # phase 129: the live pair before the raise
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair before the raise
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
@@ -1231,6 +1407,13 @@ def test_failed_terminal_clears_current_file_keeps_counts(
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 1 and s["chunks_total"] == 38
# Phase 152 (task 04): a live image position in flight when
# the run dies — fired through the runner's OWN _image_hook
# closure (image_pairs[0] — the importer's span would do the
# same); the failed terminal must clear it (the
# sub-position of an abandoned file, like the chunk
# position).
assert s["images_done"] == 3 and s["images_total"] == 15
fake_import.release.set()
finally:
clone.release.set()
@@ -1238,8 +1421,9 @@ def test_failed_terminal_clears_current_file_keeps_counts(
thread.join(10.0)
assert not thread.is_alive()
assert errors == []
# Wiring: the pair fired through the runner's own closure.
# Wiring: the pairs fired through the runner's OWN closure.
assert fake_import.chunk_hook_calls == [(1, 38)]
assert fake_import.image_hook_calls == [(3, 15)]
s = sync_api.sync_status()
assert s["state"] == "failed"
assert s["current_file"] is None # cleared in the terminal state
@@ -1251,6 +1435,11 @@ def test_failed_terminal_clears_current_file_keeps_counts(
# terminal too — the convention split, pinned (the FILE counts
# above keep their final values, the sub-position does not).
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
# Phase 152 (task 04): the image position clears in the FAILED
# terminal too — the convention split, pinned (the live 3/15 fired
# through the closure mid-run is gone, the FILE counts keep their
# final values).
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["detail"] == {}
error = s["error"] or ""
assert "*****@aipi.example.com" in error # credentials masked
@@ -1468,6 +1657,357 @@ def test_chunk_position_clears_at_phase_transitions_and_terminal(
assert s["files_done"] == 1 and s["files_total"] == 3
# ---------- phase 152 (task 01 + task 04): the in-flight file's -----
# ---------- vision-image position (images_done / images_total) -------
#
# Same seams as the phase-129 block above. Task 04 pins every live
# position through the runner's OWN closures (the phase-64/129 idiom):
# the fakes fire the image pairs through the ``on_image_progress``
# kwarg — the ``_image_hook`` under test (task 03's wiring: the
# importer's span's exact path) — and drive the per-file ``_hook``
# reset (through the ``progress`` kwarg the fake import receives) and
# the transition/terminal clears.
def test_mid_run_status_reports_image_position_between_ticks(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""Phase 152 (task 04): the status carries the in-flight file's
VISION-image position — the pairs fire through the runner's OWN
``_image_hook`` closure (task 03's wiring — the importer's
``on_image_progress`` span's exact path), and the position CLIMBS
between ticks (3/15 → 15/15 — the PDF's vision span moving, the
owner's stuck label, now moving) while ``current_file`` and the
chunk position (14/38, a concurrent embed batch) keep their own
values — the THREE positions are independent sub-positions of the
file; the prelude reports 0/0, and the success terminal clears
BOTH sub-positions (image + chunk) to 0/0 while the FILE counts
keep their final values (the convention split)."""
clone = _GatedClone(threading.Event(), threading.Event())
image_release = threading.Event()
fake_import = _GatedImport(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
threading.Event(),
chunk_pairs=[(14, 38)], # a concurrent embed batch (independent)
image_pairs=[(3, 15), (15, 15)], # two describe calls of a 15-image span
image_release=image_release,
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
# Prelude: every position 0/0 (no file, no batch, no image
# yet — alongside the bare-label pins of the phase-64 test).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["current_file"] is None
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 0 and s["images_total"] == 0
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Import phase: the file is live (the progress hook's call),
# the chunk position 14/38 (the embed batch), and the image
# position 3/15 (the vision span) — the THREE positions are
# independent sub-positions of the same file.
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 14 and s["chunks_total"] == 38
assert s["images_done"] == 3 and s["images_total"] == 15
# Release the main park: the import fires image pair two
# (15/15 — the span's last describe call) and parks on
# image_release.
fake_import.release.set()
_poll_until(
lambda: sync_api.sync_status()["images_done"] == 15,
what="the second image pair on the status",
)
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
# The image position CLIMBED (3 → 15 — monotonic, no
# backwards jump, never above the total) ... the FILE and
# chunk positions keep their own values (independent).
assert s["images_done"] == 15 and s["images_total"] == 15
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 14 and s["chunks_total"] == 38
image_release.set()
finally:
clone.release.set()
fake_import.release.set()
image_release.set()
thread.join(10.0)
assert not thread.is_alive()
assert errors == [], f"the runner raised: {errors!r}"
# Wiring: every pair fired through the runner's OWN closure (the
# status reads above came through it; a wiring regression to
# None would have skipped the firing branch and timed out).
assert fake_import.chunk_hook_calls == [(14, 38)]
assert fake_import.image_hook_calls == [(3, 15), (15, 15)]
# Success terminal: BOTH sub-positions clear (0/0) ...
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 0 and s["images_total"] == 0
# ... while the FILE counts keep their final values (the
# convention split, pinned alongside the phase-64 terminal pin).
assert s["current_file"] is None
assert s["files_done"] == 1 and s["files_total"] == 3
class _TwoFileImport:
"""The phase-152 (task 01 + task 04) fake: fires the runner's
OWN progress hook for file A, then file A's image position through
the runner's OWN ``on_image_progress`` closure (task 03's wiring —
what the importer's span would do), parks (the test reads the
live position on the status), fires the progress hook again for
file B (the runner's ``_hook`` under test must reset the image
position to 0/0 — the pairing rule, extended), parks again, and
completes."""
def __init__(self, summary: ImportSummary) -> None:
self.summary = summary
self.parked_a = threading.Event() # parked after file A's progress
self.release_a = threading.Event() # test releases → file B's progress
self.parked_b = threading.Event() # parked after file B's progress
self.release_b = threading.Event()
self.hook_calls: list[tuple[str, str, int, int]] = []
self.image_hook_calls: list[tuple[int, int]] = [] # phase 152 (task 04)
async def __call__(
self,
sources: list[Path],
llm: object,
*,
prune: bool = False,
limit: int | None = None,
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
) -> ImportSummary:
if progress is not None:
progress("repo", "docs/a.pdf", 1, 2)
self.hook_calls.append(("repo", "docs/a.pdf", 1, 2))
# Phase 152 (task 04): file A's vision span is in flight —
# its image position (3/15) fires through the runner's OWN
# on_image_progress closure (task 03's wiring), then the fake
# parks.
if on_image_progress is not None:
on_image_progress(3, 15)
self.image_hook_calls.append((3, 15))
self.parked_a.set()
await asyncio.to_thread(self.release_a.wait)
if progress is not None:
# The SECOND progress call names file B — the closure under
# test must reset the image position the closure fired
# while file A was in flight (a file with no images must
# not inherit the previous PDF's position).
progress("repo", "docs/b.md", 2, 2)
self.hook_calls.append(("repo", "docs/b.md", 2, 2))
self.parked_b.set()
await asyncio.to_thread(self.release_b.wait)
return self.summary
def test_image_position_resets_per_file(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""Phase 152 (task 01 + task 04): the pairing rule, extended — a
``progress`` call names the file AND clears the image position;
every image position until the next ``progress`` call belongs to
it. File A's live position (3/15, fired through the runner's OWN
``_image_hook`` closure while parked on file A — task 04: the
phase-64/129 idiom, the importer's image hook's exact path) is
0/0 the moment the runner's own ``_hook`` names file B: a file
with no images must not inherit the previous PDF's position
(unlike the chunk position, not every file embeds chunks). The
success terminal keeps the position at 0/0 and the FILE counts
keep their final values (the convention split)."""
clone = _GatedClone(threading.Event(), threading.Event())
fake_import = _TwoFileImport(ImportSummary(files=2, added=1, updated=1))
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
clone.release.set()
assert fake_import.parked_a.wait(5.0), "the run never parked on file A"
# File A is named by the runner's own _hook (a wiring
# regression to None would leave current_file null here).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/docs/a.pdf"
assert s["files_done"] == 1 and s["files_total"] == 2
# File A's vision span is in flight: its image position is
# live on the status — fired through the runner's OWN
# _image_hook closure (the fake fired it before parking;
# a wiring regression to None would leave it 0/0 here).
assert s["images_done"] == 3 and s["images_total"] == 15
# Release file A: the runner's _hook names file B — the image
# position must reset to 0/0 in the same call (the pairing
# rule, extended; the closure under test, driven through the
# fake's progress kwarg).
fake_import.release_a.set()
assert fake_import.parked_b.wait(5.0), "the run never parked on file B"
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/docs/b.md"
assert s["files_done"] == 2 and s["files_total"] == 2
# RESET per file: file B (no images) does not inherit file
# A's 3/15.
assert s["images_done"] == 0 and s["images_total"] == 0
fake_import.release_b.set()
finally:
clone.release.set()
fake_import.release_a.set()
fake_import.release_b.set()
thread.join(10.0)
assert not thread.is_alive()
assert errors == [], f"the runner raised: {errors!r}"
# Wiring: BOTH progress calls went through the runner's own
# closure (the reset above was its doing), and the image pair
# went through the runner's own _image_hook closure.
assert fake_import.hook_calls == [
("repo", "docs/a.pdf", 1, 2),
("repo", "docs/b.md", 2, 2),
]
assert fake_import.image_hook_calls == [(3, 15)]
# Success terminal: the image position stays 0/0 (cleared in the
# terminal too — every terminal clears it) and the FILE counts
# keep their final values (the convention split).
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
assert s["current_file"] is None
assert s["files_done"] == 2 and s["files_total"] == 2
assert s["images_done"] == 0 and s["images_total"] == 0
def test_image_position_clears_at_phase_transitions_and_terminal(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""Phase 152 (task 01 + task 04): the image position is CLEARED
(0/0) at every point the import phase ends. This run's summary
changes the KB, so BOTH post-import transitions run and are
pinned: the import→``"overview"`` transition and the
import→``"summaries"`` transition (the changed-KB branch) each
clear the live position (3/15, fired mid-import through the
runner's OWN ``_image_hook`` closure — task 04: the
phase-64/129 idiom, the importer's span's exact path) to 0/0,
and the success terminal keeps it 0/0 — through the whole span
the FILE counts keep their final values (the convention split:
the file counts survive for the result line's context, the image
position is a sub-position of a file that's done)."""
clone = _GatedClone(threading.Event(), threading.Event())
import_release = threading.Event()
overview_started = threading.Event()
overview_release = threading.Event()
folders_started = threading.Event()
folders_release = threading.Event()
fake_import = _GatedImport(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
import_release,
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair to clear
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
# Park the post-import steps (re-monkeypatched over the seams
# helper's no-ops): the test reads the status DURING the
# "overview" phase and DURING the "summaries" phase — the two
# import-phase-ending transitions the task pins (the same shape
# as the phase-129 chunk-position transition test above).
async def gated_overview(llm: object, session: object = None) -> bool:
overview_started.set()
await asyncio.to_thread(overview_release.wait)
return True
async def gated_folders(
db: object,
llm: object,
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
folders_started.set()
await asyncio.to_thread(folders_release.wait)
return {"generated": 0, "failed": 0, "pruned": 0}
monkeypatch.setattr(sync_api, "regenerate_overview", gated_overview)
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Mid-import: the file is live; its image position went live
# too — fired through the runner's OWN _image_hook closure
# (image_pairs[0] — the importer's span would do the same).
s = sync_api.sync_status()
assert s["phase"] == "import"
assert s["images_done"] == 3 and s["images_total"] == 15
# Release the import: it returns (added+updated > 0) → the
# phase machine moves to "overview" and the position clears.
import_release.set()
assert overview_started.wait(5.0), "the run never reached the overview"
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "overview"
# CLEARED at the import→overview transition (0/0) ... the file
# counts KEEP their final values (the convention split).
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["files_done"] == 1 and s["files_total"] == 3
overview_release.set()
# ... and the changed-KB branch moves to "summaries" (the
# second import-phase-ending transition — cleared there too,
# idempotently) before parking.
assert folders_started.wait(5.0), "the run never reached the summaries"
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "summaries"
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["files_done"] == 1 and s["files_total"] == 3
folders_release.set()
finally:
clone.release.set()
import_release.set()
overview_release.set()
folders_release.set()
thread.join(10.0)
assert not thread.is_alive()
assert errors == [], f"the runner raised: {errors!r}"
assert fake_import.image_hook_calls == [(3, 15)]
# Success terminal: still 0/0 (the terminal clears too — same
# spots as current_file), file counts kept.
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
assert s["current_file"] is None
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["files_done"] == 1 and s["files_total"] == 3
# ---------- phase 134 (task 01): the cooperative cancel (5th state) ----------
#
# The cancel runs are driven through the REAL ``start_sync`` on a
@@ -1629,11 +2169,17 @@ def test_cancel_endpoint_stops_the_in_flight_run(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
threading.Event(),
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair before the cancel
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, _fast_clone)
def act(loop: asyncio.AbstractEventLoop) -> dict[str, str]:
assert fake_import.started.wait(5.0), "the run never reached the import"
# Phase 152 (task 04): the live image position in flight when
# the cancel converts was fired through the runner's OWN
# _image_hook closure (image_pairs[0] — the fake fired it
# before parking, like the importer's span would); the
# cancelled terminal must clear it.
return loop.run_until_complete(sync_api.cancel_sync())
try:
@@ -1665,6 +2211,11 @@ def test_cancel_endpoint_stops_the_in_flight_run(
assert s["files_done"] == 1 and s["files_total"] == 3 # final counts kept
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
# Phase 152 (task 04): the image position clears in the CANCELLED
# terminal too — every terminal clears it (the live 3/15 fired
# through the closure in flight is gone; the FILE counts above
# keep their final values).
assert s["images_done"] == 0 and s["images_total"] == 0
# PLAN §9: the ample-logging line for the converted cancel.
assert any(
r.getMessage()
@@ -1879,6 +2430,7 @@ def test_event_set_during_import_cancels_before_the_lite_spans(
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106