brain-of-reese/.agents/phases/todo/129_sync_chunk_progress/00_phase.md

Phase 129 — The sync label shows what chunk we're on out of how many

Source: TODO.md L5 — "The Syncing… name/of/file/here (2/3) needs to show what chunk we're on out of how many" (ASSUMPTION A6, owner-confirmed 2026-09-27: "chunk" = the current file's EMBEDDING CHUNKS; position reported at BATCH granularity — n = first chunk of the in-flight token-budgeted embed batch — because the transport embeds batches per HTTP call and per-chunk calls would be a latency regression; the summary-chunk embed is excluded from the counter; label Syncing… <file> (2/3) · chunk 14/38). Story: n/a (TODO-derived phase, Protocol B append; a NEW feature — distinct from phase 128's mobile fix, per the owner). Context: app/rag/llm.py — LLMClient.embed(texts) (L346: iterates self._batch_texts(texts) — token-budgeted batches, BOR_EMBED_BATCH_SIZE cap, _REQUEST_TOKEN_BUDGET ≈ 2800 chars), _embed_batch (L289: the halving backstop on _TooLarge), self.embed_batches counter (L249); app/rag/importer.py — import_sources (L408, the per-file progress(source, rel, done, total) hook fired BEFORE each _index_file), _index_file (L647 — text branch → _replace_content_chunks, image branch → _index_image_file, PDF branch → _index_pdf_file), _replace_content_chunks (L603 — chunks the content, llm.embed([c.content for c in doc.chunks]) in ONE call, token-cap re-chunk retry loop), _store_summary (embeds the ONE summary chunk — EXCLUDED from the counter); app/api/sync.py — SyncStatus (the phase-64/98 progress fields: current_file, files_done/total, phase, current_summary, summaries_done/total; _run_sync's _hook closure convention; terminal-state clearing); frontend/assets/sources.js — the phase-128 part builders (setSyncLabel, .sync-label-main / .sync-label-count); tests/e2e/slow_llm.py + tests/e2e/test_sync_summary_visibility.py — the slow-proxy sampling pattern.

Objective

While a file is being imported, the sync status (endpoint + button label) shows the file's embedding-chunk position: Syncing… <file> (2/3) · chunk 14/38 — 14 = the first chunk of the embed batch in flight, 38 = the file's total content chunks. The position climbs through the file's batches and is cleared when the file (or the run) moves on. Everything that passes no callback stays a zero-cost no-op (the CLI, the chat path).

Dependencies

• 128_sync_label_mobile (todo) — the label's part structure (.sync-label-main + pinned count span) is the carrier; this phase adds the pinned chunk span.

Design (shared by all tasks — the executor reads this, not the chat)

• The embed hook (task 01): LLMClient.embed(texts, on_progress: Callable[[int, int], None] | None = None) — before EACH top-level batch request, on_progress(first_1based, len(texts)) where first_1based = 1 + the cumulative chunk counts of the already-sent batches. None (every existing caller — the chat retrieval path, embed_one, the summarizer, the CLI) is a zero-cost guarded no-op. The _embed_batch halving backstop does NOT sub-report — the top-level loop reports batch starts; a halved batch means the position is approximate (document in the docstring; the backstop is rare by construction).
• Importer plumbing (task 02): import_sources(..., on_chunk_progress: Callable[[int, int], None] | None = None) — a new keyword-only param, plumbed: import_sources → _index_file(..., on_chunk=...) (all three branches: text, image, PDF — the PDF branch's content embed is the same _replace_content_chunks call; the image branch's ONE description-chunk embed reports 1/1 — acceptable, it IS the file's only chunk) → _replace_content_chunks(..., on_chunk=...) → llm.embed([...], on_progress=on_chunk). The token-cap re-chunk retry re-embeds with the NEW (smaller-target) chunk list — the hook naturally reports against the new total (no special case). _store_summary's single summary-chunk embed NEVER receives the hook (the counter is content chunks only — no 1/1 noise after the real progress ends). The scripts/import_docs.py CLI passes nothing — byte-identical.
• Sync status (task 03): SyncStatus gains current_chunk: int = 0 and chunks_total: int = 0 (documented: the current file's embed-batch position — 0/0 before the first batch and in terminal states). _run_sync: reset to 0/0 at run start (with the other progress fields); a closure _chunk_hook(chunk, total) (the file's _hook convention) sets them; passed as on_chunk_progress to import_sources. CLEARED (0/0) when the import phase ends — set at the phase = "overview" and phase = "summaries" transitions AND in both terminal states (the current_file clearing points — same convention, same spots). GET /api/sync/status exposes both fields (0/0 idle — the dataclass defaults). The SyncStatus docstring + module docstring get the chunk-position paragraph (the phase-64/98 style).
• UI (task 04): sources.js — the import-phase label gains a THIRD pinned part .sync-label-chunk (markup in index.html's #sync-label, hidden by default): when status.chunks_total > 0 the import fall-through emits · chunk ${status.current_chunk}/${status.chunks_total} into it; every other phase/state leaves it hidden/empty (the summaries phase has its own (n/m) — no chunk part; the prelude has no file yet). The UNTRUNCATED concatenation (main + count + chunk part) rides the button title + #sync-result (A4). The pinned unit tests (tests/unit/test_frontend_sync_upload.py) gain the chunk-part pins.

Tasks

1. 01_llm_embed_hook.md — LLMClient.embed's optional per-batch on_progress.
2. 02_importer_chunk_progress.md — import_sources → _index_file → _replace_content_chunks → embed plumbing.
3. 03_sync_status_chunk_fields.md — SyncStatus fields, the closure, clearing, the endpoint.
4. 04_ui_chunk_part.md — the label's chunk part + unit pins.
5. 05_e2e_chunk_progress.md — dedicated Playwright E2E, run in isolation.

Testing & Quality

• Unit/integration: the embed hook's call sequence (batch boundaries, None no-op, halving-approximate documented); the importer plumbing (hook fires per file with its own total; the CLI path unchanged; the summary-chunk embed never fires it); the sync status (fields in the endpoint JSON, reset per run, cleared at phase transitions + terminal, 0/0 idle).
• Coverage: >90% on new/modified code (app/).
• E2E: one dedicated Playwright file, run in isolation, slow-LLM sampling.

Completion Criteria

• GET /api/sync/status reports current_chunk climbing 1 → N with chunks_total = the file's true content-chunk count while a file imports (E2E, sampled)
• the rendered mobile/desktop label shows · chunk n/m during the import phase only; the untruncated text rides title + #sync-result
• uv run pytest --cov=app --cov-report=term-missing green, app/ coverage >90%
• uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov green in isolation
• no behavior change in completed phases (chat embed path, CLI import, the phase-64/98 label contract minus the new part)