phase: 129_sync_chunk_progress
Build and Push Containers / build-and-push-app (push) Successful in 8m23s
Build and Push Containers / build-and-push-db (push) Successful in 40s

All green. Final report:

**Phase 129 final verification pass — 100% green** (previous work intact; this pass re-verified everything and fixed two pre-existing README inconsistencies)

- Verified on-spec: `LLMClient.embed` per-batch `on_progress` hook; importer `on_chunk_progress` plumbing (text/image/PDF branches; summary embed excluded); `SyncStatus.current_chunk/chunks_total` reset+clear contract; UI `.sync-label-chunk` pinned part with untruncated `title`/`#sync-result`
- `uv run pytest --cov=app -q`: exit 0, ~3041 passed, 0 failed, `app/` coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` (isolated, DB up): **2 passed** (endpoint chunk climb with true `chunk_document` total + UI `· chunk n/m` part)
- `uv run ruff check .`: all checks passed · `uv run pyright`: 0 errors, 0 warnings
- Completion criteria: all 5 ✅ (incl. no behavior change in completed phases — full suite green, CLI/chat no-op paths byte-identical)
- Docs: README.md fixed 2 pre-existing inconsistencies (upload section claimed "unpacks + scans immediately" — phase 90 made it unpack+register only, scan deferred to Sync; troubleshooting claimed "120s client timeout" — actual guard is the 300 s pre-token `TURN_TIMEOUT_MS`). Phase-129 label behavior already documented. AGENTS.md re-verified consistent — no changes
- Removed stray untracked `NOTES.md` (8-byte artifact); no live-infra changes; no commits — all changes left in working tree for the harness
- Next pending phase: `130_chat_answer_clusters`
This commit is contained in:
2026-09-28 03:38:14 -04:00
parent 1139f007cf
commit 2d0f608246
53 changed files with 3590 additions and 157 deletions
+96 -6
View File
@@ -112,6 +112,21 @@ Terminal states (success AND failed) clear ``phase`` +
``current_summary`` but keep the run's final ``summaries_done`` /
``summaries_total`` (the phase-64 keep-final-counts convention).
Phase 129 (task 03) adds the in-flight file's EMBED-chunk position —
``current_chunk`` (the 1-based first chunk of the in-flight
token-budgeted batch) + ``chunks_total`` (the file's total content
chunks), batch granularity via the :meth:`app.rag.llm.LLMClient.embed`
``on_progress`` hook (tasks 01/02). They are 0/0 before the first
batch (the prelude), during the ``"overview"`` / ``"summaries"``
phases, and in terminal states — the import phase clearing them at
EVERY point it ends (the ``phase = "overview"`` and
``phase = "summaries"`` transitions and both terminals, the same spots
``current_file`` clears). Unlike the phase-64 FILE counts, the chunk
position is a SUB-position of a file that's done — the keep-final-
counts convention does NOT extend to it: the file counts survive the
terminal for the result line's context, the chunk position clears
with the file.
The ``failed`` state's ``error`` string is masked by the shared
sanitizer — the ``user:pass@`` masker now lives in :mod:`app.core.errors`
(imported here under the private name ``_sanitize_error``).
@@ -178,6 +193,19 @@ class SyncStatus:
Terminal states clear ``phase`` + ``current_summary`` but keep the
run's final summary counts (the phase-64 keep-final-counts
convention).
Phase 129 (task 03) chunk-position fields: ``current_chunk`` is
the 1-based first chunk of the in-flight token-budgeted batch and
``chunks_total`` the in-flight file's total content chunks (batch
granularity, the ``LLMClient.embed`` hook — the module docstring's
contract). They are 0/0 before the first batch, during the
``"overview"`` / ``"summaries"`` phases, and in terminal states —
the import phase clears them at every point it ends (the
overview/summaries transitions + both terminals, where
``current_file`` also clears). The phase-64 keep-final-counts
convention covers the FILE counts only — the chunk position is a
sub-position of a file that's done, so it clears in the terminal
while ``files_done`` / ``files_total`` keep their final values.
"""
state: Literal["idle", "running", "success", "failed"] = "idle"
@@ -197,6 +225,13 @@ class SyncStatus:
current_summary: str | None = None
summaries_done: int = 0
summaries_total: int = 0
# Phase 129 (task 03): the current file's EMBED chunk position —
# the first chunk of the in-flight token-budgeted batch and the
# file's total content chunks (the LLMClient.embed hook, batch
# granularity). 0/0 before the first batch, during the
# prelude/overview/summaries phases, and in terminal states.
current_chunk: int = 0
chunks_total: int = 0
_status = SyncStatus()
@@ -211,11 +246,17 @@ def sync_status() -> dict[str, Any]:
``current_file`` (phase 64) is the ``source/relative/path`` the
import is processing right now — null during the clone/pull phase
and in terminal states; ``files_done`` / ``files_total`` carry the
hook's position (0/0 idle). ``phase`` (phase 98) is the pipeline
stage — null in the prelude and terminal states; while in
``"summaries"``, ``current_summary`` + ``summaries_done`` /
``summaries_total`` carry the generator's position (the terminal
keeps the final summary counts).
hook's position (0/0 idle) and keep their final values in the
terminal. ``current_chunk`` / ``chunks_total`` (phase 129) are the
in-flight file's embed-chunk position (batch granularity — the
first chunk of the in-flight batch over the file's total content
chunks): 0/0 before the first batch, in the prelude/overview/
summaries phases, and in terminal states (they clear at every
point the import phase ends; the FILE counts do not). ``phase``
(phase 98) is the pipeline stage — null in the prelude and
terminal states; while in ``"summaries"``, ``current_summary`` +
``summaries_done`` / ``summaries_total`` carry the generator's
position (the terminal keeps the final summary counts).
"""
return {
"state": _status.state,
@@ -226,6 +267,12 @@ def sync_status() -> dict[str, Any]:
"current_file": _status.current_file,
"files_done": _status.files_done,
"files_total": _status.files_total,
# Phase 129 (task 03): the in-flight file's embed-chunk
# position — the dataclass defaults make idle 0/0, and the
# runner clears it at every point the import phase ends
# (overview/summaries transitions + both terminals).
"current_chunk": _status.current_chunk,
"chunks_total": _status.chunks_total,
# Phase 98 (task 01): the phase machine — null/0/0 idle (the
# dataclass defaults) and in terminal states (which clear
# phase + current_summary but keep the final summary counts).
@@ -278,6 +325,10 @@ async def _run_sync() -> None:
_status.current_summary = None
_status.summaries_done = 0
_status.summaries_total = 0
# Phase 129 (task 03): the chunk-position fields reset with the
# run — 0/0 until the import's first embed batch reports.
_status.current_chunk = 0
_status.chunks_total = 0
try:
settings = get_settings()
# Step 1 (phase 41): fail fast — verify both models the sync
@@ -359,18 +410,35 @@ async def _run_sync() -> None:
_status.files_done = done
_status.files_total = total
# Phase 129 (task 03): the in-flight file's embed-chunk
# position — the LLMClient.embed hook (task 01) fires before
# each token-budgeted batch with (first_chunk_1based, total);
# the file's IDENTITY is the sibling ``progress`` hook's (the
# two hooks are paired — a progress call names the file, and
# every chunk position until the next one belongs to it).
def _chunk_hook(chunk: int, total: int) -> None:
_status.current_chunk = chunk
_status.chunks_total = total
# Phase 98 (task 01): the phase machine — the import phase
# starts NOW (the model-check + clone/pull prelude above
# reported ``phase: null``).
_status.phase = "import"
summary: ImportSummary = await import_sources(
sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root,
sources, llm, prune=True, progress=_hook, on_chunk_progress=_chunk_hook,
ignore_by_root=ignore_by_root,
include_hidden_by_root=include_hidden_by_root,
doc_dates_by_root=doc_dates_by_root,
)
overview = False
if summary.added + summary.updated > 0:
_status.phase = "overview" # phase 98 (task 01)
# Phase 129 (task 03): the import phase ends here — the
# chunk position (a sub-position of a done file) clears
# with the phase change; the file counts keep their final
# values (the phase-64 convention covers files only).
_status.current_chunk = 0
_status.chunks_total = 0
overview = await regenerate_overview(llm)
# Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03):
# the folder summaries — the drill-down ls's per-level
@@ -419,6 +487,11 @@ async def _run_sync() -> None:
# position through the (long) span (its total is the
# affected count).
_status.phase = "summaries"
# Phase 129 (task 03): the import phase ends here too
# (the changed-KB branch skips no phase) — same
# chunk-position clear as the overview transition.
_status.current_chunk = 0
_status.chunks_total = 0
folder_stats = await generate_folder_summaries(
fs_db, llm, changed_docs=summary.changed_paths,
on_progress=_summary_hook,
@@ -435,6 +508,11 @@ async def _run_sync() -> None:
# phase + hook (the fill's total is the missing
# count — the hook's loop-start ``total``).
_status.phase = "summaries"
# Phase 129 (task 03): the import phase ends here
# too (the unchanged-walk gap-fill branch) — same
# chunk-position clear.
_status.current_chunk = 0
_status.chunks_total = 0
folder_stats = await generate_folder_summaries(
fs_db, llm, only_missing=True, on_progress=_summary_hook
)
@@ -475,6 +553,12 @@ async def _run_sync() -> None:
_status.current_file = None # phase 64: keep the final counts
_status.phase = None # phase 98: keep the final summary counts
_status.current_summary = None
# Phase 129 (task 03): the chunk position is a sub-position of
# a file that's done — it clears in the terminal (unlike the
# FILE counts, which keep their final values for the result
# line's context).
_status.current_chunk = 0
_status.chunks_total = 0
_status.detail = {
"files": summary.files,
"added": summary.added,
@@ -498,3 +582,9 @@ async def _run_sync() -> None:
_status.current_file = None # phase 64: keep the final counts
_status.phase = None # phase 98: keep the final summary counts
_status.current_summary = None
# Phase 129 (task 03): the chunk position clears in the
# failed terminal too (the convention split — the FILE counts
# keep their final values next to the error, the sub-position
# of a file that's done/abandoned does not).
_status.current_chunk = 0
_status.chunks_total = 0
+104 -10
View File
@@ -144,7 +144,10 @@ is NOT cleared on success — it is the retrieval display's record
also, task 05).
``import_sources`` accepts an optional per-file ``progress`` callback
(phase 64, task 01) reporting the file being processed right now.
(phase 64, task 01) reporting the file being processed right now, and
an optional ``on_chunk_progress`` callback (phase 129, task 02)
reporting the in-flight file's embedding-chunk position at batch
granularity (see the function docstring).
"""
from __future__ import annotations
@@ -198,7 +201,14 @@ class Embedder(Protocol):
settings: Settings
embed_batches: int
async def embed(self, texts: list[str]) -> list[list[float]]: ...
async def embed(
self,
texts: list[str],
*,
# Phase 129 (task 01): the per-batch position hook — ``None``
# (every pre-phase caller) is a zero-cost guarded no-op.
on_progress: Callable[[int, int], None] | None = None,
) -> list[list[float]]: ...
async def chat(self, messages: list[dict[str, Any]], model: str | None = None) -> str: ...
# ^ the one-shot completion the summarizer uses — the ``lite`` model
@@ -419,6 +429,7 @@ async def import_sources(
limit: int | None = None,
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None,
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
@@ -442,6 +453,32 @@ async def import_sources(
exceeds the limit, but ``total`` stays the full pre-walk count (an
incomplete walk must not misreport the denominator).
``on_chunk_progress`` (phase 129, task 02) is an optional
chunk-position hook for the file currently being indexed: while that
file's content chunks embed, it is called — through
:meth:`app.rag.llm.LLMClient.embed`'s ``on_progress`` — before EACH
embed batch with ``(first_chunk_1based, total_chunks)``: the 1-based
position of the first chunk of the in-flight batch and the file's
total content-chunk count (batch granularity, the phase-129 A6
assumption — the transport embeds token-budgeted batches per HTTP
call). It carries the in-flight FILE's chunk position only, never
the file identity — the per-file ``progress`` hook owns that: the
caller pairs them (a ``progress`` call names the file, and every
chunk position reported until the next ``progress`` call belongs to
it; the position climbs through the file's batches and restarts at
1 for the next file). Same "only when provided" rule as
``progress``: ``None`` (every existing caller — the CLI, the chat
path) is a zero-cost guarded no-op, and the run is byte-identical
to a hook-less run. Only the content-chunk embeds report (the
shared :func:`_replace_content_chunks` — text, image, and PDF
branches; the image branch's one description chunk reports
``1/1``, which IS its only chunk): the document's summary-chunk
embed (:func:`_store_summary`) NEVER receives the hook, so the
counter counts content chunks only (no ``1/1`` noise after the
real progress ends), and the post-index enrichment pass / the
PDF's embedded-image docs (separate documents, not the walked
file) do not report either.
``ignore_by_root`` (phase 89, A1/A2) maps ``str(root)`` — the root path
string exactly as passed in *sources* — to that source's RAW ignore-path
lines (the importer normalizes them via ``_ignore_for_root``, the single
@@ -574,6 +611,7 @@ async def import_sources(
await _index_file(
session, source=source, rel=rel, full_path=path, llm=llm,
summary=summary, raw_date=dates_map.get(rel),
on_chunk=on_chunk_progress,
)
except EmbeddingError as e:
# A pathological file (e.g. content the embedding endpoint
@@ -608,7 +646,13 @@ async def import_sources(
async def _replace_content_chunks(
session: Session, *, doc: Document, llm: Embedder, content: str, rel: str
session: Session,
*,
doc: Document,
llm: Embedder,
content: str,
rel: str,
on_chunk: Callable[[int, int], None] | None = None,
) -> int:
"""Phase 1+2 on *content*: chunk it (format-aware, PLAN §5), replace
the doc's content chunk rows (embeddings NULL), and embed the new
@@ -622,6 +666,20 @@ async def _replace_content_chunks(
verbatim by the text path (:func:`_index_file`), the image path
(task 03), and the task-04 enrichment pass — one copy, three call
sites.
``on_chunk`` (phase 129, task 02) is the in-flight file's
chunk-position hook, forwarded as ``on_progress`` to the content
embed: it fires before each top-level embed batch with
``(first_chunk_1based, total_chunks)``. The token-cap re-chunk retry
(the ``while True`` above that halves *target*) re-embeds the NEW
(smaller-target) chunk list, and the hook reports against the NEW
total automatically — no special case: the announced total is
always the chunk list actually in flight. The empty-content
``doc.chunks = []`` break never embeds, so the hook never fires for
it. ``None`` (the default; every pre-phase call site — the CLI, the
phase-125 enrichment pass, and the PDF's embedded-image docs, which
index separate documents rather than the walked file) keeps the call
byte-identical.
"""
settings = llm.settings
target = max(400, settings.chunk_target_chars)
@@ -635,7 +693,12 @@ async def _replace_content_chunks(
if not doc.chunks:
break
try:
vectors = await llm.embed([c.content for c in doc.chunks])
# Phase 129 (task 02): the in-flight file's chunk-position
# hook (``on_progress`` — the task-01 contract; ``None`` is a
# zero-cost no-op).
vectors = await llm.embed(
[c.content for c in doc.chunks], on_progress=on_chunk
)
for row, vec in zip(doc.chunks, vectors, strict=True):
row.embedding = vec
break
@@ -660,6 +723,7 @@ async def _index_file(
llm: Embedder,
summary: ImportSummary,
raw_date: datetime | None = None,
on_chunk: Callable[[int, int], None] | None = None,
) -> None:
"""Upsert one file: doc row + chunk rows + embeddings, one transaction.
@@ -668,6 +732,15 @@ async def _index_file(
map, or ``None`` (every non-git case): the file's mtime is read
here, once, and becomes the source date (the D2 fallback).
``on_chunk`` (phase 129, task 02) is the in-flight file's
chunk-position hook (see :func:`import_sources`) — passed through to
ALL THREE branches' content embeds: the text path's
:func:`_replace_content_chunks` directly, the image branch via
:func:`_index_image_file` (its one description chunk reports
``1/1`` — it IS the file's only chunk), and the PDF branch via
:func:`_index_pdf_file` (its content embed is the primary target).
``None`` (the default) keeps every branch byte-identical.
Image files (phase 122, toggle on) delegate to
:func:`_index_image_file` — the binary path (bytes digest,
persistent copy, ``content`` = the vision description) — BEFORE
@@ -683,7 +756,7 @@ async def _index_file(
if match_extension(full_path, image_set) is not None:
return await _index_image_file(
session, source=source, rel=rel, full_path=full_path, llm=llm,
summary=summary, raw_date=raw_date,
summary=summary, raw_date=raw_date, on_chunk=on_chunk,
)
# Phase 126: PDF files take the PDF index path — AFTER the image
# branch, BEFORE any text read (a PDF is binary; ``extract_pdf_text``
@@ -691,7 +764,7 @@ async def _index_file(
if full_path.suffix.lower() == ".pdf":
return await _index_pdf_file(
session, source=source, rel=rel, full_path=full_path, llm=llm,
summary=summary, raw_date=raw_date,
summary=summary, raw_date=raw_date, on_chunk=on_chunk,
)
content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
@@ -793,7 +866,7 @@ async def _index_file(
# lives in the shared helper — the global policy stays intact for the
# rest of the KB).
chunks_n = await _replace_content_chunks(
session, doc=doc, llm=llm, content=content, rel=rel
session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
)
session.commit()
@@ -952,6 +1025,7 @@ async def _index_pdf_file(
llm: Embedder,
summary: ImportSummary,
raw_date: datetime | None = None,
on_chunk: Callable[[int, int], None] | None = None,
) -> None:
"""The phase-126 PDF branch of :func:`_index_file` — a PDF is
indexed from its EXTRACTED TEXT (LOCKED A1/A2/A4), one transaction
@@ -985,6 +1059,14 @@ async def _index_pdf_file(
``raw_date`` follows the text path: the caller's git last-commit
date or ``None`` (the mtime fallback, read before the unchanged
early-return — the unchanged path refreshes from the same source).
``on_chunk`` (phase 129, task 02) is the in-flight file's
chunk-position hook: it is forwarded to the PDF's CONTENT embed
(the :func:`_replace_content_chunks` call — the primary target),
but NOT to the embedded-image pass (those image docs are separate
documents — their ``1/1`` positions would be noise after the
file's real progress ends, the A6 exclusion). ``None`` keeps the
branch byte-identical.
"""
# Extract the text (fail-soft: an unopenable PDF yields "" — the
# document may still carry embedded images worth indexing).
@@ -1066,9 +1148,11 @@ async def _index_pdf_file(
# Phase 1+2 — the shared chunk/replace/embed pipeline (the page-
# aware chunker dispatches on the ``.pdf`` suffix; empty content
# clears the chunk rows and returns 0), committed atomically with
# the document.
# the document. Phase 129 (task 02): the in-flight file's
# chunk-position hook rides the content embed (the embedded-image
# pass does not receive it — separate documents, the A6 exclusion).
chunks_n = await _replace_content_chunks(
session, doc=doc, llm=llm, content=content, rel=rel
session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
)
session.commit()
@@ -1815,6 +1899,7 @@ async def _index_image_file(
llm: Embedder,
summary: ImportSummary,
raw_date: datetime | None = None,
on_chunk: Callable[[int, int], None] | None = None,
) -> None:
"""The phase-122 image branch of :func:`_index_file` — a standalone
image is indexed from its BYTES, never its text:
@@ -1855,6 +1940,13 @@ async def _index_image_file(
pre-phase). The UNCHANGED path never re-scans (the stored context
is part of the row's committed content; a changed source state
re-indexes the file through this path).
``on_chunk`` (phase 129, task 02) is the in-flight file's
chunk-position hook: it is forwarded to the description's content
embed (:func:`_replace_content_chunks`). A description (enriched or
not) chunks to ONE chunk, so the hook reports ``1/1`` — acceptable,
it IS the file's only chunk. ``None`` keeps the branch
byte-identical.
"""
settings = llm.settings
data = full_path.read_bytes()
@@ -1996,8 +2088,10 @@ async def _index_image_file(
# the chunk rows (embeddings NULL), embed, commit — one transaction
# per file; the endpoint token-cap retry lives in the shared helper,
# and a description is short, so it never fires in practice).
# Phase 129 (task 02): the in-flight file's chunk-position hook
# rides this embed (one description chunk → ``1/1``).
chunks_n = await _replace_content_chunks(
session, doc=doc, llm=llm, content=content, rel=rel
session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
)
session.commit()
+34 -3
View File
@@ -22,7 +22,7 @@ from __future__ import annotations
import asyncio
import json
import logging
from collections.abc import AsyncGenerator
from collections.abc import AsyncGenerator, Callable
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any, Literal, cast
@@ -343,12 +343,43 @@ class LLMClient:
batches.append(cur)
return batches
async def embed(self, texts: list[str]) -> list[list[float]]:
"""Embed *texts* in token-budgeted batches (order always kept)."""
async def embed(
self,
texts: list[str],
*,
on_progress: Callable[[int, int], None] | None = None,
) -> list[list[float]]:
"""Embed *texts* in token-budgeted batches (order always kept).
Progress hook (phase 129, task 01): when *on_progress* is given,
it is called once per TOP-LEVEL batch, BEFORE the batch's request
(a failing batch still announces its start), with
``(first_chunk_1based, total_chunks)`` — the 1-based index of the
first text of the in-flight batch and the total number of texts
in this call — so a caller can render ``chunk n/m`` (the importer
passes this through for the sync label; PLAN §7.4 "never stale").
``len(out) + 1`` is that first index: each batch's vectors extend
the accumulator 1:1, so ``len(out)`` is exactly the number of
chunks embedded by earlier batches. The :meth:`_embed_batch`
halving backstop does NOT sub-report: a halved batch leaves the
announced position approximate (the hook already fired for the
original batch's start) — documented; the backstop is rare by
construction (phase 114's ``_TooLarge`` handling). ``None`` (the
default — every pre-phase caller: the chat retrieval path,
:meth:`embed_one`, the summarizer, the importer) is a zero-cost
guarded no-op: call sequence and return value are
byte-identical to the pre-hook client.
"""
if not texts:
return []
out: list[list[float]] = []
for chunk in self._batch_texts(texts):
# Phase 129 (task 01): announce THIS batch's start before the
# request — the hook precedes the try, so even a failing batch
# reports its position.
if on_progress is not None:
on_progress(len(out) + 1, len(texts))
try:
vecs = await self._embed_batch(chunk)
except EmbeddingError: