phase: 129_sync_chunk_progress
All green. Final report:
**Phase 129 final verification pass — 100% green** (previous work intact; this pass re-verified everything and fixed two pre-existing README inconsistencies)
- Verified on-spec: `LLMClient.embed` per-batch `on_progress` hook; importer `on_chunk_progress` plumbing (text/image/PDF branches; summary embed excluded); `SyncStatus.current_chunk/chunks_total` reset+clear contract; UI `.sync-label-chunk` pinned part with untruncated `title`/`#sync-result`
- `uv run pytest --cov=app -q`: exit 0, ~3041 passed, 0 failed, `app/` coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` (isolated, DB up): **2 passed** (endpoint chunk climb with true `chunk_document` total + UI `· chunk n/m` part)
- `uv run ruff check .`: all checks passed · `uv run pyright`: 0 errors, 0 warnings
- Completion criteria: all 5 ✅ (incl. no behavior change in completed phases — full suite green, CLI/chat no-op paths byte-identical)
- Docs: README.md fixed 2 pre-existing inconsistencies (upload section claimed "unpacks + scans immediately" — phase 90 made it unpack+register only, scan deferred to Sync; troubleshooting claimed "120s client timeout" — actual guard is the 300 s pre-token `TURN_TIMEOUT_MS`). Phase-129 label behavior already documented. AGENTS.md re-verified consistent — no changes
- Removed stray untracked `NOTES.md` (8-byte artifact); no live-infra changes; no commits — all changes left in working tree for the harness
- Next pending phase: `130_chat_answer_clusters`
This commit is contained in:
+96
-6
@@ -112,6 +112,21 @@ Terminal states (success AND failed) clear ``phase`` +
|
||||
``current_summary`` but keep the run's final ``summaries_done`` /
|
||||
``summaries_total`` (the phase-64 keep-final-counts convention).
|
||||
|
||||
Phase 129 (task 03) adds the in-flight file's EMBED-chunk position —
|
||||
``current_chunk`` (the 1-based first chunk of the in-flight
|
||||
token-budgeted batch) + ``chunks_total`` (the file's total content
|
||||
chunks), batch granularity via the :meth:`app.rag.llm.LLMClient.embed`
|
||||
``on_progress`` hook (tasks 01/02). They are 0/0 before the first
|
||||
batch (the prelude), during the ``"overview"`` / ``"summaries"``
|
||||
phases, and in terminal states — the import phase clearing them at
|
||||
EVERY point it ends (the ``phase = "overview"`` and
|
||||
``phase = "summaries"`` transitions and both terminals, the same spots
|
||||
``current_file`` clears). Unlike the phase-64 FILE counts, the chunk
|
||||
position is a SUB-position of a file that's done — the keep-final-
|
||||
counts convention does NOT extend to it: the file counts survive the
|
||||
terminal for the result line's context, the chunk position clears
|
||||
with the file.
|
||||
|
||||
The ``failed`` state's ``error`` string is masked by the shared
|
||||
sanitizer — the ``user:pass@`` masker now lives in :mod:`app.core.errors`
|
||||
(imported here under the private name ``_sanitize_error``).
|
||||
@@ -178,6 +193,19 @@ class SyncStatus:
|
||||
Terminal states clear ``phase`` + ``current_summary`` but keep the
|
||||
run's final summary counts (the phase-64 keep-final-counts
|
||||
convention).
|
||||
|
||||
Phase 129 (task 03) chunk-position fields: ``current_chunk`` is
|
||||
the 1-based first chunk of the in-flight token-budgeted batch and
|
||||
``chunks_total`` the in-flight file's total content chunks (batch
|
||||
granularity, the ``LLMClient.embed`` hook — the module docstring's
|
||||
contract). They are 0/0 before the first batch, during the
|
||||
``"overview"`` / ``"summaries"`` phases, and in terminal states —
|
||||
the import phase clears them at every point it ends (the
|
||||
overview/summaries transitions + both terminals, where
|
||||
``current_file`` also clears). The phase-64 keep-final-counts
|
||||
convention covers the FILE counts only — the chunk position is a
|
||||
sub-position of a file that's done, so it clears in the terminal
|
||||
while ``files_done`` / ``files_total`` keep their final values.
|
||||
"""
|
||||
|
||||
state: Literal["idle", "running", "success", "failed"] = "idle"
|
||||
@@ -197,6 +225,13 @@ class SyncStatus:
|
||||
current_summary: str | None = None
|
||||
summaries_done: int = 0
|
||||
summaries_total: int = 0
|
||||
# Phase 129 (task 03): the current file's EMBED chunk position —
|
||||
# the first chunk of the in-flight token-budgeted batch and the
|
||||
# file's total content chunks (the LLMClient.embed hook, batch
|
||||
# granularity). 0/0 before the first batch, during the
|
||||
# prelude/overview/summaries phases, and in terminal states.
|
||||
current_chunk: int = 0
|
||||
chunks_total: int = 0
|
||||
|
||||
|
||||
_status = SyncStatus()
|
||||
@@ -211,11 +246,17 @@ def sync_status() -> dict[str, Any]:
|
||||
``current_file`` (phase 64) is the ``source/relative/path`` the
|
||||
import is processing right now — null during the clone/pull phase
|
||||
and in terminal states; ``files_done`` / ``files_total`` carry the
|
||||
hook's position (0/0 idle). ``phase`` (phase 98) is the pipeline
|
||||
stage — null in the prelude and terminal states; while in
|
||||
``"summaries"``, ``current_summary`` + ``summaries_done`` /
|
||||
``summaries_total`` carry the generator's position (the terminal
|
||||
keeps the final summary counts).
|
||||
hook's position (0/0 idle) and keep their final values in the
|
||||
terminal. ``current_chunk`` / ``chunks_total`` (phase 129) are the
|
||||
in-flight file's embed-chunk position (batch granularity — the
|
||||
first chunk of the in-flight batch over the file's total content
|
||||
chunks): 0/0 before the first batch, in the prelude/overview/
|
||||
summaries phases, and in terminal states (they clear at every
|
||||
point the import phase ends; the FILE counts do not). ``phase``
|
||||
(phase 98) is the pipeline stage — null in the prelude and
|
||||
terminal states; while in ``"summaries"``, ``current_summary`` +
|
||||
``summaries_done`` / ``summaries_total`` carry the generator's
|
||||
position (the terminal keeps the final summary counts).
|
||||
"""
|
||||
return {
|
||||
"state": _status.state,
|
||||
@@ -226,6 +267,12 @@ def sync_status() -> dict[str, Any]:
|
||||
"current_file": _status.current_file,
|
||||
"files_done": _status.files_done,
|
||||
"files_total": _status.files_total,
|
||||
# Phase 129 (task 03): the in-flight file's embed-chunk
|
||||
# position — the dataclass defaults make idle 0/0, and the
|
||||
# runner clears it at every point the import phase ends
|
||||
# (overview/summaries transitions + both terminals).
|
||||
"current_chunk": _status.current_chunk,
|
||||
"chunks_total": _status.chunks_total,
|
||||
# Phase 98 (task 01): the phase machine — null/0/0 idle (the
|
||||
# dataclass defaults) and in terminal states (which clear
|
||||
# phase + current_summary but keep the final summary counts).
|
||||
@@ -278,6 +325,10 @@ async def _run_sync() -> None:
|
||||
_status.current_summary = None
|
||||
_status.summaries_done = 0
|
||||
_status.summaries_total = 0
|
||||
# Phase 129 (task 03): the chunk-position fields reset with the
|
||||
# run — 0/0 until the import's first embed batch reports.
|
||||
_status.current_chunk = 0
|
||||
_status.chunks_total = 0
|
||||
try:
|
||||
settings = get_settings()
|
||||
# Step 1 (phase 41): fail fast — verify both models the sync
|
||||
@@ -359,18 +410,35 @@ async def _run_sync() -> None:
|
||||
_status.files_done = done
|
||||
_status.files_total = total
|
||||
|
||||
# Phase 129 (task 03): the in-flight file's embed-chunk
|
||||
# position — the LLMClient.embed hook (task 01) fires before
|
||||
# each token-budgeted batch with (first_chunk_1based, total);
|
||||
# the file's IDENTITY is the sibling ``progress`` hook's (the
|
||||
# two hooks are paired — a progress call names the file, and
|
||||
# every chunk position until the next one belongs to it).
|
||||
def _chunk_hook(chunk: int, total: int) -> None:
|
||||
_status.current_chunk = chunk
|
||||
_status.chunks_total = total
|
||||
|
||||
# Phase 98 (task 01): the phase machine — the import phase
|
||||
# starts NOW (the model-check + clone/pull prelude above
|
||||
# reported ``phase: null``).
|
||||
_status.phase = "import"
|
||||
summary: ImportSummary = await import_sources(
|
||||
sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root,
|
||||
sources, llm, prune=True, progress=_hook, on_chunk_progress=_chunk_hook,
|
||||
ignore_by_root=ignore_by_root,
|
||||
include_hidden_by_root=include_hidden_by_root,
|
||||
doc_dates_by_root=doc_dates_by_root,
|
||||
)
|
||||
overview = False
|
||||
if summary.added + summary.updated > 0:
|
||||
_status.phase = "overview" # phase 98 (task 01)
|
||||
# Phase 129 (task 03): the import phase ends here — the
|
||||
# chunk position (a sub-position of a done file) clears
|
||||
# with the phase change; the file counts keep their final
|
||||
# values (the phase-64 convention covers files only).
|
||||
_status.current_chunk = 0
|
||||
_status.chunks_total = 0
|
||||
overview = await regenerate_overview(llm)
|
||||
# Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03):
|
||||
# the folder summaries — the drill-down ls's per-level
|
||||
@@ -419,6 +487,11 @@ async def _run_sync() -> None:
|
||||
# position through the (long) span (its total is the
|
||||
# affected count).
|
||||
_status.phase = "summaries"
|
||||
# Phase 129 (task 03): the import phase ends here too
|
||||
# (the changed-KB branch skips no phase) — same
|
||||
# chunk-position clear as the overview transition.
|
||||
_status.current_chunk = 0
|
||||
_status.chunks_total = 0
|
||||
folder_stats = await generate_folder_summaries(
|
||||
fs_db, llm, changed_docs=summary.changed_paths,
|
||||
on_progress=_summary_hook,
|
||||
@@ -435,6 +508,11 @@ async def _run_sync() -> None:
|
||||
# phase + hook (the fill's total is the missing
|
||||
# count — the hook's loop-start ``total``).
|
||||
_status.phase = "summaries"
|
||||
# Phase 129 (task 03): the import phase ends here
|
||||
# too (the unchanged-walk gap-fill branch) — same
|
||||
# chunk-position clear.
|
||||
_status.current_chunk = 0
|
||||
_status.chunks_total = 0
|
||||
folder_stats = await generate_folder_summaries(
|
||||
fs_db, llm, only_missing=True, on_progress=_summary_hook
|
||||
)
|
||||
@@ -475,6 +553,12 @@ async def _run_sync() -> None:
|
||||
_status.current_file = None # phase 64: keep the final counts
|
||||
_status.phase = None # phase 98: keep the final summary counts
|
||||
_status.current_summary = None
|
||||
# Phase 129 (task 03): the chunk position is a sub-position of
|
||||
# a file that's done — it clears in the terminal (unlike the
|
||||
# FILE counts, which keep their final values for the result
|
||||
# line's context).
|
||||
_status.current_chunk = 0
|
||||
_status.chunks_total = 0
|
||||
_status.detail = {
|
||||
"files": summary.files,
|
||||
"added": summary.added,
|
||||
@@ -498,3 +582,9 @@ async def _run_sync() -> None:
|
||||
_status.current_file = None # phase 64: keep the final counts
|
||||
_status.phase = None # phase 98: keep the final summary counts
|
||||
_status.current_summary = None
|
||||
# Phase 129 (task 03): the chunk position clears in the
|
||||
# failed terminal too (the convention split — the FILE counts
|
||||
# keep their final values next to the error, the sub-position
|
||||
# of a file that's done/abandoned does not).
|
||||
_status.current_chunk = 0
|
||||
_status.chunks_total = 0
|
||||
|
||||
+104
-10
@@ -144,7 +144,10 @@ is NOT cleared on success — it is the retrieval display's record
|
||||
also, task 05).
|
||||
|
||||
``import_sources`` accepts an optional per-file ``progress`` callback
|
||||
(phase 64, task 01) reporting the file being processed right now.
|
||||
(phase 64, task 01) reporting the file being processed right now, and
|
||||
an optional ``on_chunk_progress`` callback (phase 129, task 02)
|
||||
reporting the in-flight file's embedding-chunk position at batch
|
||||
granularity (see the function docstring).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -198,7 +201,14 @@ class Embedder(Protocol):
|
||||
settings: Settings
|
||||
embed_batches: int
|
||||
|
||||
async def embed(self, texts: list[str]) -> list[list[float]]: ...
|
||||
async def embed(
|
||||
self,
|
||||
texts: list[str],
|
||||
*,
|
||||
# Phase 129 (task 01): the per-batch position hook — ``None``
|
||||
# (every pre-phase caller) is a zero-cost guarded no-op.
|
||||
on_progress: Callable[[int, int], None] | None = None,
|
||||
) -> list[list[float]]: ...
|
||||
|
||||
async def chat(self, messages: list[dict[str, Any]], model: str | None = None) -> str: ...
|
||||
# ^ the one-shot completion the summarizer uses — the ``lite`` model
|
||||
@@ -419,6 +429,7 @@ async def import_sources(
|
||||
limit: int | None = None,
|
||||
session: Session | None = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None,
|
||||
@@ -442,6 +453,32 @@ async def import_sources(
|
||||
exceeds the limit, but ``total`` stays the full pre-walk count (an
|
||||
incomplete walk must not misreport the denominator).
|
||||
|
||||
``on_chunk_progress`` (phase 129, task 02) is an optional
|
||||
chunk-position hook for the file currently being indexed: while that
|
||||
file's content chunks embed, it is called — through
|
||||
:meth:`app.rag.llm.LLMClient.embed`'s ``on_progress`` — before EACH
|
||||
embed batch with ``(first_chunk_1based, total_chunks)``: the 1-based
|
||||
position of the first chunk of the in-flight batch and the file's
|
||||
total content-chunk count (batch granularity, the phase-129 A6
|
||||
assumption — the transport embeds token-budgeted batches per HTTP
|
||||
call). It carries the in-flight FILE's chunk position only, never
|
||||
the file identity — the per-file ``progress`` hook owns that: the
|
||||
caller pairs them (a ``progress`` call names the file, and every
|
||||
chunk position reported until the next ``progress`` call belongs to
|
||||
it; the position climbs through the file's batches and restarts at
|
||||
1 for the next file). Same "only when provided" rule as
|
||||
``progress``: ``None`` (every existing caller — the CLI, the chat
|
||||
path) is a zero-cost guarded no-op, and the run is byte-identical
|
||||
to a hook-less run. Only the content-chunk embeds report (the
|
||||
shared :func:`_replace_content_chunks` — text, image, and PDF
|
||||
branches; the image branch's one description chunk reports
|
||||
``1/1``, which IS its only chunk): the document's summary-chunk
|
||||
embed (:func:`_store_summary`) NEVER receives the hook, so the
|
||||
counter counts content chunks only (no ``1/1`` noise after the
|
||||
real progress ends), and the post-index enrichment pass / the
|
||||
PDF's embedded-image docs (separate documents, not the walked
|
||||
file) do not report either.
|
||||
|
||||
``ignore_by_root`` (phase 89, A1/A2) maps ``str(root)`` — the root path
|
||||
string exactly as passed in *sources* — to that source's RAW ignore-path
|
||||
lines (the importer normalizes them via ``_ignore_for_root``, the single
|
||||
@@ -574,6 +611,7 @@ async def import_sources(
|
||||
await _index_file(
|
||||
session, source=source, rel=rel, full_path=path, llm=llm,
|
||||
summary=summary, raw_date=dates_map.get(rel),
|
||||
on_chunk=on_chunk_progress,
|
||||
)
|
||||
except EmbeddingError as e:
|
||||
# A pathological file (e.g. content the embedding endpoint
|
||||
@@ -608,7 +646,13 @@ async def import_sources(
|
||||
|
||||
|
||||
async def _replace_content_chunks(
|
||||
session: Session, *, doc: Document, llm: Embedder, content: str, rel: str
|
||||
session: Session,
|
||||
*,
|
||||
doc: Document,
|
||||
llm: Embedder,
|
||||
content: str,
|
||||
rel: str,
|
||||
on_chunk: Callable[[int, int], None] | None = None,
|
||||
) -> int:
|
||||
"""Phase 1+2 on *content*: chunk it (format-aware, PLAN §5), replace
|
||||
the doc's content chunk rows (embeddings NULL), and embed the new
|
||||
@@ -622,6 +666,20 @@ async def _replace_content_chunks(
|
||||
verbatim by the text path (:func:`_index_file`), the image path
|
||||
(task 03), and the task-04 enrichment pass — one copy, three call
|
||||
sites.
|
||||
|
||||
``on_chunk`` (phase 129, task 02) is the in-flight file's
|
||||
chunk-position hook, forwarded as ``on_progress`` to the content
|
||||
embed: it fires before each top-level embed batch with
|
||||
``(first_chunk_1based, total_chunks)``. The token-cap re-chunk retry
|
||||
(the ``while True`` above that halves *target*) re-embeds the NEW
|
||||
(smaller-target) chunk list, and the hook reports against the NEW
|
||||
total automatically — no special case: the announced total is
|
||||
always the chunk list actually in flight. The empty-content
|
||||
``doc.chunks = []`` break never embeds, so the hook never fires for
|
||||
it. ``None`` (the default; every pre-phase call site — the CLI, the
|
||||
phase-125 enrichment pass, and the PDF's embedded-image docs, which
|
||||
index separate documents rather than the walked file) keeps the call
|
||||
byte-identical.
|
||||
"""
|
||||
settings = llm.settings
|
||||
target = max(400, settings.chunk_target_chars)
|
||||
@@ -635,7 +693,12 @@ async def _replace_content_chunks(
|
||||
if not doc.chunks:
|
||||
break
|
||||
try:
|
||||
vectors = await llm.embed([c.content for c in doc.chunks])
|
||||
# Phase 129 (task 02): the in-flight file's chunk-position
|
||||
# hook (``on_progress`` — the task-01 contract; ``None`` is a
|
||||
# zero-cost no-op).
|
||||
vectors = await llm.embed(
|
||||
[c.content for c in doc.chunks], on_progress=on_chunk
|
||||
)
|
||||
for row, vec in zip(doc.chunks, vectors, strict=True):
|
||||
row.embedding = vec
|
||||
break
|
||||
@@ -660,6 +723,7 @@ async def _index_file(
|
||||
llm: Embedder,
|
||||
summary: ImportSummary,
|
||||
raw_date: datetime | None = None,
|
||||
on_chunk: Callable[[int, int], None] | None = None,
|
||||
) -> None:
|
||||
"""Upsert one file: doc row + chunk rows + embeddings, one transaction.
|
||||
|
||||
@@ -668,6 +732,15 @@ async def _index_file(
|
||||
map, or ``None`` (every non-git case): the file's mtime is read
|
||||
here, once, and becomes the source date (the D2 fallback).
|
||||
|
||||
``on_chunk`` (phase 129, task 02) is the in-flight file's
|
||||
chunk-position hook (see :func:`import_sources`) — passed through to
|
||||
ALL THREE branches' content embeds: the text path's
|
||||
:func:`_replace_content_chunks` directly, the image branch via
|
||||
:func:`_index_image_file` (its one description chunk reports
|
||||
``1/1`` — it IS the file's only chunk), and the PDF branch via
|
||||
:func:`_index_pdf_file` (its content embed is the primary target).
|
||||
``None`` (the default) keeps every branch byte-identical.
|
||||
|
||||
Image files (phase 122, toggle on) delegate to
|
||||
:func:`_index_image_file` — the binary path (bytes digest,
|
||||
persistent copy, ``content`` = the vision description) — BEFORE
|
||||
@@ -683,7 +756,7 @@ async def _index_file(
|
||||
if match_extension(full_path, image_set) is not None:
|
||||
return await _index_image_file(
|
||||
session, source=source, rel=rel, full_path=full_path, llm=llm,
|
||||
summary=summary, raw_date=raw_date,
|
||||
summary=summary, raw_date=raw_date, on_chunk=on_chunk,
|
||||
)
|
||||
# Phase 126: PDF files take the PDF index path — AFTER the image
|
||||
# branch, BEFORE any text read (a PDF is binary; ``extract_pdf_text``
|
||||
@@ -691,7 +764,7 @@ async def _index_file(
|
||||
if full_path.suffix.lower() == ".pdf":
|
||||
return await _index_pdf_file(
|
||||
session, source=source, rel=rel, full_path=full_path, llm=llm,
|
||||
summary=summary, raw_date=raw_date,
|
||||
summary=summary, raw_date=raw_date, on_chunk=on_chunk,
|
||||
)
|
||||
content = full_path.read_text(encoding="utf-8", errors="replace").replace("\x00", "")
|
||||
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
|
||||
@@ -793,7 +866,7 @@ async def _index_file(
|
||||
# lives in the shared helper — the global policy stays intact for the
|
||||
# rest of the KB).
|
||||
chunks_n = await _replace_content_chunks(
|
||||
session, doc=doc, llm=llm, content=content, rel=rel
|
||||
session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
|
||||
)
|
||||
session.commit()
|
||||
|
||||
@@ -952,6 +1025,7 @@ async def _index_pdf_file(
|
||||
llm: Embedder,
|
||||
summary: ImportSummary,
|
||||
raw_date: datetime | None = None,
|
||||
on_chunk: Callable[[int, int], None] | None = None,
|
||||
) -> None:
|
||||
"""The phase-126 PDF branch of :func:`_index_file` — a PDF is
|
||||
indexed from its EXTRACTED TEXT (LOCKED A1/A2/A4), one transaction
|
||||
@@ -985,6 +1059,14 @@ async def _index_pdf_file(
|
||||
``raw_date`` follows the text path: the caller's git last-commit
|
||||
date or ``None`` (the mtime fallback, read before the unchanged
|
||||
early-return — the unchanged path refreshes from the same source).
|
||||
|
||||
``on_chunk`` (phase 129, task 02) is the in-flight file's
|
||||
chunk-position hook: it is forwarded to the PDF's CONTENT embed
|
||||
(the :func:`_replace_content_chunks` call — the primary target),
|
||||
but NOT to the embedded-image pass (those image docs are separate
|
||||
documents — their ``1/1`` positions would be noise after the
|
||||
file's real progress ends, the A6 exclusion). ``None`` keeps the
|
||||
branch byte-identical.
|
||||
"""
|
||||
# Extract the text (fail-soft: an unopenable PDF yields "" — the
|
||||
# document may still carry embedded images worth indexing).
|
||||
@@ -1066,9 +1148,11 @@ async def _index_pdf_file(
|
||||
# Phase 1+2 — the shared chunk/replace/embed pipeline (the page-
|
||||
# aware chunker dispatches on the ``.pdf`` suffix; empty content
|
||||
# clears the chunk rows and returns 0), committed atomically with
|
||||
# the document.
|
||||
# the document. Phase 129 (task 02): the in-flight file's
|
||||
# chunk-position hook rides the content embed (the embedded-image
|
||||
# pass does not receive it — separate documents, the A6 exclusion).
|
||||
chunks_n = await _replace_content_chunks(
|
||||
session, doc=doc, llm=llm, content=content, rel=rel
|
||||
session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
|
||||
)
|
||||
session.commit()
|
||||
|
||||
@@ -1815,6 +1899,7 @@ async def _index_image_file(
|
||||
llm: Embedder,
|
||||
summary: ImportSummary,
|
||||
raw_date: datetime | None = None,
|
||||
on_chunk: Callable[[int, int], None] | None = None,
|
||||
) -> None:
|
||||
"""The phase-122 image branch of :func:`_index_file` — a standalone
|
||||
image is indexed from its BYTES, never its text:
|
||||
@@ -1855,6 +1940,13 @@ async def _index_image_file(
|
||||
pre-phase). The UNCHANGED path never re-scans (the stored context
|
||||
is part of the row's committed content; a changed source state
|
||||
re-indexes the file through this path).
|
||||
|
||||
``on_chunk`` (phase 129, task 02) is the in-flight file's
|
||||
chunk-position hook: it is forwarded to the description's content
|
||||
embed (:func:`_replace_content_chunks`). A description (enriched or
|
||||
not) chunks to ONE chunk, so the hook reports ``1/1`` — acceptable,
|
||||
it IS the file's only chunk. ``None`` keeps the branch
|
||||
byte-identical.
|
||||
"""
|
||||
settings = llm.settings
|
||||
data = full_path.read_bytes()
|
||||
@@ -1996,8 +2088,10 @@ async def _index_image_file(
|
||||
# the chunk rows (embeddings NULL), embed, commit — one transaction
|
||||
# per file; the endpoint token-cap retry lives in the shared helper,
|
||||
# and a description is short, so it never fires in practice).
|
||||
# Phase 129 (task 02): the in-flight file's chunk-position hook
|
||||
# rides this embed (one description chunk → ``1/1``).
|
||||
chunks_n = await _replace_content_chunks(
|
||||
session, doc=doc, llm=llm, content=content, rel=rel
|
||||
session, doc=doc, llm=llm, content=content, rel=rel, on_chunk=on_chunk
|
||||
)
|
||||
session.commit()
|
||||
|
||||
|
||||
+34
-3
@@ -22,7 +22,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from collections.abc import AsyncGenerator
|
||||
from collections.abc import AsyncGenerator, Callable
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Any, Literal, cast
|
||||
|
||||
@@ -343,12 +343,43 @@ class LLMClient:
|
||||
batches.append(cur)
|
||||
return batches
|
||||
|
||||
async def embed(self, texts: list[str]) -> list[list[float]]:
|
||||
"""Embed *texts* in token-budgeted batches (order always kept)."""
|
||||
async def embed(
|
||||
self,
|
||||
texts: list[str],
|
||||
*,
|
||||
on_progress: Callable[[int, int], None] | None = None,
|
||||
) -> list[list[float]]:
|
||||
"""Embed *texts* in token-budgeted batches (order always kept).
|
||||
|
||||
Progress hook (phase 129, task 01): when *on_progress* is given,
|
||||
it is called once per TOP-LEVEL batch, BEFORE the batch's request
|
||||
(a failing batch still announces its start), with
|
||||
``(first_chunk_1based, total_chunks)`` — the 1-based index of the
|
||||
first text of the in-flight batch and the total number of texts
|
||||
in this call — so a caller can render ``chunk n/m`` (the importer
|
||||
passes this through for the sync label; PLAN §7.4 "never stale").
|
||||
|
||||
``len(out) + 1`` is that first index: each batch's vectors extend
|
||||
the accumulator 1:1, so ``len(out)`` is exactly the number of
|
||||
chunks embedded by earlier batches. The :meth:`_embed_batch`
|
||||
halving backstop does NOT sub-report: a halved batch leaves the
|
||||
announced position approximate (the hook already fired for the
|
||||
original batch's start) — documented; the backstop is rare by
|
||||
construction (phase 114's ``_TooLarge`` handling). ``None`` (the
|
||||
default — every pre-phase caller: the chat retrieval path,
|
||||
:meth:`embed_one`, the summarizer, the importer) is a zero-cost
|
||||
guarded no-op: call sequence and return value are
|
||||
byte-identical to the pre-hook client.
|
||||
"""
|
||||
if not texts:
|
||||
return []
|
||||
out: list[list[float]] = []
|
||||
for chunk in self._batch_texts(texts):
|
||||
# Phase 129 (task 01): announce THIS batch's start before the
|
||||
# request — the hook precedes the try, so even a failing batch
|
||||
# reports its position.
|
||||
if on_progress is not None:
|
||||
on_progress(len(out) + 1, len(texts))
|
||||
try:
|
||||
vecs = await self._embed_batch(chunk)
|
||||
except EmbeddingError:
|
||||
|
||||
Reference in New Issue
Block a user