phase: 129_sync_chunk_progress
All green. Final report:
**Phase 129 final verification pass — 100% green** (previous work intact; this pass re-verified everything and fixed two pre-existing README inconsistencies)
- Verified on-spec: `LLMClient.embed` per-batch `on_progress` hook; importer `on_chunk_progress` plumbing (text/image/PDF branches; summary embed excluded); `SyncStatus.current_chunk/chunks_total` reset+clear contract; UI `.sync-label-chunk` pinned part with untruncated `title`/`#sync-result`
- `uv run pytest --cov=app -q`: exit 0, ~3041 passed, 0 failed, `app/` coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` (isolated, DB up): **2 passed** (endpoint chunk climb with true `chunk_document` total + UI `· chunk n/m` part)
- `uv run ruff check .`: all checks passed · `uv run pyright`: 0 errors, 0 warnings
- Completion criteria: all 5 ✅ (incl. no behavior change in completed phases — full suite green, CLI/chat no-op paths byte-identical)
- Docs: README.md fixed 2 pre-existing inconsistencies (upload section claimed "unpacks + scans immediately" — phase 90 made it unpack+register only, scan deferred to Sync; troubleshooting claimed "120s client timeout" — actual guard is the 300 s pre-token `TURN_TIMEOUT_MS`). Phase-129 label behavior already documented. AGENTS.md re-verified consistent — no changes
- Removed stray untracked `NOTES.md` (8-byte artifact); no live-infra changes; no commits — all changes left in working tree for the harness
- Next pending phase: `130_chat_answer_clusters`
This commit is contained in:
+96
-6
@@ -112,6 +112,21 @@ Terminal states (success AND failed) clear ``phase`` +
|
||||
``current_summary`` but keep the run's final ``summaries_done`` /
|
||||
``summaries_total`` (the phase-64 keep-final-counts convention).
|
||||
|
||||
Phase 129 (task 03) adds the in-flight file's EMBED-chunk position —
|
||||
``current_chunk`` (the 1-based first chunk of the in-flight
|
||||
token-budgeted batch) + ``chunks_total`` (the file's total content
|
||||
chunks), batch granularity via the :meth:`app.rag.llm.LLMClient.embed`
|
||||
``on_progress`` hook (tasks 01/02). They are 0/0 before the first
|
||||
batch (the prelude), during the ``"overview"`` / ``"summaries"``
|
||||
phases, and in terminal states — the import phase clearing them at
|
||||
EVERY point it ends (the ``phase = "overview"`` and
|
||||
``phase = "summaries"`` transitions and both terminals, the same spots
|
||||
``current_file`` clears). Unlike the phase-64 FILE counts, the chunk
|
||||
position is a SUB-position of a file that's done — the keep-final-
|
||||
counts convention does NOT extend to it: the file counts survive the
|
||||
terminal for the result line's context, the chunk position clears
|
||||
with the file.
|
||||
|
||||
The ``failed`` state's ``error`` string is masked by the shared
|
||||
sanitizer — the ``user:pass@`` masker now lives in :mod:`app.core.errors`
|
||||
(imported here under the private name ``_sanitize_error``).
|
||||
@@ -178,6 +193,19 @@ class SyncStatus:
|
||||
Terminal states clear ``phase`` + ``current_summary`` but keep the
|
||||
run's final summary counts (the phase-64 keep-final-counts
|
||||
convention).
|
||||
|
||||
Phase 129 (task 03) chunk-position fields: ``current_chunk`` is
|
||||
the 1-based first chunk of the in-flight token-budgeted batch and
|
||||
``chunks_total`` the in-flight file's total content chunks (batch
|
||||
granularity, the ``LLMClient.embed`` hook — the module docstring's
|
||||
contract). They are 0/0 before the first batch, during the
|
||||
``"overview"`` / ``"summaries"`` phases, and in terminal states —
|
||||
the import phase clears them at every point it ends (the
|
||||
overview/summaries transitions + both terminals, where
|
||||
``current_file`` also clears). The phase-64 keep-final-counts
|
||||
convention covers the FILE counts only — the chunk position is a
|
||||
sub-position of a file that's done, so it clears in the terminal
|
||||
while ``files_done`` / ``files_total`` keep their final values.
|
||||
"""
|
||||
|
||||
state: Literal["idle", "running", "success", "failed"] = "idle"
|
||||
@@ -197,6 +225,13 @@ class SyncStatus:
|
||||
current_summary: str | None = None
|
||||
summaries_done: int = 0
|
||||
summaries_total: int = 0
|
||||
# Phase 129 (task 03): the current file's EMBED chunk position —
|
||||
# the first chunk of the in-flight token-budgeted batch and the
|
||||
# file's total content chunks (the LLMClient.embed hook, batch
|
||||
# granularity). 0/0 before the first batch, during the
|
||||
# prelude/overview/summaries phases, and in terminal states.
|
||||
current_chunk: int = 0
|
||||
chunks_total: int = 0
|
||||
|
||||
|
||||
_status = SyncStatus()
|
||||
@@ -211,11 +246,17 @@ def sync_status() -> dict[str, Any]:
|
||||
``current_file`` (phase 64) is the ``source/relative/path`` the
|
||||
import is processing right now — null during the clone/pull phase
|
||||
and in terminal states; ``files_done`` / ``files_total`` carry the
|
||||
hook's position (0/0 idle). ``phase`` (phase 98) is the pipeline
|
||||
stage — null in the prelude and terminal states; while in
|
||||
``"summaries"``, ``current_summary`` + ``summaries_done`` /
|
||||
``summaries_total`` carry the generator's position (the terminal
|
||||
keeps the final summary counts).
|
||||
hook's position (0/0 idle) and keep their final values in the
|
||||
terminal. ``current_chunk`` / ``chunks_total`` (phase 129) are the
|
||||
in-flight file's embed-chunk position (batch granularity — the
|
||||
first chunk of the in-flight batch over the file's total content
|
||||
chunks): 0/0 before the first batch, in the prelude/overview/
|
||||
summaries phases, and in terminal states (they clear at every
|
||||
point the import phase ends; the FILE counts do not). ``phase``
|
||||
(phase 98) is the pipeline stage — null in the prelude and
|
||||
terminal states; while in ``"summaries"``, ``current_summary`` +
|
||||
``summaries_done`` / ``summaries_total`` carry the generator's
|
||||
position (the terminal keeps the final summary counts).
|
||||
"""
|
||||
return {
|
||||
"state": _status.state,
|
||||
@@ -226,6 +267,12 @@ def sync_status() -> dict[str, Any]:
|
||||
"current_file": _status.current_file,
|
||||
"files_done": _status.files_done,
|
||||
"files_total": _status.files_total,
|
||||
# Phase 129 (task 03): the in-flight file's embed-chunk
|
||||
# position — the dataclass defaults make idle 0/0, and the
|
||||
# runner clears it at every point the import phase ends
|
||||
# (overview/summaries transitions + both terminals).
|
||||
"current_chunk": _status.current_chunk,
|
||||
"chunks_total": _status.chunks_total,
|
||||
# Phase 98 (task 01): the phase machine — null/0/0 idle (the
|
||||
# dataclass defaults) and in terminal states (which clear
|
||||
# phase + current_summary but keep the final summary counts).
|
||||
@@ -278,6 +325,10 @@ async def _run_sync() -> None:
|
||||
_status.current_summary = None
|
||||
_status.summaries_done = 0
|
||||
_status.summaries_total = 0
|
||||
# Phase 129 (task 03): the chunk-position fields reset with the
|
||||
# run — 0/0 until the import's first embed batch reports.
|
||||
_status.current_chunk = 0
|
||||
_status.chunks_total = 0
|
||||
try:
|
||||
settings = get_settings()
|
||||
# Step 1 (phase 41): fail fast — verify both models the sync
|
||||
@@ -359,18 +410,35 @@ async def _run_sync() -> None:
|
||||
_status.files_done = done
|
||||
_status.files_total = total
|
||||
|
||||
# Phase 129 (task 03): the in-flight file's embed-chunk
|
||||
# position — the LLMClient.embed hook (task 01) fires before
|
||||
# each token-budgeted batch with (first_chunk_1based, total);
|
||||
# the file's IDENTITY is the sibling ``progress`` hook's (the
|
||||
# two hooks are paired — a progress call names the file, and
|
||||
# every chunk position until the next one belongs to it).
|
||||
def _chunk_hook(chunk: int, total: int) -> None:
|
||||
_status.current_chunk = chunk
|
||||
_status.chunks_total = total
|
||||
|
||||
# Phase 98 (task 01): the phase machine — the import phase
|
||||
# starts NOW (the model-check + clone/pull prelude above
|
||||
# reported ``phase: null``).
|
||||
_status.phase = "import"
|
||||
summary: ImportSummary = await import_sources(
|
||||
sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root,
|
||||
sources, llm, prune=True, progress=_hook, on_chunk_progress=_chunk_hook,
|
||||
ignore_by_root=ignore_by_root,
|
||||
include_hidden_by_root=include_hidden_by_root,
|
||||
doc_dates_by_root=doc_dates_by_root,
|
||||
)
|
||||
overview = False
|
||||
if summary.added + summary.updated > 0:
|
||||
_status.phase = "overview" # phase 98 (task 01)
|
||||
# Phase 129 (task 03): the import phase ends here — the
|
||||
# chunk position (a sub-position of a done file) clears
|
||||
# with the phase change; the file counts keep their final
|
||||
# values (the phase-64 convention covers files only).
|
||||
_status.current_chunk = 0
|
||||
_status.chunks_total = 0
|
||||
overview = await regenerate_overview(llm)
|
||||
# Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03):
|
||||
# the folder summaries — the drill-down ls's per-level
|
||||
@@ -419,6 +487,11 @@ async def _run_sync() -> None:
|
||||
# position through the (long) span (its total is the
|
||||
# affected count).
|
||||
_status.phase = "summaries"
|
||||
# Phase 129 (task 03): the import phase ends here too
|
||||
# (the changed-KB branch skips no phase) — same
|
||||
# chunk-position clear as the overview transition.
|
||||
_status.current_chunk = 0
|
||||
_status.chunks_total = 0
|
||||
folder_stats = await generate_folder_summaries(
|
||||
fs_db, llm, changed_docs=summary.changed_paths,
|
||||
on_progress=_summary_hook,
|
||||
@@ -435,6 +508,11 @@ async def _run_sync() -> None:
|
||||
# phase + hook (the fill's total is the missing
|
||||
# count — the hook's loop-start ``total``).
|
||||
_status.phase = "summaries"
|
||||
# Phase 129 (task 03): the import phase ends here
|
||||
# too (the unchanged-walk gap-fill branch) — same
|
||||
# chunk-position clear.
|
||||
_status.current_chunk = 0
|
||||
_status.chunks_total = 0
|
||||
folder_stats = await generate_folder_summaries(
|
||||
fs_db, llm, only_missing=True, on_progress=_summary_hook
|
||||
)
|
||||
@@ -475,6 +553,12 @@ async def _run_sync() -> None:
|
||||
_status.current_file = None # phase 64: keep the final counts
|
||||
_status.phase = None # phase 98: keep the final summary counts
|
||||
_status.current_summary = None
|
||||
# Phase 129 (task 03): the chunk position is a sub-position of
|
||||
# a file that's done — it clears in the terminal (unlike the
|
||||
# FILE counts, which keep their final values for the result
|
||||
# line's context).
|
||||
_status.current_chunk = 0
|
||||
_status.chunks_total = 0
|
||||
_status.detail = {
|
||||
"files": summary.files,
|
||||
"added": summary.added,
|
||||
@@ -498,3 +582,9 @@ async def _run_sync() -> None:
|
||||
_status.current_file = None # phase 64: keep the final counts
|
||||
_status.phase = None # phase 98: keep the final summary counts
|
||||
_status.current_summary = None
|
||||
# Phase 129 (task 03): the chunk position clears in the
|
||||
# failed terminal too (the convention split — the FILE counts
|
||||
# keep their final values next to the error, the sub-position
|
||||
# of a file that's done/abandoned does not).
|
||||
_status.current_chunk = 0
|
||||
_status.chunks_total = 0
|
||||
|
||||
Reference in New Issue
Block a user