phase: 152_sync_progress_image_pass
Build and Push Containers / build-and-push-app (push) Successful in 3m34s
Build and Push Containers / build-and-push-db (push) Successful in 23s

All criteria verified. The CLI path is hook-less (only `app/api/sync.py` passes the hook), and the byte-identity is pinned by `test_on_image_progress_none_run_is_byte_identical`.

## Final Report — Phase 152 final verification pass

- **Verified (no new code needed):** `SyncStatus.images_done/images_total` + reset/clear rules in `app/api/sync.py`; `on_image_progress` threaded through the importer's three PDF vision spans with the continuous per-file counter; the `_image_hook` closure; `sources.js` `· image n/m` label part (box + title + aria).
- **C1** status climbs 1→N + UI shows `· image n/m`: ✅ unit 3/15→15/15, integration 1/15→7/15, E2E 1/4→4/4 (title/aria ride the full string)
- **C2** never frozen: ✅ E2E asserts ≥3 distinct positions, starting at 1, non-decreasing, ≤ total, reaching it
- **C3** cleared at overview/summaries + all terminals; per-file reset via `_hook`; no stale position for image-less files: ✅ unit pins + E2E settle (image part gone in box/title/aria, endpoint 0/0)
- **C4** `/api/sync/status` additive-only: ✅ full-dict pin passes (all pre-existing keys unchanged + the two new 0/0 keys)
- **C5** hook-less caller byte-identical: ✅ CLI passes no hook; `test_on_image_progress_none_run_is_byte_identical` green
- **C6** `uv run pytest`: **3526 passed, exit 0** (95s); dedicated E2E in isolation: **2 passed** (95s); `uv run ruff check . && uv run pyright`: **clean**; coverage: **99% on `app/` (>90% gate)**
- Regression: all five neighbouring E2E suites green in isolation (progress_box 8, chunk_progress 2, label_mobile 4, pdf_support 6, image_documents 4)
- **C7** atomic commit: left to the harness (no git index/history touched per rules)
- **Docs:** README.md — sync-label paragraph already updated this phase, accurately; all `.env.example` vars documented; routes/commands verified — **no changes needed**. AGENTS.md — all references (PLAN, user_stories, test_smoke.py, .env.example, gpg config, .gitignore) verified — **no changes needed**
- **Defects found in prior work:** none (one prior env-leak defect was already fixed inside this phase's task 04, conftest)
- No live-infrastructure touched; no ntfy used. **Next pending phase:** none — `todo/` held only this phase
This commit is contained in:
2026-10-03 01:52:41 -04:00
parent c7cfcd26b5
commit 12fa69b7ea
39 changed files with 3971 additions and 133 deletions
+124 -1
View File
@@ -188,6 +188,23 @@ counts convention does NOT extend to it: the file counts survive the
terminal for the result line's context, the chunk position clears
with the file.
Phase 152 (task 01) adds the in-flight file's VISION-image position —
``images_done`` (the 1-based index of the image about to be described
and embedded) + ``images_total`` (the file's total images for this
run — the three PDF vision spans of ONE file share ONE continuous
counter per file; task 02 wires the importer's
``on_image_progress`` hook, task 03 the runner's closure that feeds
these fields). They are 0/0 before the first image, during the
``"overview"`` / ``"summaries"`` phases, and in terminal states — the
same spots ``current_chunk`` clears. Unlike the chunk position (every
file embeds chunks), a file may have NO images at all, so the per-file
``progress`` hook ALSO resets them to 0/0: the pairing rule, extended
— a ``progress`` call names the file and clears the image position,
and every image position until the next ``progress`` call belongs to
it (a file with no images must not inherit the previous PDF's
position). ASSUMPTION A5: no new ``phase`` value — the image spans
live inside ``phase: "import"``.
The ``failed`` state's ``error`` string is masked by the shared
sanitizer — the ``user:pass@`` masker now lives in :mod:`app.core.errors`
(imported here under the private name ``_sanitize_error``).
@@ -272,6 +289,21 @@ class SyncStatus:
convention covers the FILE counts only — the chunk position is a
sub-position of a file that's done, so it clears in the terminal
while ``files_done`` / ``files_total`` keep their final values.
Phase 152 (task 01) image-position fields: ``images_done`` is
the 1-based index of the vision image about to be described and
``images_total`` the in-flight file's total images for this run
(the importer's ``on_image_progress`` hook — the three PDF vision
spans of one file share ONE continuous counter, the module
docstring's contract). Like the chunk position, the image
position is a SUB-position of a file that's done: the
keep-final-counts convention covers the FILE counts only, so it
clears with the file — 0/0 before the first image, during the
``"overview"`` / ``"summaries"`` phases, and in terminal states
(the same spots the chunk position clears) — and the per-file
``progress`` hook resets it to 0/0 (a ``progress`` call names the
file and clears the image position; a file with no images must
not inherit the previous PDF's position).
"""
state: Literal["idle", "running", "success", "failed", "cancelled"] = "idle"
@@ -298,6 +330,14 @@ class SyncStatus:
# prelude/overview/summaries phases, and in terminal states.
current_chunk: int = 0
chunks_total: int = 0
# Phase 152 (task 01): the in-flight file's VISION-image position —
# the 1-based index of the image about to be described and the
# file's total images for this run (the importer's
# on_image_progress hook; the three PDF vision spans share ONE
# continuous counter per file). 0/0 before the first image, during
# the prelude/overview/summaries phases, and in terminal states.
images_done: int = 0
images_total: int = 0
_status = SyncStatus()
@@ -334,6 +374,15 @@ def sync_status() -> dict[str, Any]:
user-cancelled run — a terminal state with ``error`` null and
``detail`` empty; the run's final file + summary counts are kept,
the chunk position cleared.
Phase 152 (task 01): ``images_done`` / ``images_total`` are the
in-flight file's vision-image position (the importer's
``on_image_progress`` hook — the three PDF vision spans of one
file share ONE continuous counter): 0/0 before the first image, in
the prelude/overview/summaries phases, and in terminal states; the
per-file ``progress`` hook clears it at every file boundary (a
file with no images never shows a stale position). Additive only —
every pre-existing key keeps its name, type, and meaning.
"""
return {
"state": _status.state,
@@ -350,6 +399,15 @@ def sync_status() -> dict[str, Any]:
# (overview/summaries transitions + every terminal).
"current_chunk": _status.current_chunk,
"chunks_total": _status.chunks_total,
# Phase 152 (task 01): the in-flight file's VISION-image
# position (the importer's on_image_progress hook — the three
# PDF vision spans share ONE continuous counter per file):
# 0/0 before the first image, during the
# prelude/overview/summaries phases, and in terminal states
# (cleared at every point the import phase ends; the per-file
# progress hook re-clears it at every file boundary).
"images_done": _status.images_done,
"images_total": _status.images_total,
# Phase 98 (task 01): the phase machine — null/0/0 idle (the
# dataclass defaults) and in terminal states (which clear
# phase + current_summary but keep the final summary counts).
@@ -461,6 +519,11 @@ async def _run_sync() -> None:
# run — 0/0 until the import's first embed batch reports.
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image-position fields reset with the
# run — 0/0 until the import's first vision span reports (the
# per-file progress hook re-clears them at every file boundary).
_status.images_done = 0
_status.images_total = 0
try:
settings = get_settings()
# Step 1 (phase 41): fail fast — verify both models the sync
@@ -556,6 +619,15 @@ async def _run_sync() -> None:
_status.current_file = f"{source}/{rel}"
_status.files_done = done
_status.files_total = total
# Phase 152 (task 01): the pairing rule, extended — a
# progress call names the file AND clears the image
# position: every image position until the next progress
# call belongs to this file. A file with no images must
# not inherit the previous PDF's position (unlike the
# chunk position, not every file embeds chunks, so the
# run-start reset alone cannot cover it).
_status.images_done = 0
_status.images_total = 0
# Phase 129 (task 03): the in-flight file's embed-chunk
# position — the LLMClient.embed hook (task 01) fires before
@@ -567,6 +639,21 @@ async def _run_sync() -> None:
_status.current_chunk = chunk
_status.chunks_total = total
# Phase 152 (task 03): the in-flight file's VISION-image
# position — the importer's ``on_image_progress`` hook (task
# 02) fires before each of the PDF's vision describe calls
# with (position_1based, total), the file's ONE continuous
# counter across its three vision spans (the importer owns
# the continuity — this closure only lands the pair); the
# file's IDENTITY is the sibling ``progress`` hook's, which
# ALSO resets this position (the three hooks are paired:
# ``progress`` names the file and clears the image position,
# ``on_chunk_progress`` reports the embed-batch position,
# ``on_image_progress`` reports the vision-image position).
def _image_hook(done: int, total: int) -> None:
_status.images_done = done
_status.images_total = total
# Phase 98 (task 01): the phase machine — the import phase
# starts NOW (the model-check + clone/pull prelude above
# reported ``phase: null``).
@@ -577,7 +664,12 @@ async def _run_sync() -> None:
# imported.
_check_cancelled()
summary: ImportSummary = await import_sources(
sources, llm, prune=True, progress=_hook, on_chunk_progress=_chunk_hook,
sources,
llm,
prune=True,
progress=_hook,
on_chunk_progress=_chunk_hook,
on_image_progress=_image_hook, # phase 152 (task 03)
ignore_by_root=ignore_by_root,
include_hidden_by_root=include_hidden_by_root,
doc_dates_by_root=doc_dates_by_root,
@@ -596,6 +688,11 @@ async def _run_sync() -> None:
# values (the phase-64 convention covers files only).
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image position clears at the
# same point (a sub-position of a done file, like the
# chunk position).
_status.images_done = 0
_status.images_total = 0
overview = await regenerate_overview(llm)
# Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03):
# the folder summaries — the drill-down ls's per-level
@@ -649,6 +746,10 @@ async def _run_sync() -> None:
# chunk-position clear as the overview transition.
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image position clears at
# the same point (a sub-position of a done file).
_status.images_done = 0
_status.images_total = 0
folder_stats = await generate_folder_summaries(
fs_db, llm, changed_docs=summary.changed_paths,
on_progress=_summary_hook,
@@ -670,6 +771,11 @@ async def _run_sync() -> None:
# chunk-position clear.
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image position clears
# at the same point (a sub-position of a done
# file).
_status.images_done = 0
_status.images_total = 0
folder_stats = await generate_folder_summaries(
fs_db, llm, only_missing=True, on_progress=_summary_hook
)
@@ -716,6 +822,11 @@ async def _run_sync() -> None:
# line's context).
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image position is a sub-position of
# a file that's done — it clears in the terminal too (the FILE
# counts keep their final values, the sub-position does not).
_status.images_done = 0
_status.images_total = 0
_status.detail = {
"files": summary.files,
"added": summary.added,
@@ -754,6 +865,12 @@ async def _run_sync() -> None:
# success and failed (the convention split, pinned).
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image position clears in the
# CANCELLED terminal too — every terminal clears it (a
# sub-position of a file that's done/abandoned, the FILE
# counts above keep their final values).
_status.images_done = 0
_status.images_total = 0
logger.info(
"sync: cancelled by user state=cancelled files_done=%d files_total=%d",
_status.files_done, _status.files_total,
@@ -773,3 +890,9 @@ async def _run_sync() -> None:
# of a file that's done/abandoned does not).
_status.current_chunk = 0
_status.chunks_total = 0
# Phase 152 (task 01): the image position clears in the FAILED
# terminal too — the convention split (a sub-position of a
# file that's done/abandoned; the FILE counts keep their final
# values next to the error).
_status.images_done = 0
_status.images_total = 0