phase: 152_sync_progress_image_pass
Build and Push Containers / build-and-push-app (push) Successful in 3m34s
Build and Push Containers / build-and-push-db (push) Successful in 23s

All criteria verified. The CLI path is hook-less (only `app/api/sync.py` passes the hook), and the byte-identity is pinned by `test_on_image_progress_none_run_is_byte_identical`.

## Final Report — Phase 152 final verification pass

- **Verified (no new code needed):** `SyncStatus.images_done/images_total` + reset/clear rules in `app/api/sync.py`; `on_image_progress` threaded through the importer's three PDF vision spans with the continuous per-file counter; the `_image_hook` closure; `sources.js` `· image n/m` label part (box + title + aria).
- **C1** status climbs 1→N + UI shows `· image n/m`: ✅ unit 3/15→15/15, integration 1/15→7/15, E2E 1/4→4/4 (title/aria ride the full string)
- **C2** never frozen: ✅ E2E asserts ≥3 distinct positions, starting at 1, non-decreasing, ≤ total, reaching it
- **C3** cleared at overview/summaries + all terminals; per-file reset via `_hook`; no stale position for image-less files: ✅ unit pins + E2E settle (image part gone in box/title/aria, endpoint 0/0)
- **C4** `/api/sync/status` additive-only: ✅ full-dict pin passes (all pre-existing keys unchanged + the two new 0/0 keys)
- **C5** hook-less caller byte-identical: ✅ CLI passes no hook; `test_on_image_progress_none_run_is_byte_identical` green
- **C6** `uv run pytest`: **3526 passed, exit 0** (95s); dedicated E2E in isolation: **2 passed** (95s); `uv run ruff check . && uv run pyright`: **clean**; coverage: **99% on `app/` (>90% gate)**
- Regression: all five neighbouring E2E suites green in isolation (progress_box 8, chunk_progress 2, label_mobile 4, pdf_support 6, image_documents 4)
- **C7** atomic commit: left to the harness (no git index/history touched per rules)
- **Docs:** README.md — sync-label paragraph already updated this phase, accurately; all `.env.example` vars documented; routes/commands verified — **no changes needed**. AGENTS.md — all references (PLAN, user_stories, test_smoke.py, .env.example, gpg config, .gitignore) verified — **no changes needed**
- **Defects found in prior work:** none (one prior env-leak defect was already fixed inside this phase's task 04, conftest)
- No live-infrastructure touched; no ntfy used. **Next pending phase:** none — `todo/` held only this phase
This commit is contained in:
2026-10-03 01:52:41 -04:00
parent c7cfcd26b5
commit 12fa69b7ea
39 changed files with 3971 additions and 133 deletions
+567 -15
View File
@@ -110,6 +110,56 @@ poll tree's new 5th branch (after failed, before the upload branches
— the 7-branch tree is now 8), the neutral ``applySyncCancelled``
(no modal, no banner, no error styling), and the
``initSyncButton`` re-attach of a ``cancelled`` run.
Phase 152 (task 01): the status contract gains the in-flight file's
VISION-image position — ``images_done`` (the 1-based index of the
image about to be described) + ``images_total`` (the file's total
images for this run — the three PDF vision spans share ONE continuous
counter per file, the importer's ``on_image_progress`` hook that
task 02/03 wire). The idle dict pins them 0/0; the per-file reset is
the pairing rule, extended — a ``progress`` call names the file AND
clears the image position, pinned through the runner's own ``_hook``
closure (the two-file fake below: the position fired through the
runner's own ``_image_hook`` closure while file A is in flight is
0/0 when file B is named — a file with no images must not inherit
the previous PDF's position); the clear rule mirrors the chunk
position — 0/0 at the overview/summaries transitions and in EVERY
terminal (success, failed, and cancelled — each pinned with a live
position in flight; task 04 moves every live position below onto the
runner's OWN closures, the phase-64/129 idiom), while the FILE
counts keep their final values (the convention split: a
sub-position of a file that's done clears with the file).
Phase 152 (task 03): the runner's THIRD paired closure —
``_image_hook`` (beside ``_hook`` and ``_chunk_hook``) — lands the
importer's ``on_image_progress`` pairs on the status, and the import
call now passes ``on_image_progress=_image_hook`` (the wiring pin:
the fakes receive the kwarg and record the live closure). The
frontend gains the "· image n/m" label PART: the running entry
destructures ``image = ""`` and the writer call becomes
``setSyncLabel(main, count, chunk, true, image)`` — the image part
rides ONLY the composed full string (box + title + running
aria-label — no new button span; the running button is the
spinner), the chunk part keeps its own span, and the bare builders
+ non-running calls pass nothing (both parts default "").
Phase 152 (task 04): the position pins move onto the runner's OWN
closures (the phase-64/129 idiom — the fakes fire the image pairs
through the ``on_image_progress`` kwarg, the ``_image_hook`` under
test, instead of standing in on the status object): a mid-run pair
(3/15, then 15/15) climbs on the status BETWEEN ticks while
``current_file`` and the chunk position keep their own values (the
THREE positions are independent sub-positions of the file), the
per-file reset, the overview/summaries transition clears, and the
FAILED + CANCELLED terminals all fire their live positions through
the closure too. The frontend source pins the ``fmtSyncPhaseLabel``
image part (present iff ``images_total > 0`` — the exact guard +
template), the ``overview`` / ``summaries`` branches' ``image: ""``
(unchanged labels), and ``setSyncLabel``'s composed ``full`` string
byte-for-byte (a status carrying both positions reads "… (2540/2540)
· chunk 125/126 · image 3/15" on the box + title + running
aria-label — the runtime string is behaviorally pinned in
tests/unit/test_frontend_sync_upload.py's node check).
"""
from __future__ import annotations
@@ -393,13 +443,16 @@ def test_sources_js_running_state_is_never_stale() -> None:
Phase 128 (task 01): the entry hands the label PARTS (the
phase-aware split — sync + status → fmtSyncPhaseLabel, the
fall-through → the byte-identical fmtSyncLabel) to the SINGLE
writer, setSyncLabel(main, count, chunk — phase 129: the chunk
part destructures with a "" default), which lands the main part,
writer, setSyncLabel(main, count, chunk, running, image — phase
129: the chunk part destructures with a "" default, phase 152:
the image part does too), which lands the main part,
the pinned count part (hidden when absent), the pinned chunk part
(phase 129 — hidden when absent), and the FULL untruncated label
on the button title + #sync-progress (A4) — the direct span /
title / announcer writes are GONE from the entry (they live in
setSyncLabel + setSyncProgress). Phase 149 (task 01): the writer call passes
on the button title + #sync-progress (A4 — phase 152: with the
image part appended after the chunk part when a PDF's vision span
is in flight; the image part gets no span of its own) — the
direct span / title / announcer writes are GONE from the entry
(they live in setSyncLabel + setSyncProgress). Phase 149 (task 01): the writer call passes
`running = true` (the detail box reveals with the full label),
and the button's accessible name rides in the writer too (the
full live label — the icon-only running button keeps a name);
@@ -414,8 +467,10 @@ def test_sources_js_running_state_is_never_stale() -> None:
assert "fmtSyncLabel(kind, currentFile, done, total)" in body
# …and the single writer lands the parts (A4 in it — the title is
# set for EVERY running label: the file-only rule is adjusted
# away) with the phase-149 `running` flag (the box reveals).
assert "setSyncLabel(main, count, chunk, true)" in body
# away) with the phase-149 `running` flag (the box reveals) +
# the phase-152 image part (rides only the composed full string —
# no span).
assert "setSyncLabel(main, count, chunk, true, image)" in body
assert body.count("setSyncLabel(") == 1, "the entry writes ONLY through the parts writer"
assert "syncBtn.title =" not in body, ("the title write lives in setSyncLabel (A4)")
assert "syncLabel.textContent" not in body, ("no direct span write — the parts")
@@ -433,6 +488,70 @@ def test_sources_js_running_state_is_never_stale() -> None:
)
def test_fmt_sync_phase_label_image_part_is_pinned_source_level() -> None:
"""Phase 152 (task 04): the frontend SOURCE pins the "· image
n/m" label part (the browser behavior is pinned in
tests/unit/test_frontend_sync_upload.py's node check — the REAL
extracted functions run against stub refs; this pin keeps the
template itself from drifting without a browser):
* the import fall-through is the ONLY reader of the status's
image fields, guarded on ``images_total > 0`` (the part is
present IFF ``images_total > 0`` — 0/0 hides it, the exact
guard + template + empty false-branch);
* the ``overview`` / ``summaries`` branches return ``image: ""``
(the position belongs to the import phase only — those labels
stay unchanged by the phase);
* the running entry destructures the part with a `""` default and
hands it to the single writer as the FIFTH argument (after the
`running` flag);
* ``setSyncLabel``'s composed ``full`` string appends the image
part LAST (after the chunk part) — so a status carrying both
positions reads exactly "Syncing… <file> (2540/2540) · chunk
125/126 · image 3/15" on the box, the title, and the running
aria-label (byte-for-byte; the docstring's worked example
carries the shape).
"""
js = _text(SOURCES_JS)
builder = _body(js, "fmtSyncPhaseLabel")
# The import fall-through: present IFF images_total > 0 — the
# exact guard + template (the phase-129 chunk part's sibling,
# same shape).
assert (
"status.images_total > 0\n"
" ? `· image ${status.images_done}/${status.images_total}`\n"
" : \"\";"
) in builder
assert "return { main, count, chunk, image };" in builder
# The overview branch: the unchanged label + the part absent.
assert (
'return { main: "Writing KB overview…", count: "", chunk: "", image: "" };'
) in builder
# The summaries branch: the unchanged label + the part absent
# (the phase's own (n/m) is its position).
assert "image: \"\", // the image position belongs to the import phase only" in builder
assert "`Summarizing folders…${folder}`" in builder
# The running entry: the part destructures with a `""` default
# and rides the writer call as the fifth argument.
entry = _body(js, "enterSyncRunningState")
assert 'const { main, count, chunk = "", image = "" }' in entry
assert "setSyncLabel(main, count, chunk, true, image)" in entry
# The writer: the composed full string — the image part LAST,
# after the chunk part (the "… · chunk 125/126 · image 3/15"
# shape the docstring's worked example carries).
writer = _body(js, "setSyncLabel")
assert (
"const full =\n"
" mainText +\n"
" (countText ? ` ${countText}` : \"\") +\n"
" (chunkText ? ` ${chunkText}` : \"\") +\n"
" (imageText ? ` ${imageText}` : \"\");"
) in writer
assert "· chunk 125/126 · image 3/15" in js, (
"the worked example (both positions at once) is documented"
)
def test_sources_js_terminal_states() -> None:
"""Terminal rendering: success → enabled + 'Synced HH:MM' (local
time of finished_at); failed → enabled + retry-ready 'Sync sources'
@@ -927,7 +1046,20 @@ class _GatedImport:
``_chunk_hook`` closure: pair one lands before the main park
(readable mid-run), and every later pair lands after the main
park + its own park (``chunk_release``), so each position is
readable on the status between ticks."""
readable on the status between ticks.
Phase 152 (task 03): the runner now ALWAYS passes the
image-position closure (task 03's wiring) — recorded for the
wiring pins.
Phase 152 (task 04): optionally ALSO fires the in-flight file's
image-position pairs (``image_pairs``) through the runner's OWN
``_image_hook`` closure (pair one before the main park —
readable mid-run; every later pair after the main park + its own
park ``image_release`` — the phase-129 chunk idiom, mirrored for
the vision-image position). With no pairs the fake never fires
it (the status image fields stay 0/0, like the chunk fields of a
no-batch import)."""
def __init__(
self,
@@ -937,6 +1069,8 @@ class _GatedImport:
fail: BaseException | None = None,
chunk_pairs: Sequence[tuple[int, int]] = (),
chunk_release: threading.Event | None = None,
image_pairs: Sequence[tuple[int, int]] = (), # phase 152 (task 04)
image_release: threading.Event | None = None, # phase 152 (task 04)
) -> None:
self.summary = summary
self.started = started
@@ -947,6 +1081,12 @@ class _GatedImport:
self.chunk_pairs = list(chunk_pairs)
self.chunk_release = chunk_release
self.chunk_hook_calls: list[tuple[int, int]] = []
# Phase 152 (task 03): the image-position hook the runner
# passes (recorded for the wiring pins).
self.image_hook: Callable[[int, int], None] | None = None
self.image_pairs = list(image_pairs)
self.image_release = image_release
self.image_hook_calls: list[tuple[int, int]] = []
async def __call__(
self,
@@ -958,6 +1098,7 @@ class _GatedImport:
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -966,6 +1107,16 @@ class _GatedImport:
if progress is not None:
progress("repo", "notes/deep.md", 1, 3)
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
# Phase 152 (task 03): the runner now ALWAYS passes the
# image-position closure (task 03's wiring) — recorded for the
# wiring pins.
self.image_hook = on_image_progress
# Phase 152 (task 04): image pair one lands BEFORE the main
# park (readable mid-run — the phase-129 chunk-pair-one
# idiom, mirrored for the vision-image position).
if on_image_progress is not None and self.image_pairs:
on_image_progress(*self.image_pairs[0])
self.image_hook_calls.append(self.image_pairs[0])
if on_chunk_progress is not None and self.chunk_pairs:
# Phase 129 (task 03): the file's chunk-position pairs,
# fired through the runner's OWN hook closure (the closure
@@ -986,6 +1137,17 @@ class _GatedImport:
else:
self.started.set()
await asyncio.to_thread(self.release.wait) # park without freezing the worker loop
# Phase 152 (task 04): every later image pair lands after the
# main park + its own park (``image_release``), so each
# position is readable on the status between ticks. A wiring
# regression to None skips this branch and the pair tests
# time out.
if on_image_progress is not None:
for pair in self.image_pairs[1:]:
on_image_progress(*pair)
self.image_hook_calls.append(pair)
if self.image_release is not None:
await asyncio.to_thread(self.image_release.wait)
if self.fail is not None:
raise self.fail
return self.summary
@@ -1088,10 +1250,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
fresh_sync_status: None,
) -> None:
"""Idle: the three phase-64 progress keys ride along as null/0/0,
the two phase-129 chunk keys as 0/0, the four phase-98 phase keys
as null/0/0/0, and EVERY pre-existing key is unchanged — the full
response dict is pinned, so the current UI and every existing
consumer keep working."""
the two phase-129 chunk keys as 0/0, the two phase-152 image keys
as 0/0, the four phase-98 phase keys as null/0/0/0, and EVERY
pre-existing key is unchanged — the full response dict is pinned,
so the current UI and every existing consumer keep working."""
assert sync_api.sync_status() == {
"state": "idle",
"started_at": None,
@@ -1105,6 +1267,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
# (the dataclass defaults, no special-casing).
"current_chunk": 0,
"chunks_total": 0,
# Phase 152 (task 01): the image-position keys — 0/0 idle
# (the dataclass defaults, no special-casing).
"images_done": 0,
"images_total": 0,
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
"phase": None,
"current_summary": None,
@@ -1145,13 +1311,19 @@ def test_mid_run_status_reports_current_file(
assert s["files_done"] == 0 and s["files_total"] == 0
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
# Phase 152 (task 01): the image position is 0/0 in the
# prelude too (no vision span yet).
assert s["images_done"] == 0 and s["images_total"] == 0
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Import phase: the hook's file is live on the status, and the
# phase machine names the phase (the summary span has not
# started — its counters are still 0/0); no embed batch has
# fired the chunk hook (this canned import passes no pairs),
# so the chunk position stays 0/0.
# so the chunk position stays 0/0; the image position stays
# 0/0 too (this canned import fires no image hook — task 03
# wires the runner's image closure, the importer's span is
# what would feed it).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
@@ -1161,6 +1333,7 @@ def test_mid_run_status_reports_current_file(
assert s["current_summary"] is None
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 0 and s["images_total"] == 0
fake_import.release.set()
finally:
clone.release.set()
@@ -1178,7 +1351,8 @@ def test_mid_run_status_reports_current_file(
# folder step never fired the summary hook, so the summary counts
# stay 0/0 (the keep-final-counts convention keeps what fired);
# the chunk position clears with the file (phase 129 — it is a
# sub-position of a done file, never a keep-final counter).
# sub-position of a done file, never a keep-final counter), and
# so does the image position (phase 152 — same convention).
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
@@ -1187,6 +1361,7 @@ def test_mid_run_status_reports_current_file(
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["error"] is None
assert s["started_at"] is not None and s["finished_at"] is not None
@@ -1215,6 +1390,7 @@ def test_failed_terminal_clears_current_file_keeps_counts(
"failed: connection refused"
),
chunk_pairs=[(1, 38)], # phase 129: the live pair before the raise
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair before the raise
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
@@ -1231,6 +1407,13 @@ def test_failed_terminal_clears_current_file_keeps_counts(
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 1 and s["chunks_total"] == 38
# Phase 152 (task 04): a live image position in flight when
# the run dies — fired through the runner's OWN _image_hook
# closure (image_pairs[0] — the importer's span would do the
# same); the failed terminal must clear it (the
# sub-position of an abandoned file, like the chunk
# position).
assert s["images_done"] == 3 and s["images_total"] == 15
fake_import.release.set()
finally:
clone.release.set()
@@ -1238,8 +1421,9 @@ def test_failed_terminal_clears_current_file_keeps_counts(
thread.join(10.0)
assert not thread.is_alive()
assert errors == []
# Wiring: the pair fired through the runner's own closure.
# Wiring: the pairs fired through the runner's OWN closure.
assert fake_import.chunk_hook_calls == [(1, 38)]
assert fake_import.image_hook_calls == [(3, 15)]
s = sync_api.sync_status()
assert s["state"] == "failed"
assert s["current_file"] is None # cleared in the terminal state
@@ -1251,6 +1435,11 @@ def test_failed_terminal_clears_current_file_keeps_counts(
# terminal too — the convention split, pinned (the FILE counts
# above keep their final values, the sub-position does not).
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
# Phase 152 (task 04): the image position clears in the FAILED
# terminal too — the convention split, pinned (the live 3/15 fired
# through the closure mid-run is gone, the FILE counts keep their
# final values).
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["detail"] == {}
error = s["error"] or ""
assert "*****@aipi.example.com" in error # credentials masked
@@ -1468,6 +1657,357 @@ def test_chunk_position_clears_at_phase_transitions_and_terminal(
assert s["files_done"] == 1 and s["files_total"] == 3
# ---------- phase 152 (task 01 + task 04): the in-flight file's -----
# ---------- vision-image position (images_done / images_total) -------
#
# Same seams as the phase-129 block above. Task 04 pins every live
# position through the runner's OWN closures (the phase-64/129 idiom):
# the fakes fire the image pairs through the ``on_image_progress``
# kwarg — the ``_image_hook`` under test (task 03's wiring: the
# importer's span's exact path) — and drive the per-file ``_hook``
# reset (through the ``progress`` kwarg the fake import receives) and
# the transition/terminal clears.
def test_mid_run_status_reports_image_position_between_ticks(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""Phase 152 (task 04): the status carries the in-flight file's
VISION-image position — the pairs fire through the runner's OWN
``_image_hook`` closure (task 03's wiring — the importer's
``on_image_progress`` span's exact path), and the position CLIMBS
between ticks (3/15 → 15/15 — the PDF's vision span moving, the
owner's stuck label, now moving) while ``current_file`` and the
chunk position (14/38, a concurrent embed batch) keep their own
values — the THREE positions are independent sub-positions of the
file; the prelude reports 0/0, and the success terminal clears
BOTH sub-positions (image + chunk) to 0/0 while the FILE counts
keep their final values (the convention split)."""
clone = _GatedClone(threading.Event(), threading.Event())
image_release = threading.Event()
fake_import = _GatedImport(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
threading.Event(),
chunk_pairs=[(14, 38)], # a concurrent embed batch (independent)
image_pairs=[(3, 15), (15, 15)], # two describe calls of a 15-image span
image_release=image_release,
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
# Prelude: every position 0/0 (no file, no batch, no image
# yet — alongside the bare-label pins of the phase-64 test).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["current_file"] is None
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 0 and s["images_total"] == 0
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Import phase: the file is live (the progress hook's call),
# the chunk position 14/38 (the embed batch), and the image
# position 3/15 (the vision span) — the THREE positions are
# independent sub-positions of the same file.
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 14 and s["chunks_total"] == 38
assert s["images_done"] == 3 and s["images_total"] == 15
# Release the main park: the import fires image pair two
# (15/15 — the span's last describe call) and parks on
# image_release.
fake_import.release.set()
_poll_until(
lambda: sync_api.sync_status()["images_done"] == 15,
what="the second image pair on the status",
)
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
# The image position CLIMBED (3 → 15 — monotonic, no
# backwards jump, never above the total) ... the FILE and
# chunk positions keep their own values (independent).
assert s["images_done"] == 15 and s["images_total"] == 15
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 14 and s["chunks_total"] == 38
image_release.set()
finally:
clone.release.set()
fake_import.release.set()
image_release.set()
thread.join(10.0)
assert not thread.is_alive()
assert errors == [], f"the runner raised: {errors!r}"
# Wiring: every pair fired through the runner's OWN closure (the
# status reads above came through it; a wiring regression to
# None would have skipped the firing branch and timed out).
assert fake_import.chunk_hook_calls == [(14, 38)]
assert fake_import.image_hook_calls == [(3, 15), (15, 15)]
# Success terminal: BOTH sub-positions clear (0/0) ...
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 0 and s["images_total"] == 0
# ... while the FILE counts keep their final values (the
# convention split, pinned alongside the phase-64 terminal pin).
assert s["current_file"] is None
assert s["files_done"] == 1 and s["files_total"] == 3
class _TwoFileImport:
"""The phase-152 (task 01 + task 04) fake: fires the runner's
OWN progress hook for file A, then file A's image position through
the runner's OWN ``on_image_progress`` closure (task 03's wiring —
what the importer's span would do), parks (the test reads the
live position on the status), fires the progress hook again for
file B (the runner's ``_hook`` under test must reset the image
position to 0/0 — the pairing rule, extended), parks again, and
completes."""
def __init__(self, summary: ImportSummary) -> None:
self.summary = summary
self.parked_a = threading.Event() # parked after file A's progress
self.release_a = threading.Event() # test releases → file B's progress
self.parked_b = threading.Event() # parked after file B's progress
self.release_b = threading.Event()
self.hook_calls: list[tuple[str, str, int, int]] = []
self.image_hook_calls: list[tuple[int, int]] = [] # phase 152 (task 04)
async def __call__(
self,
sources: list[Path],
llm: object,
*,
prune: bool = False,
limit: int | None = None,
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
) -> ImportSummary:
if progress is not None:
progress("repo", "docs/a.pdf", 1, 2)
self.hook_calls.append(("repo", "docs/a.pdf", 1, 2))
# Phase 152 (task 04): file A's vision span is in flight —
# its image position (3/15) fires through the runner's OWN
# on_image_progress closure (task 03's wiring), then the fake
# parks.
if on_image_progress is not None:
on_image_progress(3, 15)
self.image_hook_calls.append((3, 15))
self.parked_a.set()
await asyncio.to_thread(self.release_a.wait)
if progress is not None:
# The SECOND progress call names file B — the closure under
# test must reset the image position the closure fired
# while file A was in flight (a file with no images must
# not inherit the previous PDF's position).
progress("repo", "docs/b.md", 2, 2)
self.hook_calls.append(("repo", "docs/b.md", 2, 2))
self.parked_b.set()
await asyncio.to_thread(self.release_b.wait)
return self.summary
def test_image_position_resets_per_file(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""Phase 152 (task 01 + task 04): the pairing rule, extended — a
``progress`` call names the file AND clears the image position;
every image position until the next ``progress`` call belongs to
it. File A's live position (3/15, fired through the runner's OWN
``_image_hook`` closure while parked on file A — task 04: the
phase-64/129 idiom, the importer's image hook's exact path) is
0/0 the moment the runner's own ``_hook`` names file B: a file
with no images must not inherit the previous PDF's position
(unlike the chunk position, not every file embeds chunks). The
success terminal keeps the position at 0/0 and the FILE counts
keep their final values (the convention split)."""
clone = _GatedClone(threading.Event(), threading.Event())
fake_import = _TwoFileImport(ImportSummary(files=2, added=1, updated=1))
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
clone.release.set()
assert fake_import.parked_a.wait(5.0), "the run never parked on file A"
# File A is named by the runner's own _hook (a wiring
# regression to None would leave current_file null here).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/docs/a.pdf"
assert s["files_done"] == 1 and s["files_total"] == 2
# File A's vision span is in flight: its image position is
# live on the status — fired through the runner's OWN
# _image_hook closure (the fake fired it before parking;
# a wiring regression to None would leave it 0/0 here).
assert s["images_done"] == 3 and s["images_total"] == 15
# Release file A: the runner's _hook names file B — the image
# position must reset to 0/0 in the same call (the pairing
# rule, extended; the closure under test, driven through the
# fake's progress kwarg).
fake_import.release_a.set()
assert fake_import.parked_b.wait(5.0), "the run never parked on file B"
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/docs/b.md"
assert s["files_done"] == 2 and s["files_total"] == 2
# RESET per file: file B (no images) does not inherit file
# A's 3/15.
assert s["images_done"] == 0 and s["images_total"] == 0
fake_import.release_b.set()
finally:
clone.release.set()
fake_import.release_a.set()
fake_import.release_b.set()
thread.join(10.0)
assert not thread.is_alive()
assert errors == [], f"the runner raised: {errors!r}"
# Wiring: BOTH progress calls went through the runner's own
# closure (the reset above was its doing), and the image pair
# went through the runner's own _image_hook closure.
assert fake_import.hook_calls == [
("repo", "docs/a.pdf", 1, 2),
("repo", "docs/b.md", 2, 2),
]
assert fake_import.image_hook_calls == [(3, 15)]
# Success terminal: the image position stays 0/0 (cleared in the
# terminal too — every terminal clears it) and the FILE counts
# keep their final values (the convention split).
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
assert s["current_file"] is None
assert s["files_done"] == 2 and s["files_total"] == 2
assert s["images_done"] == 0 and s["images_total"] == 0
def test_image_position_clears_at_phase_transitions_and_terminal(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""Phase 152 (task 01 + task 04): the image position is CLEARED
(0/0) at every point the import phase ends. This run's summary
changes the KB, so BOTH post-import transitions run and are
pinned: the import→``"overview"`` transition and the
import→``"summaries"`` transition (the changed-KB branch) each
clear the live position (3/15, fired mid-import through the
runner's OWN ``_image_hook`` closure — task 04: the
phase-64/129 idiom, the importer's span's exact path) to 0/0,
and the success terminal keeps it 0/0 — through the whole span
the FILE counts keep their final values (the convention split:
the file counts survive for the result line's context, the image
position is a sub-position of a file that's done)."""
clone = _GatedClone(threading.Event(), threading.Event())
import_release = threading.Event()
overview_started = threading.Event()
overview_release = threading.Event()
folders_started = threading.Event()
folders_release = threading.Event()
fake_import = _GatedImport(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
import_release,
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair to clear
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
# Park the post-import steps (re-monkeypatched over the seams
# helper's no-ops): the test reads the status DURING the
# "overview" phase and DURING the "summaries" phase — the two
# import-phase-ending transitions the task pins (the same shape
# as the phase-129 chunk-position transition test above).
async def gated_overview(llm: object, session: object = None) -> bool:
overview_started.set()
await asyncio.to_thread(overview_release.wait)
return True
async def gated_folders(
db: object,
llm: object,
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
folders_started.set()
await asyncio.to_thread(folders_release.wait)
return {"generated": 0, "failed": 0, "pruned": 0}
monkeypatch.setattr(sync_api, "regenerate_overview", gated_overview)
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Mid-import: the file is live; its image position went live
# too — fired through the runner's OWN _image_hook closure
# (image_pairs[0] — the importer's span would do the same).
s = sync_api.sync_status()
assert s["phase"] == "import"
assert s["images_done"] == 3 and s["images_total"] == 15
# Release the import: it returns (added+updated > 0) → the
# phase machine moves to "overview" and the position clears.
import_release.set()
assert overview_started.wait(5.0), "the run never reached the overview"
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "overview"
# CLEARED at the import→overview transition (0/0) ... the file
# counts KEEP their final values (the convention split).
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["files_done"] == 1 and s["files_total"] == 3
overview_release.set()
# ... and the changed-KB branch moves to "summaries" (the
# second import-phase-ending transition — cleared there too,
# idempotently) before parking.
assert folders_started.wait(5.0), "the run never reached the summaries"
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "summaries"
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["files_done"] == 1 and s["files_total"] == 3
folders_release.set()
finally:
clone.release.set()
import_release.set()
overview_release.set()
folders_release.set()
thread.join(10.0)
assert not thread.is_alive()
assert errors == [], f"the runner raised: {errors!r}"
assert fake_import.image_hook_calls == [(3, 15)]
# Success terminal: still 0/0 (the terminal clears too — same
# spots as current_file), file counts kept.
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
assert s["current_file"] is None
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["files_done"] == 1 and s["files_total"] == 3
# ---------- phase 134 (task 01): the cooperative cancel (5th state) ----------
#
# The cancel runs are driven through the REAL ``start_sync`` on a
@@ -1629,11 +2169,17 @@ def test_cancel_endpoint_stops_the_in_flight_run(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
threading.Event(),
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair before the cancel
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, _fast_clone)
def act(loop: asyncio.AbstractEventLoop) -> dict[str, str]:
assert fake_import.started.wait(5.0), "the run never reached the import"
# Phase 152 (task 04): the live image position in flight when
# the cancel converts was fired through the runner's OWN
# _image_hook closure (image_pairs[0] — the fake fired it
# before parking, like the importer's span would); the
# cancelled terminal must clear it.
return loop.run_until_complete(sync_api.cancel_sync())
try:
@@ -1665,6 +2211,11 @@ def test_cancel_endpoint_stops_the_in_flight_run(
assert s["files_done"] == 1 and s["files_total"] == 3 # final counts kept
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
# Phase 152 (task 04): the image position clears in the CANCELLED
# terminal too — every terminal clears it (the live 3/15 fired
# through the closure in flight is gone; the FILE counts above
# keep their final values).
assert s["images_done"] == 0 and s["images_total"] == 0
# PLAN §9: the ample-logging line for the converted cancel.
assert any(
r.getMessage()
@@ -1879,6 +2430,7 @@ def test_event_set_during_import_cancels_before_the_lite_spans(
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106