phase: 152_sync_progress_image_pass
All criteria verified. The CLI path is hook-less (only `app/api/sync.py` passes the hook), and the byte-identity is pinned by `test_on_image_progress_none_run_is_byte_identical`. ## Final Report — Phase 152 final verification pass - **Verified (no new code needed):** `SyncStatus.images_done/images_total` + reset/clear rules in `app/api/sync.py`; `on_image_progress` threaded through the importer's three PDF vision spans with the continuous per-file counter; the `_image_hook` closure; `sources.js` `· image n/m` label part (box + title + aria). - **C1** status climbs 1→N + UI shows `· image n/m`: ✅ unit 3/15→15/15, integration 1/15→7/15, E2E 1/4→4/4 (title/aria ride the full string) - **C2** never frozen: ✅ E2E asserts ≥3 distinct positions, starting at 1, non-decreasing, ≤ total, reaching it - **C3** cleared at overview/summaries + all terminals; per-file reset via `_hook`; no stale position for image-less files: ✅ unit pins + E2E settle (image part gone in box/title/aria, endpoint 0/0) - **C4** `/api/sync/status` additive-only: ✅ full-dict pin passes (all pre-existing keys unchanged + the two new 0/0 keys) - **C5** hook-less caller byte-identical: ✅ CLI passes no hook; `test_on_image_progress_none_run_is_byte_identical` green - **C6** `uv run pytest`: **3526 passed, exit 0** (95s); dedicated E2E in isolation: **2 passed** (95s); `uv run ruff check . && uv run pyright`: **clean**; coverage: **99% on `app/` (>90% gate)** - Regression: all five neighbouring E2E suites green in isolation (progress_box 8, chunk_progress 2, label_mobile 4, pdf_support 6, image_documents 4) - **C7** atomic commit: left to the harness (no git index/history touched per rules) - **Docs:** README.md — sync-label paragraph already updated this phase, accurately; all `.env.example` vars documented; routes/commands verified — **no changes needed**. AGENTS.md — all references (PLAN, user_stories, test_smoke.py, .env.example, gpg config, .gitignore) verified — **no changes needed** - **Defects found in prior work:** none (one prior env-leak defect was already fixed inside this phase's task 04, conftest) - No live-infrastructure touched; no ntfy used. **Next pending phase:** none — `todo/` held only this phase
This commit is contained in:
+567
-15
@@ -110,6 +110,56 @@ poll tree's new 5th branch (after failed, before the upload branches
|
||||
— the 7-branch tree is now 8), the neutral ``applySyncCancelled``
|
||||
(no modal, no banner, no error styling), and the
|
||||
``initSyncButton`` re-attach of a ``cancelled`` run.
|
||||
|
||||
Phase 152 (task 01): the status contract gains the in-flight file's
|
||||
VISION-image position — ``images_done`` (the 1-based index of the
|
||||
image about to be described) + ``images_total`` (the file's total
|
||||
images for this run — the three PDF vision spans share ONE continuous
|
||||
counter per file, the importer's ``on_image_progress`` hook that
|
||||
task 02/03 wire). The idle dict pins them 0/0; the per-file reset is
|
||||
the pairing rule, extended — a ``progress`` call names the file AND
|
||||
clears the image position, pinned through the runner's own ``_hook``
|
||||
closure (the two-file fake below: the position fired through the
|
||||
runner's own ``_image_hook`` closure while file A is in flight is
|
||||
0/0 when file B is named — a file with no images must not inherit
|
||||
the previous PDF's position); the clear rule mirrors the chunk
|
||||
position — 0/0 at the overview/summaries transitions and in EVERY
|
||||
terminal (success, failed, and cancelled — each pinned with a live
|
||||
position in flight; task 04 moves every live position below onto the
|
||||
runner's OWN closures, the phase-64/129 idiom), while the FILE
|
||||
counts keep their final values (the convention split: a
|
||||
sub-position of a file that's done clears with the file).
|
||||
|
||||
Phase 152 (task 03): the runner's THIRD paired closure —
|
||||
``_image_hook`` (beside ``_hook`` and ``_chunk_hook``) — lands the
|
||||
importer's ``on_image_progress`` pairs on the status, and the import
|
||||
call now passes ``on_image_progress=_image_hook`` (the wiring pin:
|
||||
the fakes receive the kwarg and record the live closure). The
|
||||
frontend gains the "· image n/m" label PART: the running entry
|
||||
destructures ``image = ""`` and the writer call becomes
|
||||
``setSyncLabel(main, count, chunk, true, image)`` — the image part
|
||||
rides ONLY the composed full string (box + title + running
|
||||
aria-label — no new button span; the running button is the
|
||||
spinner), the chunk part keeps its own span, and the bare builders
|
||||
+ non-running calls pass nothing (both parts default "").
|
||||
|
||||
Phase 152 (task 04): the position pins move onto the runner's OWN
|
||||
closures (the phase-64/129 idiom — the fakes fire the image pairs
|
||||
through the ``on_image_progress`` kwarg, the ``_image_hook`` under
|
||||
test, instead of standing in on the status object): a mid-run pair
|
||||
(3/15, then 15/15) climbs on the status BETWEEN ticks while
|
||||
``current_file`` and the chunk position keep their own values (the
|
||||
THREE positions are independent sub-positions of the file), the
|
||||
per-file reset, the overview/summaries transition clears, and the
|
||||
FAILED + CANCELLED terminals all fire their live positions through
|
||||
the closure too. The frontend source pins the ``fmtSyncPhaseLabel``
|
||||
image part (present iff ``images_total > 0`` — the exact guard +
|
||||
template), the ``overview`` / ``summaries`` branches' ``image: ""``
|
||||
(unchanged labels), and ``setSyncLabel``'s composed ``full`` string
|
||||
byte-for-byte (a status carrying both positions reads "… (2540/2540)
|
||||
· chunk 125/126 · image 3/15" on the box + title + running
|
||||
aria-label — the runtime string is behaviorally pinned in
|
||||
tests/unit/test_frontend_sync_upload.py's node check).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -393,13 +443,16 @@ def test_sources_js_running_state_is_never_stale() -> None:
|
||||
Phase 128 (task 01): the entry hands the label PARTS (the
|
||||
phase-aware split — sync + status → fmtSyncPhaseLabel, the
|
||||
fall-through → the byte-identical fmtSyncLabel) to the SINGLE
|
||||
writer, setSyncLabel(main, count, chunk — phase 129: the chunk
|
||||
part destructures with a "" default), which lands the main part,
|
||||
writer, setSyncLabel(main, count, chunk, running, image — phase
|
||||
129: the chunk part destructures with a "" default, phase 152:
|
||||
the image part does too), which lands the main part,
|
||||
the pinned count part (hidden when absent), the pinned chunk part
|
||||
(phase 129 — hidden when absent), and the FULL untruncated label
|
||||
on the button title + #sync-progress (A4) — the direct span /
|
||||
title / announcer writes are GONE from the entry (they live in
|
||||
setSyncLabel + setSyncProgress). Phase 149 (task 01): the writer call passes
|
||||
on the button title + #sync-progress (A4 — phase 152: with the
|
||||
image part appended after the chunk part when a PDF's vision span
|
||||
is in flight; the image part gets no span of its own) — the
|
||||
direct span / title / announcer writes are GONE from the entry
|
||||
(they live in setSyncLabel + setSyncProgress). Phase 149 (task 01): the writer call passes
|
||||
`running = true` (the detail box reveals with the full label),
|
||||
and the button's accessible name rides in the writer too (the
|
||||
full live label — the icon-only running button keeps a name);
|
||||
@@ -414,8 +467,10 @@ def test_sources_js_running_state_is_never_stale() -> None:
|
||||
assert "fmtSyncLabel(kind, currentFile, done, total)" in body
|
||||
# …and the single writer lands the parts (A4 in it — the title is
|
||||
# set for EVERY running label: the file-only rule is adjusted
|
||||
# away) with the phase-149 `running` flag (the box reveals).
|
||||
assert "setSyncLabel(main, count, chunk, true)" in body
|
||||
# away) with the phase-149 `running` flag (the box reveals) +
|
||||
# the phase-152 image part (rides only the composed full string —
|
||||
# no span).
|
||||
assert "setSyncLabel(main, count, chunk, true, image)" in body
|
||||
assert body.count("setSyncLabel(") == 1, "the entry writes ONLY through the parts writer"
|
||||
assert "syncBtn.title =" not in body, ("the title write lives in setSyncLabel (A4)")
|
||||
assert "syncLabel.textContent" not in body, ("no direct span write — the parts")
|
||||
@@ -433,6 +488,70 @@ def test_sources_js_running_state_is_never_stale() -> None:
|
||||
)
|
||||
|
||||
|
||||
def test_fmt_sync_phase_label_image_part_is_pinned_source_level() -> None:
|
||||
"""Phase 152 (task 04): the frontend SOURCE pins the "· image
|
||||
n/m" label part (the browser behavior is pinned in
|
||||
tests/unit/test_frontend_sync_upload.py's node check — the REAL
|
||||
extracted functions run against stub refs; this pin keeps the
|
||||
template itself from drifting without a browser):
|
||||
|
||||
* the import fall-through is the ONLY reader of the status's
|
||||
image fields, guarded on ``images_total > 0`` (the part is
|
||||
present IFF ``images_total > 0`` — 0/0 hides it, the exact
|
||||
guard + template + empty false-branch);
|
||||
* the ``overview`` / ``summaries`` branches return ``image: ""``
|
||||
(the position belongs to the import phase only — those labels
|
||||
stay unchanged by the phase);
|
||||
* the running entry destructures the part with a `""` default and
|
||||
hands it to the single writer as the FIFTH argument (after the
|
||||
`running` flag);
|
||||
* ``setSyncLabel``'s composed ``full`` string appends the image
|
||||
part LAST (after the chunk part) — so a status carrying both
|
||||
positions reads exactly "Syncing… <file> (2540/2540) · chunk
|
||||
125/126 · image 3/15" on the box, the title, and the running
|
||||
aria-label (byte-for-byte; the docstring's worked example
|
||||
carries the shape).
|
||||
"""
|
||||
js = _text(SOURCES_JS)
|
||||
builder = _body(js, "fmtSyncPhaseLabel")
|
||||
# The import fall-through: present IFF images_total > 0 — the
|
||||
# exact guard + template (the phase-129 chunk part's sibling,
|
||||
# same shape).
|
||||
assert (
|
||||
"status.images_total > 0\n"
|
||||
" ? `· image ${status.images_done}/${status.images_total}`\n"
|
||||
" : \"\";"
|
||||
) in builder
|
||||
assert "return { main, count, chunk, image };" in builder
|
||||
# The overview branch: the unchanged label + the part absent.
|
||||
assert (
|
||||
'return { main: "Writing KB overview…", count: "", chunk: "", image: "" };'
|
||||
) in builder
|
||||
# The summaries branch: the unchanged label + the part absent
|
||||
# (the phase's own (n/m) is its position).
|
||||
assert "image: \"\", // the image position belongs to the import phase only" in builder
|
||||
assert "`Summarizing folders…${folder}`" in builder
|
||||
# The running entry: the part destructures with a `""` default
|
||||
# and rides the writer call as the fifth argument.
|
||||
entry = _body(js, "enterSyncRunningState")
|
||||
assert 'const { main, count, chunk = "", image = "" }' in entry
|
||||
assert "setSyncLabel(main, count, chunk, true, image)" in entry
|
||||
# The writer: the composed full string — the image part LAST,
|
||||
# after the chunk part (the "… · chunk 125/126 · image 3/15"
|
||||
# shape the docstring's worked example carries).
|
||||
writer = _body(js, "setSyncLabel")
|
||||
assert (
|
||||
"const full =\n"
|
||||
" mainText +\n"
|
||||
" (countText ? ` ${countText}` : \"\") +\n"
|
||||
" (chunkText ? ` ${chunkText}` : \"\") +\n"
|
||||
" (imageText ? ` ${imageText}` : \"\");"
|
||||
) in writer
|
||||
assert "· chunk 125/126 · image 3/15" in js, (
|
||||
"the worked example (both positions at once) is documented"
|
||||
)
|
||||
|
||||
|
||||
def test_sources_js_terminal_states() -> None:
|
||||
"""Terminal rendering: success → enabled + 'Synced HH:MM' (local
|
||||
time of finished_at); failed → enabled + retry-ready 'Sync sources'
|
||||
@@ -927,7 +1046,20 @@ class _GatedImport:
|
||||
``_chunk_hook`` closure: pair one lands before the main park
|
||||
(readable mid-run), and every later pair lands after the main
|
||||
park + its own park (``chunk_release``), so each position is
|
||||
readable on the status between ticks."""
|
||||
readable on the status between ticks.
|
||||
|
||||
Phase 152 (task 03): the runner now ALWAYS passes the
|
||||
image-position closure (task 03's wiring) — recorded for the
|
||||
wiring pins.
|
||||
|
||||
Phase 152 (task 04): optionally ALSO fires the in-flight file's
|
||||
image-position pairs (``image_pairs``) through the runner's OWN
|
||||
``_image_hook`` closure (pair one before the main park —
|
||||
readable mid-run; every later pair after the main park + its own
|
||||
park ``image_release`` — the phase-129 chunk idiom, mirrored for
|
||||
the vision-image position). With no pairs the fake never fires
|
||||
it (the status image fields stay 0/0, like the chunk fields of a
|
||||
no-batch import)."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -937,6 +1069,8 @@ class _GatedImport:
|
||||
fail: BaseException | None = None,
|
||||
chunk_pairs: Sequence[tuple[int, int]] = (),
|
||||
chunk_release: threading.Event | None = None,
|
||||
image_pairs: Sequence[tuple[int, int]] = (), # phase 152 (task 04)
|
||||
image_release: threading.Event | None = None, # phase 152 (task 04)
|
||||
) -> None:
|
||||
self.summary = summary
|
||||
self.started = started
|
||||
@@ -947,6 +1081,12 @@ class _GatedImport:
|
||||
self.chunk_pairs = list(chunk_pairs)
|
||||
self.chunk_release = chunk_release
|
||||
self.chunk_hook_calls: list[tuple[int, int]] = []
|
||||
# Phase 152 (task 03): the image-position hook the runner
|
||||
# passes (recorded for the wiring pins).
|
||||
self.image_hook: Callable[[int, int], None] | None = None
|
||||
self.image_pairs = list(image_pairs)
|
||||
self.image_release = image_release
|
||||
self.image_hook_calls: list[tuple[int, int]] = []
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -958,6 +1098,7 @@ class _GatedImport:
|
||||
session: object = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -966,6 +1107,16 @@ class _GatedImport:
|
||||
if progress is not None:
|
||||
progress("repo", "notes/deep.md", 1, 3)
|
||||
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
|
||||
# Phase 152 (task 03): the runner now ALWAYS passes the
|
||||
# image-position closure (task 03's wiring) — recorded for the
|
||||
# wiring pins.
|
||||
self.image_hook = on_image_progress
|
||||
# Phase 152 (task 04): image pair one lands BEFORE the main
|
||||
# park (readable mid-run — the phase-129 chunk-pair-one
|
||||
# idiom, mirrored for the vision-image position).
|
||||
if on_image_progress is not None and self.image_pairs:
|
||||
on_image_progress(*self.image_pairs[0])
|
||||
self.image_hook_calls.append(self.image_pairs[0])
|
||||
if on_chunk_progress is not None and self.chunk_pairs:
|
||||
# Phase 129 (task 03): the file's chunk-position pairs,
|
||||
# fired through the runner's OWN hook closure (the closure
|
||||
@@ -986,6 +1137,17 @@ class _GatedImport:
|
||||
else:
|
||||
self.started.set()
|
||||
await asyncio.to_thread(self.release.wait) # park without freezing the worker loop
|
||||
# Phase 152 (task 04): every later image pair lands after the
|
||||
# main park + its own park (``image_release``), so each
|
||||
# position is readable on the status between ticks. A wiring
|
||||
# regression to None skips this branch and the pair tests
|
||||
# time out.
|
||||
if on_image_progress is not None:
|
||||
for pair in self.image_pairs[1:]:
|
||||
on_image_progress(*pair)
|
||||
self.image_hook_calls.append(pair)
|
||||
if self.image_release is not None:
|
||||
await asyncio.to_thread(self.image_release.wait)
|
||||
if self.fail is not None:
|
||||
raise self.fail
|
||||
return self.summary
|
||||
@@ -1088,10 +1250,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
|
||||
fresh_sync_status: None,
|
||||
) -> None:
|
||||
"""Idle: the three phase-64 progress keys ride along as null/0/0,
|
||||
the two phase-129 chunk keys as 0/0, the four phase-98 phase keys
|
||||
as null/0/0/0, and EVERY pre-existing key is unchanged — the full
|
||||
response dict is pinned, so the current UI and every existing
|
||||
consumer keep working."""
|
||||
the two phase-129 chunk keys as 0/0, the two phase-152 image keys
|
||||
as 0/0, the four phase-98 phase keys as null/0/0/0, and EVERY
|
||||
pre-existing key is unchanged — the full response dict is pinned,
|
||||
so the current UI and every existing consumer keep working."""
|
||||
assert sync_api.sync_status() == {
|
||||
"state": "idle",
|
||||
"started_at": None,
|
||||
@@ -1105,6 +1267,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
|
||||
# (the dataclass defaults, no special-casing).
|
||||
"current_chunk": 0,
|
||||
"chunks_total": 0,
|
||||
# Phase 152 (task 01): the image-position keys — 0/0 idle
|
||||
# (the dataclass defaults, no special-casing).
|
||||
"images_done": 0,
|
||||
"images_total": 0,
|
||||
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
|
||||
"phase": None,
|
||||
"current_summary": None,
|
||||
@@ -1145,13 +1311,19 @@ def test_mid_run_status_reports_current_file(
|
||||
assert s["files_done"] == 0 and s["files_total"] == 0
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
# Phase 152 (task 01): the image position is 0/0 in the
|
||||
# prelude too (no vision span yet).
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
clone.release.set()
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
# Import phase: the hook's file is live on the status, and the
|
||||
# phase machine names the phase (the summary span has not
|
||||
# started — its counters are still 0/0); no embed batch has
|
||||
# fired the chunk hook (this canned import passes no pairs),
|
||||
# so the chunk position stays 0/0.
|
||||
# so the chunk position stays 0/0; the image position stays
|
||||
# 0/0 too (this canned import fires no image hook — task 03
|
||||
# wires the runner's image closure, the importer's span is
|
||||
# what would feed it).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "import"
|
||||
@@ -1161,6 +1333,7 @@ def test_mid_run_status_reports_current_file(
|
||||
assert s["current_summary"] is None
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
fake_import.release.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
@@ -1178,7 +1351,8 @@ def test_mid_run_status_reports_current_file(
|
||||
# folder step never fired the summary hook, so the summary counts
|
||||
# stay 0/0 (the keep-final-counts convention keeps what fired);
|
||||
# the chunk position clears with the file (phase 129 — it is a
|
||||
# sub-position of a done file, never a keep-final counter).
|
||||
# sub-position of a done file, never a keep-final counter), and
|
||||
# so does the image position (phase 152 — same convention).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "success"
|
||||
assert s["phase"] is None
|
||||
@@ -1187,6 +1361,7 @@ def test_mid_run_status_reports_current_file(
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
assert s["error"] is None
|
||||
assert s["started_at"] is not None and s["finished_at"] is not None
|
||||
|
||||
@@ -1215,6 +1390,7 @@ def test_failed_terminal_clears_current_file_keeps_counts(
|
||||
"failed: connection refused"
|
||||
),
|
||||
chunk_pairs=[(1, 38)], # phase 129: the live pair before the raise
|
||||
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair before the raise
|
||||
)
|
||||
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
|
||||
|
||||
@@ -1231,6 +1407,13 @@ def test_failed_terminal_clears_current_file_keeps_counts(
|
||||
assert s["current_file"] == "repo/notes/deep.md"
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
assert s["current_chunk"] == 1 and s["chunks_total"] == 38
|
||||
# Phase 152 (task 04): a live image position in flight when
|
||||
# the run dies — fired through the runner's OWN _image_hook
|
||||
# closure (image_pairs[0] — the importer's span would do the
|
||||
# same); the failed terminal must clear it (the
|
||||
# sub-position of an abandoned file, like the chunk
|
||||
# position).
|
||||
assert s["images_done"] == 3 and s["images_total"] == 15
|
||||
fake_import.release.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
@@ -1238,8 +1421,9 @@ def test_failed_terminal_clears_current_file_keeps_counts(
|
||||
thread.join(10.0)
|
||||
assert not thread.is_alive()
|
||||
assert errors == []
|
||||
# Wiring: the pair fired through the runner's own closure.
|
||||
# Wiring: the pairs fired through the runner's OWN closure.
|
||||
assert fake_import.chunk_hook_calls == [(1, 38)]
|
||||
assert fake_import.image_hook_calls == [(3, 15)]
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "failed"
|
||||
assert s["current_file"] is None # cleared in the terminal state
|
||||
@@ -1251,6 +1435,11 @@ def test_failed_terminal_clears_current_file_keeps_counts(
|
||||
# terminal too — the convention split, pinned (the FILE counts
|
||||
# above keep their final values, the sub-position does not).
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
# Phase 152 (task 04): the image position clears in the FAILED
|
||||
# terminal too — the convention split, pinned (the live 3/15 fired
|
||||
# through the closure mid-run is gone, the FILE counts keep their
|
||||
# final values).
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
assert s["detail"] == {}
|
||||
error = s["error"] or ""
|
||||
assert "*****@aipi.example.com" in error # credentials masked
|
||||
@@ -1468,6 +1657,357 @@ def test_chunk_position_clears_at_phase_transitions_and_terminal(
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
|
||||
|
||||
# ---------- phase 152 (task 01 + task 04): the in-flight file's -----
|
||||
# ---------- vision-image position (images_done / images_total) -------
|
||||
#
|
||||
# Same seams as the phase-129 block above. Task 04 pins every live
|
||||
# position through the runner's OWN closures (the phase-64/129 idiom):
|
||||
# the fakes fire the image pairs through the ``on_image_progress``
|
||||
# kwarg — the ``_image_hook`` under test (task 03's wiring: the
|
||||
# importer's span's exact path) — and drive the per-file ``_hook``
|
||||
# reset (through the ``progress`` kwarg the fake import receives) and
|
||||
# the transition/terminal clears.
|
||||
|
||||
|
||||
def test_mid_run_status_reports_image_position_between_ticks(
|
||||
fresh_sync_status: None,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Phase 152 (task 04): the status carries the in-flight file's
|
||||
VISION-image position — the pairs fire through the runner's OWN
|
||||
``_image_hook`` closure (task 03's wiring — the importer's
|
||||
``on_image_progress`` span's exact path), and the position CLIMBS
|
||||
between ticks (3/15 → 15/15 — the PDF's vision span moving, the
|
||||
owner's stuck label, now moving) while ``current_file`` and the
|
||||
chunk position (14/38, a concurrent embed batch) keep their own
|
||||
values — the THREE positions are independent sub-positions of the
|
||||
file; the prelude reports 0/0, and the success terminal clears
|
||||
BOTH sub-positions (image + chunk) to 0/0 while the FILE counts
|
||||
keep their final values (the convention split)."""
|
||||
clone = _GatedClone(threading.Event(), threading.Event())
|
||||
image_release = threading.Event()
|
||||
fake_import = _GatedImport(
|
||||
ImportSummary(files=3, added=1, updated=1, unchanged=1),
|
||||
threading.Event(),
|
||||
threading.Event(),
|
||||
chunk_pairs=[(14, 38)], # a concurrent embed batch (independent)
|
||||
image_pairs=[(3, 15), (15, 15)], # two describe calls of a 15-image span
|
||||
image_release=image_release,
|
||||
)
|
||||
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
|
||||
|
||||
thread, errors = _start_run()
|
||||
try:
|
||||
assert clone.started.wait(5.0), "the run never reached the clone phase"
|
||||
# Prelude: every position 0/0 (no file, no batch, no image
|
||||
# yet — alongside the bare-label pins of the phase-64 test).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["current_file"] is None
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
clone.release.set()
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
# Import phase: the file is live (the progress hook's call),
|
||||
# the chunk position 14/38 (the embed batch), and the image
|
||||
# position 3/15 (the vision span) — the THREE positions are
|
||||
# independent sub-positions of the same file.
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "import"
|
||||
assert s["current_file"] == "repo/notes/deep.md"
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
assert s["current_chunk"] == 14 and s["chunks_total"] == 38
|
||||
assert s["images_done"] == 3 and s["images_total"] == 15
|
||||
# Release the main park: the import fires image pair two
|
||||
# (15/15 — the span's last describe call) and parks on
|
||||
# image_release.
|
||||
fake_import.release.set()
|
||||
_poll_until(
|
||||
lambda: sync_api.sync_status()["images_done"] == 15,
|
||||
what="the second image pair on the status",
|
||||
)
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "import"
|
||||
# The image position CLIMBED (3 → 15 — monotonic, no
|
||||
# backwards jump, never above the total) ... the FILE and
|
||||
# chunk positions keep their own values (independent).
|
||||
assert s["images_done"] == 15 and s["images_total"] == 15
|
||||
assert s["current_file"] == "repo/notes/deep.md"
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
assert s["current_chunk"] == 14 and s["chunks_total"] == 38
|
||||
image_release.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
fake_import.release.set()
|
||||
image_release.set()
|
||||
thread.join(10.0)
|
||||
assert not thread.is_alive()
|
||||
assert errors == [], f"the runner raised: {errors!r}"
|
||||
# Wiring: every pair fired through the runner's OWN closure (the
|
||||
# status reads above came through it; a wiring regression to
|
||||
# None would have skipped the firing branch and timed out).
|
||||
assert fake_import.chunk_hook_calls == [(14, 38)]
|
||||
assert fake_import.image_hook_calls == [(3, 15), (15, 15)]
|
||||
# Success terminal: BOTH sub-positions clear (0/0) ...
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "success"
|
||||
assert s["phase"] is None
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
# ... while the FILE counts keep their final values (the
|
||||
# convention split, pinned alongside the phase-64 terminal pin).
|
||||
assert s["current_file"] is None
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
|
||||
|
||||
class _TwoFileImport:
|
||||
"""The phase-152 (task 01 + task 04) fake: fires the runner's
|
||||
OWN progress hook for file A, then file A's image position through
|
||||
the runner's OWN ``on_image_progress`` closure (task 03's wiring —
|
||||
what the importer's span would do), parks (the test reads the
|
||||
live position on the status), fires the progress hook again for
|
||||
file B (the runner's ``_hook`` under test must reset the image
|
||||
position to 0/0 — the pairing rule, extended), parks again, and
|
||||
completes."""
|
||||
|
||||
def __init__(self, summary: ImportSummary) -> None:
|
||||
self.summary = summary
|
||||
self.parked_a = threading.Event() # parked after file A's progress
|
||||
self.release_a = threading.Event() # test releases → file B's progress
|
||||
self.parked_b = threading.Event() # parked after file B's progress
|
||||
self.release_b = threading.Event()
|
||||
self.hook_calls: list[tuple[str, str, int, int]] = []
|
||||
self.image_hook_calls: list[tuple[int, int]] = [] # phase 152 (task 04)
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
sources: list[Path],
|
||||
llm: object,
|
||||
*,
|
||||
prune: bool = False,
|
||||
limit: int | None = None,
|
||||
session: object = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
) -> ImportSummary:
|
||||
if progress is not None:
|
||||
progress("repo", "docs/a.pdf", 1, 2)
|
||||
self.hook_calls.append(("repo", "docs/a.pdf", 1, 2))
|
||||
# Phase 152 (task 04): file A's vision span is in flight —
|
||||
# its image position (3/15) fires through the runner's OWN
|
||||
# on_image_progress closure (task 03's wiring), then the fake
|
||||
# parks.
|
||||
if on_image_progress is not None:
|
||||
on_image_progress(3, 15)
|
||||
self.image_hook_calls.append((3, 15))
|
||||
self.parked_a.set()
|
||||
await asyncio.to_thread(self.release_a.wait)
|
||||
if progress is not None:
|
||||
# The SECOND progress call names file B — the closure under
|
||||
# test must reset the image position the closure fired
|
||||
# while file A was in flight (a file with no images must
|
||||
# not inherit the previous PDF's position).
|
||||
progress("repo", "docs/b.md", 2, 2)
|
||||
self.hook_calls.append(("repo", "docs/b.md", 2, 2))
|
||||
self.parked_b.set()
|
||||
await asyncio.to_thread(self.release_b.wait)
|
||||
return self.summary
|
||||
|
||||
|
||||
def test_image_position_resets_per_file(
|
||||
fresh_sync_status: None,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Phase 152 (task 01 + task 04): the pairing rule, extended — a
|
||||
``progress`` call names the file AND clears the image position;
|
||||
every image position until the next ``progress`` call belongs to
|
||||
it. File A's live position (3/15, fired through the runner's OWN
|
||||
``_image_hook`` closure while parked on file A — task 04: the
|
||||
phase-64/129 idiom, the importer's image hook's exact path) is
|
||||
0/0 the moment the runner's own ``_hook`` names file B: a file
|
||||
with no images must not inherit the previous PDF's position
|
||||
(unlike the chunk position, not every file embeds chunks). The
|
||||
success terminal keeps the position at 0/0 and the FILE counts
|
||||
keep their final values (the convention split)."""
|
||||
clone = _GatedClone(threading.Event(), threading.Event())
|
||||
fake_import = _TwoFileImport(ImportSummary(files=2, added=1, updated=1))
|
||||
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
|
||||
|
||||
thread, errors = _start_run()
|
||||
try:
|
||||
assert clone.started.wait(5.0), "the run never reached the clone phase"
|
||||
clone.release.set()
|
||||
assert fake_import.parked_a.wait(5.0), "the run never parked on file A"
|
||||
# File A is named by the runner's own _hook (a wiring
|
||||
# regression to None would leave current_file null here).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "import"
|
||||
assert s["current_file"] == "repo/docs/a.pdf"
|
||||
assert s["files_done"] == 1 and s["files_total"] == 2
|
||||
# File A's vision span is in flight: its image position is
|
||||
# live on the status — fired through the runner's OWN
|
||||
# _image_hook closure (the fake fired it before parking;
|
||||
# a wiring regression to None would leave it 0/0 here).
|
||||
assert s["images_done"] == 3 and s["images_total"] == 15
|
||||
# Release file A: the runner's _hook names file B — the image
|
||||
# position must reset to 0/0 in the same call (the pairing
|
||||
# rule, extended; the closure under test, driven through the
|
||||
# fake's progress kwarg).
|
||||
fake_import.release_a.set()
|
||||
assert fake_import.parked_b.wait(5.0), "the run never parked on file B"
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "import"
|
||||
assert s["current_file"] == "repo/docs/b.md"
|
||||
assert s["files_done"] == 2 and s["files_total"] == 2
|
||||
# RESET per file: file B (no images) does not inherit file
|
||||
# A's 3/15.
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
fake_import.release_b.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
fake_import.release_a.set()
|
||||
fake_import.release_b.set()
|
||||
thread.join(10.0)
|
||||
assert not thread.is_alive()
|
||||
assert errors == [], f"the runner raised: {errors!r}"
|
||||
# Wiring: BOTH progress calls went through the runner's own
|
||||
# closure (the reset above was its doing), and the image pair
|
||||
# went through the runner's own _image_hook closure.
|
||||
assert fake_import.hook_calls == [
|
||||
("repo", "docs/a.pdf", 1, 2),
|
||||
("repo", "docs/b.md", 2, 2),
|
||||
]
|
||||
assert fake_import.image_hook_calls == [(3, 15)]
|
||||
# Success terminal: the image position stays 0/0 (cleared in the
|
||||
# terminal too — every terminal clears it) and the FILE counts
|
||||
# keep their final values (the convention split).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "success"
|
||||
assert s["phase"] is None
|
||||
assert s["current_file"] is None
|
||||
assert s["files_done"] == 2 and s["files_total"] == 2
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
|
||||
|
||||
def test_image_position_clears_at_phase_transitions_and_terminal(
|
||||
fresh_sync_status: None,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Phase 152 (task 01 + task 04): the image position is CLEARED
|
||||
(0/0) at every point the import phase ends. This run's summary
|
||||
changes the KB, so BOTH post-import transitions run and are
|
||||
pinned: the import→``"overview"`` transition and the
|
||||
import→``"summaries"`` transition (the changed-KB branch) each
|
||||
clear the live position (3/15, fired mid-import through the
|
||||
runner's OWN ``_image_hook`` closure — task 04: the
|
||||
phase-64/129 idiom, the importer's span's exact path) to 0/0,
|
||||
and the success terminal keeps it 0/0 — through the whole span
|
||||
the FILE counts keep their final values (the convention split:
|
||||
the file counts survive for the result line's context, the image
|
||||
position is a sub-position of a file that's done)."""
|
||||
clone = _GatedClone(threading.Event(), threading.Event())
|
||||
import_release = threading.Event()
|
||||
overview_started = threading.Event()
|
||||
overview_release = threading.Event()
|
||||
folders_started = threading.Event()
|
||||
folders_release = threading.Event()
|
||||
fake_import = _GatedImport(
|
||||
ImportSummary(files=3, added=1, updated=1, unchanged=1),
|
||||
threading.Event(),
|
||||
import_release,
|
||||
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair to clear
|
||||
)
|
||||
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
|
||||
|
||||
# Park the post-import steps (re-monkeypatched over the seams
|
||||
# helper's no-ops): the test reads the status DURING the
|
||||
# "overview" phase and DURING the "summaries" phase — the two
|
||||
# import-phase-ending transitions the task pins (the same shape
|
||||
# as the phase-129 chunk-position transition test above).
|
||||
async def gated_overview(llm: object, session: object = None) -> bool:
|
||||
overview_started.set()
|
||||
await asyncio.to_thread(overview_release.wait)
|
||||
return True
|
||||
|
||||
async def gated_folders(
|
||||
db: object,
|
||||
llm: object,
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
folders_started.set()
|
||||
await asyncio.to_thread(folders_release.wait)
|
||||
return {"generated": 0, "failed": 0, "pruned": 0}
|
||||
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", gated_overview)
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
|
||||
|
||||
thread, errors = _start_run()
|
||||
try:
|
||||
assert clone.started.wait(5.0), "the run never reached the clone phase"
|
||||
clone.release.set()
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
# Mid-import: the file is live; its image position went live
|
||||
# too — fired through the runner's OWN _image_hook closure
|
||||
# (image_pairs[0] — the importer's span would do the same).
|
||||
s = sync_api.sync_status()
|
||||
assert s["phase"] == "import"
|
||||
assert s["images_done"] == 3 and s["images_total"] == 15
|
||||
# Release the import: it returns (added+updated > 0) → the
|
||||
# phase machine moves to "overview" and the position clears.
|
||||
import_release.set()
|
||||
assert overview_started.wait(5.0), "the run never reached the overview"
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "overview"
|
||||
# CLEARED at the import→overview transition (0/0) ... the file
|
||||
# counts KEEP their final values (the convention split).
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
overview_release.set()
|
||||
# ... and the changed-KB branch moves to "summaries" (the
|
||||
# second import-phase-ending transition — cleared there too,
|
||||
# idempotently) before parking.
|
||||
assert folders_started.wait(5.0), "the run never reached the summaries"
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "summaries"
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
folders_release.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
import_release.set()
|
||||
overview_release.set()
|
||||
folders_release.set()
|
||||
thread.join(10.0)
|
||||
assert not thread.is_alive()
|
||||
assert errors == [], f"the runner raised: {errors!r}"
|
||||
assert fake_import.image_hook_calls == [(3, 15)]
|
||||
# Success terminal: still 0/0 (the terminal clears too — same
|
||||
# spots as current_file), file counts kept.
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "success"
|
||||
assert s["phase"] is None
|
||||
assert s["current_file"] is None
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
|
||||
|
||||
# ---------- phase 134 (task 01): the cooperative cancel (5th state) ----------
|
||||
#
|
||||
# The cancel runs are driven through the REAL ``start_sync`` on a
|
||||
@@ -1629,11 +2169,17 @@ def test_cancel_endpoint_stops_the_in_flight_run(
|
||||
ImportSummary(files=3, added=1, updated=1, unchanged=1),
|
||||
threading.Event(),
|
||||
threading.Event(),
|
||||
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair before the cancel
|
||||
)
|
||||
_patch_sync_seams(monkeypatch, tmp_path, fake_import, _fast_clone)
|
||||
|
||||
def act(loop: asyncio.AbstractEventLoop) -> dict[str, str]:
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
# Phase 152 (task 04): the live image position in flight when
|
||||
# the cancel converts was fired through the runner's OWN
|
||||
# _image_hook closure (image_pairs[0] — the fake fired it
|
||||
# before parking, like the importer's span would); the
|
||||
# cancelled terminal must clear it.
|
||||
return loop.run_until_complete(sync_api.cancel_sync())
|
||||
|
||||
try:
|
||||
@@ -1665,6 +2211,11 @@ def test_cancel_endpoint_stops_the_in_flight_run(
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3 # final counts kept
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
# Phase 152 (task 04): the image position clears in the CANCELLED
|
||||
# terminal too — every terminal clears it (the live 3/15 fired
|
||||
# through the closure in flight is gone; the FILE counts above
|
||||
# keep their final values).
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
# PLAN §9: the ample-logging line for the converted cancel.
|
||||
assert any(
|
||||
r.getMessage()
|
||||
@@ -1879,6 +2430,7 @@ def test_event_set_during_import_cancels_before_the_lite_spans(
|
||||
session: object = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
|
||||
Reference in New Issue
Block a user