phase: 152_sync_progress_image_pass
All criteria verified. The CLI path is hook-less (only `app/api/sync.py` passes the hook), and the byte-identity is pinned by `test_on_image_progress_none_run_is_byte_identical`. ## Final Report — Phase 152 final verification pass - **Verified (no new code needed):** `SyncStatus.images_done/images_total` + reset/clear rules in `app/api/sync.py`; `on_image_progress` threaded through the importer's three PDF vision spans with the continuous per-file counter; the `_image_hook` closure; `sources.js` `· image n/m` label part (box + title + aria). - **C1** status climbs 1→N + UI shows `· image n/m`: ✅ unit 3/15→15/15, integration 1/15→7/15, E2E 1/4→4/4 (title/aria ride the full string) - **C2** never frozen: ✅ E2E asserts ≥3 distinct positions, starting at 1, non-decreasing, ≤ total, reaching it - **C3** cleared at overview/summaries + all terminals; per-file reset via `_hook`; no stale position for image-less files: ✅ unit pins + E2E settle (image part gone in box/title/aria, endpoint 0/0) - **C4** `/api/sync/status` additive-only: ✅ full-dict pin passes (all pre-existing keys unchanged + the two new 0/0 keys) - **C5** hook-less caller byte-identical: ✅ CLI passes no hook; `test_on_image_progress_none_run_is_byte_identical` green - **C6** `uv run pytest`: **3526 passed, exit 0** (95s); dedicated E2E in isolation: **2 passed** (95s); `uv run ruff check . && uv run pyright`: **clean**; coverage: **99% on `app/` (>90% gate)** - Regression: all five neighbouring E2E suites green in isolation (progress_box 8, chunk_progress 2, label_mobile 4, pdf_support 6, image_documents 4) - **C7** atomic commit: left to the harness (no git index/history touched per rules) - **Docs:** README.md — sync-label paragraph already updated this phase, accurately; all `.env.example` vars documented; routes/commands verified — **no changes needed**. AGENTS.md — all references (PLAN, user_stories, test_smoke.py, .env.example, gpg config, .gitignore) verified — **no changes needed** - **Defects found in prior work:** none (one prior env-leak defect was already fixed inside this phase's task 04, conftest) - No live-infrastructure touched; no ntfy used. **Next pending phase:** none — `todo/` held only this phase
This commit is contained in:
@@ -127,6 +127,28 @@ setSyncProgress and pins the result mode behaviorally (a settle
|
||||
result line reveals the box, "" empties + hides it); the bare
|
||||
settle's row now shows the box LEFT ALONE (the settle label rides
|
||||
the button only — A2).
|
||||
|
||||
Phase 152 (task 03) adds the label's FOURTH part — the vision-image
|
||||
position: fmtSyncPhaseLabel's import fall-through gains "· image
|
||||
n/m" (rendered while images_total > 0 — a PDF's vision span is in
|
||||
flight; n the 1-based image about to be described, m the file's
|
||||
total images, ONE continuous counter across the PDF's three vision
|
||||
spans), the overview/summaries branches return image: "" (the
|
||||
position belongs to the import phase only — the status's image
|
||||
fields are cleared at those transitions, so the branches never read
|
||||
them), and the bare builder stays image-free (the upload label and
|
||||
the 202/409 click stay bare). setSyncLabel(mainText, countText =
|
||||
"", chunkText = "", running = false, imageText = "") appends the
|
||||
image part LAST to the composed full string — so the box, the
|
||||
button title, and the running aria-label all carry
|
||||
"Syncing… <file> (2/3) · chunk 14/38 · image 3/15". The image part
|
||||
gets NO span of its own (the running button is the spinner — its
|
||||
parts are hidden, phase 149): it rides ONLY the composed string,
|
||||
while the chunk part keeps its .sync-label-chunk span; the two
|
||||
positions are distinct and each disappears independently. The node
|
||||
harness gains the two in-flight image rows (both parts at once —
|
||||
the owner's stuck label — and the image-only pictorial PDF) and
|
||||
pins the composed full string byte-for-byte.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -298,8 +320,9 @@ def test_fmt_sync_label_returns_the_parts_contract() -> None:
|
||||
|
||||
def test_set_sync_label_is_the_single_parts_writer() -> None:
|
||||
"""setSyncLabel(mainText, countText = "", chunkText = "",
|
||||
running = false) — Phase 128 (task 01) + Phase 129 (task 04):
|
||||
the SINGLE point that writes #sync-label. The MAIN part gets
|
||||
running = false, imageText = "") — Phase 128 (task 01) + Phase
|
||||
129 (task 04) + Phase 152 (task 03): the SINGLE point that writes
|
||||
#sync-label. The MAIN part gets
|
||||
mainText (textContent only — the house rule, no HTML strings);
|
||||
the COUNT part gets countText when present (textContent +
|
||||
hidden=false — the "(n/m)" position is visible, NEVER clipped)
|
||||
@@ -309,11 +332,18 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
|
||||
hidden=true — the part renders only while a file's embed batch
|
||||
is in flight). Then the UNTRUNCATED full label — mainText +
|
||||
(countText ? " " + countText : "") + (chunkText ? " " + chunkText
|
||||
: "") — rides the button title (A4 — the screen reader hears the
|
||||
whole "Syncing… <file> (2/3) · chunk 14/38" there, and in the
|
||||
box — see below). Phase 149 (task 01): the fourth parameter
|
||||
: "") + (imageText ? " " + imageText : "") — rides the button
|
||||
title (A4 — the screen reader hears the
|
||||
whole "Syncing… <file> (2/3) · chunk 14/38 · image 3/15" there,
|
||||
and in the box — see below). The image part (phase 152) gets NO
|
||||
span of its own (the running button is the spinner — its parts
|
||||
are hidden, phase 149): it rides only the composed full string.
|
||||
Phase 149 (task 01): the fourth parameter
|
||||
`running` (the in-flight flag — the running entry passes true,
|
||||
the settle the default false). Phase 150 (task 02): while
|
||||
the settle the default false); phase 152 (task 03) adds the
|
||||
fifth — `imageText` (the bare builders and the non-running calls
|
||||
pass nothing, so it defaults "" and the part stays out of the
|
||||
string). Phase 150 (task 02): while
|
||||
running, the SAME full label goes into the detail box
|
||||
(#sync-progress — the SOLE aria-live region, the announcer the
|
||||
screen reader hears) through the single box writer
|
||||
@@ -324,10 +354,20 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
|
||||
running / the ship value "Sync sources" otherwise
|
||||
(overwrites/restores only — never removes)."""
|
||||
body = _fn(_js(), "setSyncLabel")
|
||||
assert (
|
||||
'function setSyncLabel(mainText, countText = "", chunkText = "", running = false)'
|
||||
in body
|
||||
), "phase 149: the in-flight flag is the fourth parameter"
|
||||
# the exact signature, byte-for-byte (concatenated — the literal
|
||||
# is 105 columns, over the line cap): main, count, chunk, the
|
||||
# phase-149 `running` flag (fourth), the phase-152 `imageText`
|
||||
# (fifth — the bare builders + non-running calls pass nothing,
|
||||
# so it defaults to the empty string).
|
||||
expected_sig = (
|
||||
'function setSyncLabel(mainText, countText = "", chunkText = "",'
|
||||
' running = false, imageText = "")'
|
||||
)
|
||||
assert expected_sig in body, (
|
||||
"phase 149: the in-flight flag is the fourth parameter; phase"
|
||||
" 152: the image part is the fifth, defaulting to the empty"
|
||||
" string (the part stays out of the composed full string)"
|
||||
)
|
||||
# the parts: textContent only (the house rule), null-guarded refs
|
||||
assert "syncLabelMain.textContent = mainText" in body
|
||||
assert "syncLabelCount.textContent = countText" in body
|
||||
@@ -353,8 +393,12 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
|
||||
"const full =\n"
|
||||
" mainText +\n"
|
||||
" (countText ? ` ${countText}` : \"\") +\n"
|
||||
" (chunkText ? ` ${chunkText}` : \"\");"
|
||||
) in body, ("the full label is main + ' ' + count + ' ' + chunk (spaces = the CSS gaps)")
|
||||
" (chunkText ? ` ${chunkText}` : \"\") +\n"
|
||||
" (imageText ? ` ${imageText}` : \"\");"
|
||||
) in body, (
|
||||
"the full label is main + ' ' + count + ' ' + chunk + ' ' +"
|
||||
" image — the image part LAST (spaces = the CSS gaps)"
|
||||
)
|
||||
i_full = body.find("const full")
|
||||
i_title = body.find("syncBtn.title = full")
|
||||
i_box = body.find("if (running) setSyncProgress(full);")
|
||||
@@ -375,7 +419,8 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
|
||||
|
||||
def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
|
||||
"""fmtSyncPhaseLabel(status) — the SYNC job's phase-aware label
|
||||
(phase 98 task 02, D2; phase 128 task 01: the PARTS). "overview"
|
||||
(phase 98 task 02, D2; phase 128 task 01: the PARTS; phase 152
|
||||
task 03: the IMAGE part). "overview"
|
||||
→ main exactly "Writing KB overview…" (no folder, no count part);
|
||||
"summaries" → main "Summarizing folders… <current_summary>" +
|
||||
count "(n/m)" — the folder part (one leading space +
|
||||
@@ -385,7 +430,14 @@ def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
|
||||
a total); anything else (the null prelude, "import") → the
|
||||
phase-64 fmtSyncLabel parts, byte-identical (same builder, same
|
||||
fields, same order). The UPLOAD job never enters here — its status
|
||||
has no phase (D2)."""
|
||||
has no phase (D2). Phase 152 (task 03): the import fall-through
|
||||
is the ONLY branch that reads the status's image fields — "·
|
||||
image n/m" renders while images_total > 0 (a PDF's vision span is
|
||||
in flight — n the 1-based image about to be described, m the
|
||||
file's total images) and hides at 0/0; the overview/summaries
|
||||
branches return image: "" and never read the fields (the position
|
||||
belongs to the import phase only — the runner clears it at those
|
||||
transitions)."""
|
||||
body = _fn(_js(), "fmtSyncPhaseLabel")
|
||||
assert "function fmtSyncPhaseLabel(status)" in body
|
||||
# "overview": the exact copy, no folder, no counts.
|
||||
@@ -421,14 +473,30 @@ def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
|
||||
)
|
||||
assert "chunks_total" not in ov, "the overview branch never reads the chunk fields"
|
||||
assert "chunks_total" not in sum_, "the summaries branch never reads the chunk fields"
|
||||
# phase 152 (task 03): the overview/summaries labels carry NO
|
||||
# image part either (the position belongs to the import phase
|
||||
# only) and never read the status's image fields (the runner
|
||||
# clears them at those transitions).
|
||||
assert 'image: ""' in ov, "the overview label carries no image part"
|
||||
assert 'image: ""' in sum_, (
|
||||
"the summaries label carries no image part (its (n/m) is its position)"
|
||||
)
|
||||
assert "images_total" not in ov, "the overview branch never reads the image fields"
|
||||
assert "images_total" not in sum_, "the summaries branch never reads the image fields"
|
||||
# anything else (null prelude, "import"): byte-identical to today's
|
||||
# sync label — same builder, same fields, same order — plus the
|
||||
# phase-129 chunk part: "· chunk n/m" ONLY while chunks_total > 0
|
||||
# (a file's embed batch is in flight — n the 1-based first chunk
|
||||
# of the batch, m the file's total content chunks, batch
|
||||
# granularity), "" otherwise (0/0 before the first batch — the
|
||||
# part hides). The fall-through is the ONLY branch that reads the
|
||||
# status's chunk fields.
|
||||
# part hides) + the phase-152 image part: "· image n/m" ONLY
|
||||
# while images_total > 0 (a PDF's vision span is in flight — n
|
||||
# the 1-based image about to be described, m the file's total
|
||||
# images — ONE continuous counter across the PDF's three vision
|
||||
# spans), "" otherwise (0/0 — the part hides). The fall-through
|
||||
# is the ONLY branch that reads the status's chunk + image
|
||||
# fields, and the two parts are independent (both may show,
|
||||
# each disappears on its own 0/0).
|
||||
i_destr = body.find("const { main, count } = fmtSyncLabel(")
|
||||
assert -1 < i_destr < i_fall, "the fall-through destructures the phase-64 parts"
|
||||
fall = body[i_destr:]
|
||||
@@ -444,11 +512,22 @@ def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
|
||||
i_guard = fall.find("status.chunks_total > 0")
|
||||
i_else = fall.find(": \"\";", i_guard)
|
||||
assert -1 < i_guard < i_else, "0/0 (before the first batch) → the empty chunk part"
|
||||
assert "return { main, count, chunk }" in fall
|
||||
# phase 152 (task 03): the image part — "· image n/m" ONLY while
|
||||
# images_total > 0, the exact template + the 0/0 → "" guard, and
|
||||
# the return gains the fourth part.
|
||||
assert "status.images_total > 0" in fall, (
|
||||
"the image part renders only while a vision span is in flight"
|
||||
)
|
||||
assert "`· image ${status.images_done}/${status.images_total}`" in fall
|
||||
i_iguard = fall.find("status.images_total > 0")
|
||||
i_ielse = fall.find(": \"\";", i_iguard)
|
||||
assert -1 < i_iguard < i_ielse, "0/0 (before the first image) → the empty image part"
|
||||
assert "return { main, count, chunk, image }" in fall
|
||||
# the bare builder (the upload job + the 202/409 click) returns NO
|
||||
# chunk part — those labels stay bare.
|
||||
# chunk OR image part — those labels stay bare.
|
||||
bare = _fn(_js(), "fmtSyncLabel")
|
||||
assert "chunk" not in bare, "fmtSyncLabel stays chunk-free (upload + the click stay bare)"
|
||||
assert "image" not in bare, "fmtSyncLabel stays image-free (upload + the click stay bare)"
|
||||
|
||||
|
||||
def test_enter_running_state_is_phase_aware_and_writes_the_parts() -> None:
|
||||
@@ -460,13 +539,16 @@ def test_enter_running_state_is_phase_aware_and_writes_the_parts() -> None:
|
||||
endpoint has no phase) and the 202/409 click keep the bare
|
||||
fmtSyncLabel parts, byte-identical. Phase 128 (task 01): the entry
|
||||
itself no longer writes the label — it hands the parts to the
|
||||
SINGLE writer (setSyncLabel(main, count, chunk) — phase 129:
|
||||
the chunk part destructures with a "" default), which lands the
|
||||
SINGLE writer (setSyncLabel(main, count, chunk, running, image) —
|
||||
phase 129: the chunk part destructures with a "" default, phase
|
||||
152: the image part does too), which lands the
|
||||
main part, the pinned count part (hidden when absent), the pinned
|
||||
chunk part (phase 129 — hidden when absent), and the FULL
|
||||
untruncated label on the button title + #sync-progress (A4 — set
|
||||
for EVERY running label; CSS ellipsizes the main part only).
|
||||
Phase 150 (task 02): the box write rides in the writer too
|
||||
for EVERY running label; CSS ellipsizes the main part only —
|
||||
phase 152: with the image part appended last when a PDF's vision
|
||||
span is in flight; it gets no span of its own). Phase 150 (task
|
||||
02): the box write rides in the writer too
|
||||
(setSyncLabel → setSyncProgress, running mode)."""
|
||||
body = _fn(_js(), "enterSyncRunningState")
|
||||
assert (
|
||||
@@ -487,13 +569,16 @@ def test_enter_running_state_is_phase_aware_and_writes_the_parts() -> None:
|
||||
# the parts are handed to the single writer (A4 lives in it) —
|
||||
# phase 129: the chunk part destructures with a "" default (the
|
||||
# bare builders — the upload job, the 202/409 click — return no
|
||||
# chunk part, so the part stays hidden there); phase 149: the
|
||||
# chunk part, so the part stays hidden there); phase 152: the
|
||||
# image part does too (it rides only the composed full string —
|
||||
# no span); phase 149: the
|
||||
# call passes `running = true` (the detail box reveals + the
|
||||
# full-label aria-label ride in the writer).
|
||||
assert "const { main, count, chunk = \"\" } =" in body, (
|
||||
"the chunk part destructures with the '' default (bare builders return none)"
|
||||
assert 'const { main, count, chunk = "", image = "" } =' in body, (
|
||||
"the chunk + image parts destructure with the '' defaults"
|
||||
" (bare builders return none)"
|
||||
)
|
||||
i_set = body.find("setSyncLabel(main, count, chunk, true)")
|
||||
i_set = body.find("setSyncLabel(main, count, chunk, true, image)")
|
||||
assert -1 < i_fall < i_set, "the parts are written after the split (running = true)"
|
||||
assert body.count("setSyncLabel(") == 1, "one writer call per entry"
|
||||
assert "syncBtn.title =" not in body, ("the title write lives in setSyncLabel (A4)")
|
||||
@@ -713,6 +798,25 @@ enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
|
||||
current_chunk: 14, chunks_total: 38,
|
||||
});
|
||||
show("import-inflight");
|
||||
// 1b. Phase 152 (task 03): import, a PDF's vision span in flight
|
||||
// (the owner's stuck label — the last chunk position 125/126 is
|
||||
// STILL on the status + the climbing image position 3/15): the
|
||||
// composed full string carries BOTH parts (chunk keeps its span,
|
||||
// the image part rides the string only — no span):
|
||||
enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
|
||||
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 3,
|
||||
current_chunk: 125, chunks_total: 126, images_done: 3, images_total: 15,
|
||||
});
|
||||
show("import-image-span");
|
||||
// 1c. Phase 152 (task 03): import, a purely pictorial PDF (no text
|
||||
// chunks — the chunk position never left 0/0) in its vision
|
||||
// span: the image part shows ALONE (the chunk part stays hidden
|
||||
// + empty — the two parts are independent):
|
||||
enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
|
||||
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 3,
|
||||
current_chunk: 0, chunks_total: 0, images_done: 7, images_total: 7,
|
||||
});
|
||||
show("import-image-only");
|
||||
// 2. import before the first batch (0/0) — the chunk part hides:
|
||||
enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
|
||||
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 3,
|
||||
@@ -893,6 +997,96 @@ test-document-viewer pattern). The pinned "· chunk n/m" part (the
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_image_part_renders_only_in_the_import_phase_under_node() -> None:
|
||||
"""Phase 152 (task 03): behavioral check (node) of the REAL label
|
||||
path — the extracted fmtSyncPhaseLabel / setSyncLabel /
|
||||
enterSyncRunningState run against the stub element refs. The
|
||||
pinned "· image n/m" part (the task's fixture values):
|
||||
|
||||
* import, a PDF's vision span in flight (images_total 15 > 0)
|
||||
WITH the last chunk position still on the status (125/126):
|
||||
the composed full string carries BOTH parts — the box, the
|
||||
button title, and the running aria-label all read exactly
|
||||
"Syncing… src/a/b.md (2/3) · chunk 125/126 · image 3/15"
|
||||
(the owner's stuck label, now moving); the chunk part KEEPS
|
||||
its span ("· chunk 125/126" visible) and the image part gets
|
||||
NO span of its own (it rides the composed string only — the
|
||||
running button is the spinner, phase 149);
|
||||
* import, a purely pictorial PDF (the chunk position never left
|
||||
0/0) in its vision span: the image part shows ALONE —
|
||||
"Syncing… src/a/b.md (2/3) · image 7/7" — and the chunk part
|
||||
stays hidden + empty (the two parts are distinct and each
|
||||
disappears independently on its own 0/0);
|
||||
* every OTHER running state (the pre-batch import, the prelude,
|
||||
the 202/409 click, the upload run, the overview, the
|
||||
summaries — the status's image fields are 0/0 or cleared at
|
||||
those transitions) leaves the composed string image-free.
|
||||
|
||||
The box / Cancel-control / is-running contracts of the running
|
||||
state apply to the new rows exactly as to every other running
|
||||
state (the harness's other pins cover them per tag)."""
|
||||
rows = _chunk_harness_rows()
|
||||
by_tag = {row[0]: row for row in rows}
|
||||
# row = [tag, title, announcer (the box's text — the sole live
|
||||
# region, phase 150), main, count, countHidden, chunk,
|
||||
# chunkHidden, cancelHidden, cancelDisabled, progressText,
|
||||
# progressHidden, btnRunning, iconSpinning, ariaLabel]
|
||||
|
||||
# 1b. both parts at once — the composed full string, byte-for-byte
|
||||
# (the image part LAST, after the chunk part), on the box, the
|
||||
# title, and the running aria-label (A4) — the chunk span
|
||||
# keeps its own part.
|
||||
full = "Syncing… src/a/b.md (2/3) · chunk 125/126 · image 3/15"
|
||||
row = by_tag["import-image-span"]
|
||||
assert row[1] == full, row[1]
|
||||
assert row[2] == full, "the announcer (the box) hears the whole thing (A4)"
|
||||
assert row[14] == full, "the running button's accessible name is the full label"
|
||||
assert row[10] == full, "the detail box carries the EXACT full label"
|
||||
assert row[6] == "· chunk 125/126" and row[7] is False, (
|
||||
"the chunk part keeps its own span alongside the image part"
|
||||
)
|
||||
|
||||
# 1c. the image part alone (the chunk position never left 0/0 —
|
||||
# the two parts are independent).
|
||||
row = by_tag["import-image-only"]
|
||||
assert row[1] == "Syncing… src/a/b.md (2/3) · image 7/7", row[1]
|
||||
assert row[2] == row[1] and row[14] == row[1], row
|
||||
assert row[6] == "" and row[7] is True, (
|
||||
"the chunk part stays hidden + empty (its own 0/0)"
|
||||
)
|
||||
|
||||
# The pre-batch import (0/0 on BOTH positions) and every other
|
||||
# running state: the composed string is image-free (the part
|
||||
# hides on its own 0/0 — the overview/summaries transitions
|
||||
# clear the position server-side, the bare builders carry none).
|
||||
for tag, expected_title in (
|
||||
("import-inflight", "Syncing… src/a/b.md (2/3) · chunk 14/38"),
|
||||
("import-prebatch", "Syncing… src/a/b.md (2/3)"),
|
||||
("prelude", "Syncing…"),
|
||||
("click", "Syncing…"),
|
||||
("upload", "Importing"),
|
||||
("overview", "Writing KB overview…"),
|
||||
("summaries", "Summarizing folders… src/alpha (2/3)"),
|
||||
):
|
||||
row = by_tag[tag]
|
||||
assert row[1] == expected_title, (tag, row)
|
||||
assert "· image" not in row[1], (
|
||||
f"{tag}: the image part shows only while images_total > 0"
|
||||
)
|
||||
|
||||
# The running-state contracts apply to the new rows too: box
|
||||
# visible + is-running + spinning, the Cancel control shown +
|
||||
# enabled (a sync run in flight).
|
||||
for tag in ("import-image-span", "import-image-only"):
|
||||
row = by_tag[tag]
|
||||
assert row[11] is False, f"{tag}: the box must be visible while in flight"
|
||||
assert row[12] is True and row[13] is True, f"{tag}: is-running + spinning"
|
||||
assert row[8] is False and row[9] is False, (
|
||||
f"{tag}: the Cancel control is shown + enabled while a sync runs"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_progress_box_and_running_button_contract_under_node() -> None:
|
||||
"""Phase 149 (task 01) — the detail box + the icon-only running
|
||||
@@ -995,14 +1189,16 @@ def test_progress_box_and_running_button_contract_under_node() -> None:
|
||||
|
||||
|
||||
def test_every_other_state_passes_no_chunk_text() -> None:
|
||||
"""Source-level pin of the render-only-while-chunks_total>0
|
||||
contract: the import fall-through is the ONLY place in sources.js
|
||||
that reads the status's chunk fields (the summaries/overview
|
||||
branches and the bare builders never touch them), and every
|
||||
NON-running label emission passes no chunk text (the part hides):
|
||||
the settle/idle labels go through setSyncLabel(label) — the only
|
||||
running-flagged call site (phase 149: `running = true`) is the
|
||||
running entry."""
|
||||
"""Source-level pin of the render-only-while-chunks_total>0 +
|
||||
render-only-while-images_total>0 contracts: the import
|
||||
fall-through is the ONLY place in sources.js that reads the
|
||||
status's chunk OR image fields (the summaries/overview branches
|
||||
and the bare builders never touch them), and every
|
||||
NON-running label emission passes no chunk/image text (both
|
||||
parts hide): the settle/idle labels go through setSyncLabel(label)
|
||||
— the only running-flagged call site (phase 149: `running = true`)
|
||||
is the running entry (phase 152: passing the image part as its
|
||||
fifth argument)."""
|
||||
js = _js()
|
||||
body = _fn(js, "fmtSyncPhaseLabel")
|
||||
i_fall = body.find("fmtSyncLabel(")
|
||||
@@ -1010,12 +1206,21 @@ def test_every_other_state_passes_no_chunk_text() -> None:
|
||||
assert "chunks_total" not in pre and "current_chunk" not in pre, (
|
||||
"the overview/summaries branches never read the chunk fields"
|
||||
)
|
||||
assert "images_total" not in pre and "images_done" not in pre, (
|
||||
"the overview/summaries branches never read the image fields (phase 152)"
|
||||
)
|
||||
fall = body[i_fall:]
|
||||
assert "status.chunks_total" in fall and "status.current_chunk" in fall
|
||||
assert js.count("setSyncLabel(main, count, chunk, true)") == 1, (
|
||||
"the ONLY running-flagged call site is the running entry (phase 149)"
|
||||
assert "status.images_total" in fall and "status.images_done" in fall, (
|
||||
"the import fall-through is the ONLY reader of the image fields (phase 152)"
|
||||
)
|
||||
assert js.count("setSyncLabel(main, count, chunk, true, image)") == 1, (
|
||||
"the ONLY running-flagged call site is the running entry (phase 149"
|
||||
" — phase 152: with the image part)"
|
||||
)
|
||||
assert "setSyncLabel(label)" in js, (
|
||||
"the settle/idle labels stay two-arg (no chunk/image)"
|
||||
)
|
||||
assert "setSyncLabel(label)" in js, "the settle/idle labels stay two-arg (no chunk)"
|
||||
|
||||
|
||||
# ---------- the two-job tick (startSyncPolling) ----------
|
||||
|
||||
+508
-1
@@ -18,7 +18,8 @@ import base64
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import Callable
|
||||
import textwrap
|
||||
from collections.abc import Callable, Sequence
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
@@ -41,6 +42,11 @@ from app.rag.importer import (
|
||||
match_extension,
|
||||
)
|
||||
from app.rag.llm import EmbeddingError, LLMError
|
||||
from app.rag.pdf import (
|
||||
extract_pdf_images_with_stats,
|
||||
extract_pdf_page_texts,
|
||||
pictorial_pages,
|
||||
)
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
#: The original seven A9 formats as dotted suffixes (pre-phase-47 default
|
||||
@@ -2088,3 +2094,504 @@ def test_fallback_embedded_images_still_join_the_pipeline(
|
||||
assert image_doc.content != d1 and image_doc.content != d2
|
||||
finally:
|
||||
_cleanup_source(db, "both")
|
||||
|
||||
|
||||
# ---------- phase 152 (task 02): on_image_progress plumbing ----------
|
||||
|
||||
#: A text block comfortably over the phase-140 100-char sparseness bar
|
||||
#: (a USABLE layer, zero pictorial candidates even with the toggle on
|
||||
#: — the "no vision span" class). Inserted wrapped at 60 chars: a
|
||||
#: single long line would be CLIPPED at the page edge on extraction,
|
||||
#: while the wrapped multi-line read round-trips (the phase-140
|
||||
#: fixture's class property).
|
||||
_P152_DENSE_TEXT = (
|
||||
"The quadlet unit file defines the container interface, the image, "
|
||||
"the network, and the restart policy. Every key maps to a docker "
|
||||
"flag in the generated service file, and comments use the usual "
|
||||
"semicolon form throughout the whole document."
|
||||
)
|
||||
assert len(_P152_DENSE_TEXT.strip()) > 100 # the class property, not a typo
|
||||
|
||||
|
||||
def _p152_solid_png(value: int) -> bytes:
|
||||
"""A 40x30 solid-color PNG (channel *value*): every value used
|
||||
below stays under the phase-151 blank gate's 235 mean floor (kept,
|
||||
never blank-skipped), and distinct values are distinct images (the
|
||||
dedupe keeps every one)."""
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 40, 30), 0)
|
||||
pix.clear_with(value)
|
||||
return pix.tobytes("png")
|
||||
|
||||
|
||||
def _build_image_pdf(
|
||||
path: Path,
|
||||
values: Sequence[int],
|
||||
dense: bool = True,
|
||||
repeat_first: bool = False,
|
||||
) -> None:
|
||||
"""A REAL one-page PDF: *dense* a text block over the 100-char bar
|
||||
(a USABLE layer — the fallback never fires; zero pictorial
|
||||
candidates even with the toggle on) carrying *values* the
|
||||
solid-color embedded images (one placement each, distinct).
|
||||
``dense=False`` inserts the sparse ``BESTA`` marker instead (a
|
||||
pictorial candidate under the default 100-char bar).
|
||||
``repeat_first`` adds a SECOND placement of the first image (the
|
||||
same stream — one xref, the parser's repeated-placement class,
|
||||
phase 151: the dedupe drops it, the position count is
|
||||
post-dedupe)."""
|
||||
doc = fitz.open()
|
||||
page = doc.new_page()
|
||||
text = _P152_DENSE_TEXT if dense else "BESTA"
|
||||
for i, line in enumerate(textwrap.wrap(text, 60)):
|
||||
page.insert_text((72, 72 + i * 14), line)
|
||||
y = 300
|
||||
for value in values:
|
||||
page.insert_image(fitz.Rect(72, y, 112, y + 30), stream=_p152_solid_png(value))
|
||||
y += 40
|
||||
if repeat_first:
|
||||
page.insert_image(
|
||||
fitz.Rect(150, 300, 190, 330), stream=_p152_solid_png(values[0])
|
||||
)
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
def _build_pictorial_image_pdf(path: Path, values: Sequence[int]) -> None:
|
||||
"""A REAL two-page PDF — the ONE continuous counter's two-span
|
||||
case: page 1 is SPARSE (``BESTA`` — a pictorial candidate under
|
||||
the default 100-char bar), page 2 is dense (never a candidate) and
|
||||
carries *values* the solid-color embedded images. Both the
|
||||
phase-140 pictorial pass (before the document commit) and the
|
||||
phase-126 embedded-image pass (after it) fire on it."""
|
||||
doc = fitz.open()
|
||||
doc.new_page().insert_text((72, 72), "BESTA")
|
||||
page = doc.new_page()
|
||||
for i, line in enumerate(textwrap.wrap(_P152_DENSE_TEXT, 60)):
|
||||
page.insert_text((72, 72 + i * 14), line)
|
||||
y = 300
|
||||
for value in values:
|
||||
page.insert_image(fitz.Rect(72, y, 112, y + 30), stream=_p152_solid_png(value))
|
||||
y += 40
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
def _build_garbage_image_pdf(path: Path, values: Sequence[int]) -> None:
|
||||
"""A REAL two-page garbage-text PDF (every page XML-ish — the
|
||||
phase-133 gate fires, the page-screenshot fallback runs) with
|
||||
*values* the solid-color embedded images on page 2 — the fallback
|
||||
span + the embedded span, ONE continuous counter."""
|
||||
doc = fitz.open()
|
||||
for p in range(1, 3):
|
||||
page = doc.new_page()
|
||||
for i in range(10):
|
||||
page.insert_text((72, 72 + i * 14), f"<foo:p{p}>{i}</foo:p{p}>")
|
||||
y = 400
|
||||
for value in values:
|
||||
doc[1].insert_image(
|
||||
fitz.Rect(72, y, 112, y + 30), stream=_p152_solid_png(value)
|
||||
)
|
||||
y += 40
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
def _p152_summary_fields(s: ImportSummary) -> tuple:
|
||||
"""One run's observable outcome (the byte-identity pin's left
|
||||
side): every count a hook could plausibly move + the changed
|
||||
paths' RELS (the source names are equal by construction — the two
|
||||
roots carry the same directory name)."""
|
||||
return (
|
||||
s.files, s.added, s.updated, s.unchanged, s.pruned, s.errors,
|
||||
s.chunks, s.embed_batches, s.summaries, s.summary_errors,
|
||||
s.images_failed, s.pdfs_screenshotted, s.pdfs_pictorial,
|
||||
s.pdf_pictorial_pages, s.pdf_images_deduped, s.pdf_images_blank_skipped,
|
||||
tuple(rel for _, rel in s.changed_paths),
|
||||
)
|
||||
|
||||
|
||||
def _p152_rows(db, source: str) -> list[tuple]:
|
||||
"""A source's document rows (the byte-identity pin's right side),
|
||||
identity-blind: the fields a hook could plausibly move."""
|
||||
docs = db.scalars(select(Document).where(Document.source == source)).all()
|
||||
return sorted(
|
||||
(d.path, d.title, d.content, d.content_hash, d.is_image, d.summary,
|
||||
d.image_context)
|
||||
for d in docs
|
||||
)
|
||||
|
||||
|
||||
|
||||
def test_on_image_progress_embedded_pass_climbs_one_per_image(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""The phase-152 core pin: a PDF with N DISTINCT embedded images
|
||||
(one placement each — post-dedupe, post-blank-gate: the work
|
||||
actually about to happen) fires the hook ONCE per image, 1-based,
|
||||
with the span's total, at the START of each image's pipeline step:
|
||||
``(1, N) … (N, N)``. A repeated placement of the first image (the
|
||||
parser's dedupe, phase 151) adds NO position (the total is the
|
||||
post-dedupe count), a clean usable text layer burns no fallback or
|
||||
pictorial span (the embedded pass's totals stand alone), and the
|
||||
phase-147 fast path on a re-sync (raw bytes unchanged) burns no
|
||||
vision call and fires the hook never."""
|
||||
root = tmp_path / "imgprog"
|
||||
root.mkdir()
|
||||
values = (90, 130, 170, 210) # 4 distinct images, all under the blank floor
|
||||
_build_image_pdf(root / "report.pdf", values, repeat_first=True)
|
||||
n = len(values)
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = _pdf_settings(tmp_path, images=True)
|
||||
events: list[tuple[int, int]] = []
|
||||
try:
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root], llm, session=db,
|
||||
on_image_progress=lambda pos, total: events.append((pos, total)),
|
||||
)
|
||||
)
|
||||
# The PDF + its N image docs; the repeat placement deduped.
|
||||
assert (summary.files, summary.added) == (1, 1 + n)
|
||||
assert summary.pdf_images_deduped == 1
|
||||
assert (summary.images_failed, summary.pdfs_screenshotted,
|
||||
summary.pdfs_pictorial) == (0, 0, 0)
|
||||
# The embedded pass is the ONLY vision span: N slow calls.
|
||||
assert llm.vision_calls == n
|
||||
# The core pin: once per image, 1-based, the span's total.
|
||||
assert events == [(i, n) for i in range(1, n + 1)]
|
||||
# The phase-147 fast path: a re-sync (bytes unchanged) burns no
|
||||
# vision call and fires the hook never — the events list grows
|
||||
# not at all.
|
||||
summary2 = asyncio.run(
|
||||
import_sources(
|
||||
[root], llm, session=db,
|
||||
on_image_progress=lambda pos, total: events.append((pos, total)),
|
||||
)
|
||||
)
|
||||
assert summary2.unchanged == 1
|
||||
assert llm.vision_calls == n
|
||||
assert len(events) == n
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_on_image_progress_pictorial_then_embedded_is_one_counter(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""ASSUMPTION A3: the spans of ONE file are ONE continuous counter
|
||||
— the embedded pass starts where the pictorial pass ended (``base
|
||||
= images_done_so_far``): the positions climb ``1…C+N`` with NO
|
||||
restart and never above the combined total (the pictorial span's
|
||||
total is its candidate-page count, the embedded span's total the
|
||||
combined count)."""
|
||||
root = tmp_path / "imgprog-two"
|
||||
root.mkdir()
|
||||
values = (90, 130, 170)
|
||||
_build_pictorial_image_pdf(root / "ikea.pdf", values)
|
||||
# The span inputs, computed IN the test (no hardcoded lists — the
|
||||
# phase-140 idiom): the deterministic gate's candidates and the
|
||||
# post-dedupe/post-gate image count.
|
||||
pdf = root / "ikea.pdf"
|
||||
page_texts = extract_pdf_page_texts(pdf)
|
||||
n_cand = len(pictorial_pages(page_texts, min_chars=100))
|
||||
images, _ = extract_pdf_images_with_stats(pdf)
|
||||
n_img = len(images)
|
||||
assert (n_cand, n_img) == (1, len(values))
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
import_extensions="md,txt,pdf",
|
||||
images=True,
|
||||
image_dir=str(tmp_path / "images"),
|
||||
pdf_pictorial=True, # default on — the pass runs on the sparse page
|
||||
)
|
||||
events: list[tuple[int, int]] = []
|
||||
try:
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root], llm, session=db,
|
||||
on_image_progress=lambda pos, total: events.append((pos, total)),
|
||||
)
|
||||
)
|
||||
assert (summary.files, summary.added) == (1, 1 + n_img)
|
||||
assert (summary.pdfs_pictorial, summary.pdf_pictorial_pages) == (1, 1)
|
||||
# The continuous counter: 1…C (total C) then C+1…C+N (total
|
||||
# C+N) — no restart, never above the combined total.
|
||||
assert events == (
|
||||
[(i, n_cand) for i in range(1, n_cand + 1)]
|
||||
+ [(n_cand + i, n_cand + n_img) for i in range(1, n_img + 1)]
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_on_image_progress_fallback_span_climbs_into_embedded(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""The owner's stuck-label class: a garbage-text PDF's per-page
|
||||
screenshot describes (the phase-133 fallback span — the span that
|
||||
froze the label for minutes) report the rendered-page count
|
||||
(``1…M`` with total ``M``) and hand their reported count to the
|
||||
embedded pass, which continues the SAME counter (``M+1`` with
|
||||
total ``M+N``) — every slow call has a position before it."""
|
||||
root = tmp_path / "imgprog-fb"
|
||||
root.mkdir()
|
||||
_build_garbage_image_pdf(root / "garbage.pdf", (90,))
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = _pdf_settings(tmp_path, images=True)
|
||||
events: list[tuple[int, int]] = []
|
||||
try:
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root], llm, session=db,
|
||||
on_image_progress=lambda pos, total: events.append((pos, total)),
|
||||
)
|
||||
)
|
||||
assert summary.pdfs_screenshotted == 1
|
||||
assert (summary.files, summary.added, summary.images_failed) == (1, 2, 0)
|
||||
# 2 fallback pages (total 2) + 1 embedded image continuing at
|
||||
# 3 (total 3) — ONE counter across the two spans.
|
||||
assert events == [(1, 2), (2, 2), (3, 3)]
|
||||
# Every slow call (2 pages + 1 image) was preceded by a
|
||||
# position — the label never sits frozen on a busy run.
|
||||
assert llm.vision_calls == 3
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_on_image_progress_text_only_pdf_never_fires(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""A clean-text PDF with NO embedded images fires the hook NEVER,
|
||||
with BOTH default-on toggles: the garbage gate stays closed (no
|
||||
fallback span), the pictorial pass's deterministic gate finds zero
|
||||
candidates (a no-op — the clean-text byte-identity class), and the
|
||||
embedded pass's list is empty. The skipped spans report nothing
|
||||
and advance the counter not at all (there is no vision call at
|
||||
all)."""
|
||||
root = tmp_path / "imgprog-none"
|
||||
root.mkdir()
|
||||
_build_image_pdf(root / "clean.pdf", ()) # dense text, zero images
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
import_extensions="md,txt,pdf",
|
||||
images=True,
|
||||
image_dir=str(tmp_path / "images"),
|
||||
pdf_pictorial=True, # default on — the pass runs and no-ops
|
||||
)
|
||||
events: list[tuple[int, int]] = []
|
||||
try:
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root], llm, session=db,
|
||||
on_image_progress=lambda pos, total: events.append((pos, total)),
|
||||
)
|
||||
)
|
||||
assert (summary.files, summary.added) == (1, 1)
|
||||
assert (summary.pdfs_screenshotted, summary.pdfs_pictorial) == (0, 0)
|
||||
assert llm.vision_calls == 0 # no span ran — zero slow calls
|
||||
assert events == []
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_on_image_progress_toggle_off_reports_nothing(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""The ``images`` toggle OFF skips the embedded span entirely (the
|
||||
prune guard keeps any existing image docs) — a PDF that WOULD
|
||||
carry N embedded images reports nothing (a skipped span advances
|
||||
the counter not at all) and indexes as text only."""
|
||||
root = tmp_path / "imgprog-off"
|
||||
root.mkdir()
|
||||
_build_image_pdf(root / "off.pdf", (90, 130))
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = _pdf_settings(tmp_path, images=False)
|
||||
events: list[tuple[int, int]] = []
|
||||
try:
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root], llm, session=db,
|
||||
on_image_progress=lambda pos, total: events.append((pos, total)),
|
||||
)
|
||||
)
|
||||
assert (summary.files, summary.added) == (1, 1)
|
||||
assert llm.vision_calls == 0
|
||||
assert events == []
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_on_image_progress_none_run_is_byte_identical(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""The regression pin: a hook-less run (the CLI/chat default —
|
||||
``on_image_progress=None``, every call site a guarded no-op) lands
|
||||
byte-identical to a run WITH the hook: the same documents (path,
|
||||
title, content, digest, image flag, summary, context) and the same
|
||||
summary counters. The hook observes; it never changes the run."""
|
||||
# Both roots carry the SAME directory name — the source (the
|
||||
# walk's root.name) is the document's provenance and part of the
|
||||
# stored image context, so the comparison needs it equal.
|
||||
root_a = tmp_path / "a" / "same_src"
|
||||
root_a.mkdir(parents=True)
|
||||
root_b = tmp_path / "b" / "same_src"
|
||||
root_b.mkdir(parents=True)
|
||||
values = (90, 130, 170, 210)
|
||||
_build_image_pdf(root_a / "same.pdf", values)
|
||||
_build_image_pdf(root_b / "same.pdf", values)
|
||||
llm_a, llm_b = _PageVisionLLM(), _PageVisionLLM()
|
||||
llm_a.settings = _pdf_settings(tmp_path, images=True)
|
||||
llm_b.settings = _pdf_settings(tmp_path, images=True)
|
||||
events: list[tuple[int, int]] = []
|
||||
try:
|
||||
# Run A (hook-less) first, snapshot its rows, clear the shared
|
||||
# source name, then run B (hooked) — the rows line up 1:1.
|
||||
summary_a = asyncio.run(import_sources([root_a], llm_a, session=db))
|
||||
rows_a = _p152_rows(db, "same_src")
|
||||
_cleanup_source(db, "same_src")
|
||||
summary_b = asyncio.run(
|
||||
import_sources(
|
||||
[root_b], llm_b, session=db,
|
||||
on_image_progress=lambda pos, total: events.append((pos, total)),
|
||||
)
|
||||
)
|
||||
rows_b = _p152_rows(db, "same_src")
|
||||
# The hook ran in run B (N image positions, the span's total)
|
||||
# — and changed nothing observable.
|
||||
assert events == [(i, len(values)) for i in range(1, len(values) + 1)]
|
||||
assert _p152_summary_fields(summary_a) == _p152_summary_fields(summary_b)
|
||||
assert rows_a == rows_b
|
||||
finally:
|
||||
_cleanup_source(db, "same_src")
|
||||
|
||||
|
||||
class _InterleavingVisionLLM(_PageVisionLLM):
|
||||
"""A ``_PageVisionLLM`` variant that records each vision call in
|
||||
a SHARED event sequence (phase 152, task 04): the importer fires
|
||||
the ``on_image_progress`` hook BEFORE the
|
||||
``_index_one_embedded_image`` describe seam, so the sequence reads
|
||||
hook, vision, hook, vision, … — the interleaving pin (the label
|
||||
moves at the START of each slow call, never after it)."""
|
||||
|
||||
def __init__(self, events: list[tuple]) -> None:
|
||||
super().__init__()
|
||||
self.events = events
|
||||
|
||||
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
|
||||
reply = await super().chat(messages, model)
|
||||
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||
if isinstance(user, list):
|
||||
self.events.append(("vision", self.vision_calls))
|
||||
return reply
|
||||
|
||||
|
||||
def test_on_image_progress_hook_fires_before_each_describe_call(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 152 (task 04): the hook fires BEFORE the corresponding
|
||||
describe call (the contract: at the START of each slow call, never
|
||||
after it) — the interleaving pinned by recording events in the
|
||||
fake: for a PDF with N embedded images the sequence reads
|
||||
hook(1,N), vision(1), hook(2,N), vision(2), …, hook(N,N),
|
||||
vision(N) (the fake's vision call IS the describe seam the hook
|
||||
precedes — the label moves while the describe is about to start,
|
||||
so a busy run never reads frozen)."""
|
||||
root = tmp_path / "imgprog-order"
|
||||
root.mkdir()
|
||||
values = (90, 130, 170)
|
||||
_build_image_pdf(root / "order.pdf", values)
|
||||
n = len(values)
|
||||
events: list[tuple] = []
|
||||
llm = _InterleavingVisionLLM(events)
|
||||
llm.settings = _pdf_settings(tmp_path, images=True)
|
||||
try:
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root], llm, session=db,
|
||||
on_image_progress=lambda pos, total: events.append(("image", pos, total)),
|
||||
)
|
||||
)
|
||||
assert (summary.files, summary.added) == (1, 1 + n)
|
||||
assert llm.vision_calls == n # one describe per image
|
||||
# The interleaving: hook BEFORE each describe call, in order.
|
||||
expected: list[tuple] = []
|
||||
for i in range(1, n + 1):
|
||||
expected.append(("image", i, n))
|
||||
expected.append(("vision", i))
|
||||
assert events == expected
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_on_image_progress_blank_gate_excludes_skipped_images_from_the_total(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 152 (task 04, the phase-151 interaction): images skipped
|
||||
by the dedupe/blank gate are NOT counted in the hook's total —
|
||||
the total is the work actually about to happen (the pass's
|
||||
post-dedupe, post-gate list). A PDF with 2 real + 1 near-white
|
||||
image (250 — the blank gate's class, mean ≥ 235): the gate
|
||||
(default ON) drops the near-white one BEFORE the pass — the hook
|
||||
fires for the 2 kept with total 2 (the skipped image burns no
|
||||
vision call, no position); with the kill switch OFF
|
||||
(``pdf_image_skip_blank=False`` — the pre-phase-151 behavior) the
|
||||
near-white image is indexed too — the hook fires for all 3 with
|
||||
total 3 (the total is the work about to happen under the knob's
|
||||
state)."""
|
||||
values_a = (90, 130, 250) # 250 = near-white (the blank gate's class)
|
||||
values_b = (110, 150, 250) # distinct bytes — a different document set
|
||||
# Both roots carry the SAME directory name — the source is part of
|
||||
# the stored image context, so the runs line up 1:1 (the house
|
||||
# cleanup idiom separates them).
|
||||
root_a = tmp_path / "a" / "same_src"
|
||||
root_a.mkdir(parents=True)
|
||||
root_b = tmp_path / "b" / "same_src"
|
||||
root_b.mkdir(parents=True)
|
||||
_build_image_pdf(root_a / "blank.pdf", values_a)
|
||||
_build_image_pdf(root_b / "blank.pdf", values_b)
|
||||
llm_a = _PageVisionLLM()
|
||||
llm_a.settings = _pdf_settings(tmp_path, images=True) # skip_blank default ON
|
||||
events_a: list[tuple[int, int]] = []
|
||||
try:
|
||||
summary_a = asyncio.run(
|
||||
import_sources(
|
||||
[root_a], llm_a, session=db,
|
||||
on_image_progress=lambda pos, total: events_a.append((pos, total)),
|
||||
)
|
||||
)
|
||||
# The blank gate dropped the near-white image BEFORE the
|
||||
# pass — the total is the 2 actually about to happen.
|
||||
assert summary_a.pdf_images_blank_skipped == 1
|
||||
assert (summary_a.files, summary_a.added) == (1, 3)
|
||||
assert llm_a.vision_calls == 2
|
||||
assert events_a == [(1, 2), (2, 2)]
|
||||
# The knob OFF: the near-white image is indexed too — the
|
||||
# total is all 3 (the pre-phase behavior, the work about to
|
||||
# happen under the knob's state). Fresh source (the house
|
||||
# cleanup idiom — run B's rows would otherwise be updates of
|
||||
# run A's rows, same synthetic image paths).
|
||||
_cleanup_source(db, "same_src")
|
||||
llm_b = _PageVisionLLM()
|
||||
llm_b.settings = Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
import_extensions="md,txt,pdf",
|
||||
images=True,
|
||||
image_dir=str(tmp_path / "images"),
|
||||
pdf_pictorial=False,
|
||||
pdf_image_skip_blank=False,
|
||||
)
|
||||
events_b: list[tuple[int, int]] = []
|
||||
summary_b = asyncio.run(
|
||||
import_sources(
|
||||
[root_b], llm_b, session=db,
|
||||
on_image_progress=lambda pos, total: events_b.append((pos, total)),
|
||||
)
|
||||
)
|
||||
assert summary_b.pdf_images_blank_skipped == 0
|
||||
assert (summary_b.files, summary_b.added) == (1, 4)
|
||||
assert llm_b.vision_calls == 3
|
||||
assert events_b == [(1, 3), (2, 3), (3, 3)]
|
||||
finally:
|
||||
_cleanup_source(db, "same_src")
|
||||
|
||||
+567
-15
@@ -110,6 +110,56 @@ poll tree's new 5th branch (after failed, before the upload branches
|
||||
— the 7-branch tree is now 8), the neutral ``applySyncCancelled``
|
||||
(no modal, no banner, no error styling), and the
|
||||
``initSyncButton`` re-attach of a ``cancelled`` run.
|
||||
|
||||
Phase 152 (task 01): the status contract gains the in-flight file's
|
||||
VISION-image position — ``images_done`` (the 1-based index of the
|
||||
image about to be described) + ``images_total`` (the file's total
|
||||
images for this run — the three PDF vision spans share ONE continuous
|
||||
counter per file, the importer's ``on_image_progress`` hook that
|
||||
task 02/03 wire). The idle dict pins them 0/0; the per-file reset is
|
||||
the pairing rule, extended — a ``progress`` call names the file AND
|
||||
clears the image position, pinned through the runner's own ``_hook``
|
||||
closure (the two-file fake below: the position fired through the
|
||||
runner's own ``_image_hook`` closure while file A is in flight is
|
||||
0/0 when file B is named — a file with no images must not inherit
|
||||
the previous PDF's position); the clear rule mirrors the chunk
|
||||
position — 0/0 at the overview/summaries transitions and in EVERY
|
||||
terminal (success, failed, and cancelled — each pinned with a live
|
||||
position in flight; task 04 moves every live position below onto the
|
||||
runner's OWN closures, the phase-64/129 idiom), while the FILE
|
||||
counts keep their final values (the convention split: a
|
||||
sub-position of a file that's done clears with the file).
|
||||
|
||||
Phase 152 (task 03): the runner's THIRD paired closure —
|
||||
``_image_hook`` (beside ``_hook`` and ``_chunk_hook``) — lands the
|
||||
importer's ``on_image_progress`` pairs on the status, and the import
|
||||
call now passes ``on_image_progress=_image_hook`` (the wiring pin:
|
||||
the fakes receive the kwarg and record the live closure). The
|
||||
frontend gains the "· image n/m" label PART: the running entry
|
||||
destructures ``image = ""`` and the writer call becomes
|
||||
``setSyncLabel(main, count, chunk, true, image)`` — the image part
|
||||
rides ONLY the composed full string (box + title + running
|
||||
aria-label — no new button span; the running button is the
|
||||
spinner), the chunk part keeps its own span, and the bare builders
|
||||
+ non-running calls pass nothing (both parts default "").
|
||||
|
||||
Phase 152 (task 04): the position pins move onto the runner's OWN
|
||||
closures (the phase-64/129 idiom — the fakes fire the image pairs
|
||||
through the ``on_image_progress`` kwarg, the ``_image_hook`` under
|
||||
test, instead of standing in on the status object): a mid-run pair
|
||||
(3/15, then 15/15) climbs on the status BETWEEN ticks while
|
||||
``current_file`` and the chunk position keep their own values (the
|
||||
THREE positions are independent sub-positions of the file), the
|
||||
per-file reset, the overview/summaries transition clears, and the
|
||||
FAILED + CANCELLED terminals all fire their live positions through
|
||||
the closure too. The frontend source pins the ``fmtSyncPhaseLabel``
|
||||
image part (present iff ``images_total > 0`` — the exact guard +
|
||||
template), the ``overview`` / ``summaries`` branches' ``image: ""``
|
||||
(unchanged labels), and ``setSyncLabel``'s composed ``full`` string
|
||||
byte-for-byte (a status carrying both positions reads "… (2540/2540)
|
||||
· chunk 125/126 · image 3/15" on the box + title + running
|
||||
aria-label — the runtime string is behaviorally pinned in
|
||||
tests/unit/test_frontend_sync_upload.py's node check).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -393,13 +443,16 @@ def test_sources_js_running_state_is_never_stale() -> None:
|
||||
Phase 128 (task 01): the entry hands the label PARTS (the
|
||||
phase-aware split — sync + status → fmtSyncPhaseLabel, the
|
||||
fall-through → the byte-identical fmtSyncLabel) to the SINGLE
|
||||
writer, setSyncLabel(main, count, chunk — phase 129: the chunk
|
||||
part destructures with a "" default), which lands the main part,
|
||||
writer, setSyncLabel(main, count, chunk, running, image — phase
|
||||
129: the chunk part destructures with a "" default, phase 152:
|
||||
the image part does too), which lands the main part,
|
||||
the pinned count part (hidden when absent), the pinned chunk part
|
||||
(phase 129 — hidden when absent), and the FULL untruncated label
|
||||
on the button title + #sync-progress (A4) — the direct span /
|
||||
title / announcer writes are GONE from the entry (they live in
|
||||
setSyncLabel + setSyncProgress). Phase 149 (task 01): the writer call passes
|
||||
on the button title + #sync-progress (A4 — phase 152: with the
|
||||
image part appended after the chunk part when a PDF's vision span
|
||||
is in flight; the image part gets no span of its own) — the
|
||||
direct span / title / announcer writes are GONE from the entry
|
||||
(they live in setSyncLabel + setSyncProgress). Phase 149 (task 01): the writer call passes
|
||||
`running = true` (the detail box reveals with the full label),
|
||||
and the button's accessible name rides in the writer too (the
|
||||
full live label — the icon-only running button keeps a name);
|
||||
@@ -414,8 +467,10 @@ def test_sources_js_running_state_is_never_stale() -> None:
|
||||
assert "fmtSyncLabel(kind, currentFile, done, total)" in body
|
||||
# …and the single writer lands the parts (A4 in it — the title is
|
||||
# set for EVERY running label: the file-only rule is adjusted
|
||||
# away) with the phase-149 `running` flag (the box reveals).
|
||||
assert "setSyncLabel(main, count, chunk, true)" in body
|
||||
# away) with the phase-149 `running` flag (the box reveals) +
|
||||
# the phase-152 image part (rides only the composed full string —
|
||||
# no span).
|
||||
assert "setSyncLabel(main, count, chunk, true, image)" in body
|
||||
assert body.count("setSyncLabel(") == 1, "the entry writes ONLY through the parts writer"
|
||||
assert "syncBtn.title =" not in body, ("the title write lives in setSyncLabel (A4)")
|
||||
assert "syncLabel.textContent" not in body, ("no direct span write — the parts")
|
||||
@@ -433,6 +488,70 @@ def test_sources_js_running_state_is_never_stale() -> None:
|
||||
)
|
||||
|
||||
|
||||
def test_fmt_sync_phase_label_image_part_is_pinned_source_level() -> None:
|
||||
"""Phase 152 (task 04): the frontend SOURCE pins the "· image
|
||||
n/m" label part (the browser behavior is pinned in
|
||||
tests/unit/test_frontend_sync_upload.py's node check — the REAL
|
||||
extracted functions run against stub refs; this pin keeps the
|
||||
template itself from drifting without a browser):
|
||||
|
||||
* the import fall-through is the ONLY reader of the status's
|
||||
image fields, guarded on ``images_total > 0`` (the part is
|
||||
present IFF ``images_total > 0`` — 0/0 hides it, the exact
|
||||
guard + template + empty false-branch);
|
||||
* the ``overview`` / ``summaries`` branches return ``image: ""``
|
||||
(the position belongs to the import phase only — those labels
|
||||
stay unchanged by the phase);
|
||||
* the running entry destructures the part with a `""` default and
|
||||
hands it to the single writer as the FIFTH argument (after the
|
||||
`running` flag);
|
||||
* ``setSyncLabel``'s composed ``full`` string appends the image
|
||||
part LAST (after the chunk part) — so a status carrying both
|
||||
positions reads exactly "Syncing… <file> (2540/2540) · chunk
|
||||
125/126 · image 3/15" on the box, the title, and the running
|
||||
aria-label (byte-for-byte; the docstring's worked example
|
||||
carries the shape).
|
||||
"""
|
||||
js = _text(SOURCES_JS)
|
||||
builder = _body(js, "fmtSyncPhaseLabel")
|
||||
# The import fall-through: present IFF images_total > 0 — the
|
||||
# exact guard + template (the phase-129 chunk part's sibling,
|
||||
# same shape).
|
||||
assert (
|
||||
"status.images_total > 0\n"
|
||||
" ? `· image ${status.images_done}/${status.images_total}`\n"
|
||||
" : \"\";"
|
||||
) in builder
|
||||
assert "return { main, count, chunk, image };" in builder
|
||||
# The overview branch: the unchanged label + the part absent.
|
||||
assert (
|
||||
'return { main: "Writing KB overview…", count: "", chunk: "", image: "" };'
|
||||
) in builder
|
||||
# The summaries branch: the unchanged label + the part absent
|
||||
# (the phase's own (n/m) is its position).
|
||||
assert "image: \"\", // the image position belongs to the import phase only" in builder
|
||||
assert "`Summarizing folders…${folder}`" in builder
|
||||
# The running entry: the part destructures with a `""` default
|
||||
# and rides the writer call as the fifth argument.
|
||||
entry = _body(js, "enterSyncRunningState")
|
||||
assert 'const { main, count, chunk = "", image = "" }' in entry
|
||||
assert "setSyncLabel(main, count, chunk, true, image)" in entry
|
||||
# The writer: the composed full string — the image part LAST,
|
||||
# after the chunk part (the "… · chunk 125/126 · image 3/15"
|
||||
# shape the docstring's worked example carries).
|
||||
writer = _body(js, "setSyncLabel")
|
||||
assert (
|
||||
"const full =\n"
|
||||
" mainText +\n"
|
||||
" (countText ? ` ${countText}` : \"\") +\n"
|
||||
" (chunkText ? ` ${chunkText}` : \"\") +\n"
|
||||
" (imageText ? ` ${imageText}` : \"\");"
|
||||
) in writer
|
||||
assert "· chunk 125/126 · image 3/15" in js, (
|
||||
"the worked example (both positions at once) is documented"
|
||||
)
|
||||
|
||||
|
||||
def test_sources_js_terminal_states() -> None:
|
||||
"""Terminal rendering: success → enabled + 'Synced HH:MM' (local
|
||||
time of finished_at); failed → enabled + retry-ready 'Sync sources'
|
||||
@@ -927,7 +1046,20 @@ class _GatedImport:
|
||||
``_chunk_hook`` closure: pair one lands before the main park
|
||||
(readable mid-run), and every later pair lands after the main
|
||||
park + its own park (``chunk_release``), so each position is
|
||||
readable on the status between ticks."""
|
||||
readable on the status between ticks.
|
||||
|
||||
Phase 152 (task 03): the runner now ALWAYS passes the
|
||||
image-position closure (task 03's wiring) — recorded for the
|
||||
wiring pins.
|
||||
|
||||
Phase 152 (task 04): optionally ALSO fires the in-flight file's
|
||||
image-position pairs (``image_pairs``) through the runner's OWN
|
||||
``_image_hook`` closure (pair one before the main park —
|
||||
readable mid-run; every later pair after the main park + its own
|
||||
park ``image_release`` — the phase-129 chunk idiom, mirrored for
|
||||
the vision-image position). With no pairs the fake never fires
|
||||
it (the status image fields stay 0/0, like the chunk fields of a
|
||||
no-batch import)."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -937,6 +1069,8 @@ class _GatedImport:
|
||||
fail: BaseException | None = None,
|
||||
chunk_pairs: Sequence[tuple[int, int]] = (),
|
||||
chunk_release: threading.Event | None = None,
|
||||
image_pairs: Sequence[tuple[int, int]] = (), # phase 152 (task 04)
|
||||
image_release: threading.Event | None = None, # phase 152 (task 04)
|
||||
) -> None:
|
||||
self.summary = summary
|
||||
self.started = started
|
||||
@@ -947,6 +1081,12 @@ class _GatedImport:
|
||||
self.chunk_pairs = list(chunk_pairs)
|
||||
self.chunk_release = chunk_release
|
||||
self.chunk_hook_calls: list[tuple[int, int]] = []
|
||||
# Phase 152 (task 03): the image-position hook the runner
|
||||
# passes (recorded for the wiring pins).
|
||||
self.image_hook: Callable[[int, int], None] | None = None
|
||||
self.image_pairs = list(image_pairs)
|
||||
self.image_release = image_release
|
||||
self.image_hook_calls: list[tuple[int, int]] = []
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -958,6 +1098,7 @@ class _GatedImport:
|
||||
session: object = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
@@ -966,6 +1107,16 @@ class _GatedImport:
|
||||
if progress is not None:
|
||||
progress("repo", "notes/deep.md", 1, 3)
|
||||
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
|
||||
# Phase 152 (task 03): the runner now ALWAYS passes the
|
||||
# image-position closure (task 03's wiring) — recorded for the
|
||||
# wiring pins.
|
||||
self.image_hook = on_image_progress
|
||||
# Phase 152 (task 04): image pair one lands BEFORE the main
|
||||
# park (readable mid-run — the phase-129 chunk-pair-one
|
||||
# idiom, mirrored for the vision-image position).
|
||||
if on_image_progress is not None and self.image_pairs:
|
||||
on_image_progress(*self.image_pairs[0])
|
||||
self.image_hook_calls.append(self.image_pairs[0])
|
||||
if on_chunk_progress is not None and self.chunk_pairs:
|
||||
# Phase 129 (task 03): the file's chunk-position pairs,
|
||||
# fired through the runner's OWN hook closure (the closure
|
||||
@@ -986,6 +1137,17 @@ class _GatedImport:
|
||||
else:
|
||||
self.started.set()
|
||||
await asyncio.to_thread(self.release.wait) # park without freezing the worker loop
|
||||
# Phase 152 (task 04): every later image pair lands after the
|
||||
# main park + its own park (``image_release``), so each
|
||||
# position is readable on the status between ticks. A wiring
|
||||
# regression to None skips this branch and the pair tests
|
||||
# time out.
|
||||
if on_image_progress is not None:
|
||||
for pair in self.image_pairs[1:]:
|
||||
on_image_progress(*pair)
|
||||
self.image_hook_calls.append(pair)
|
||||
if self.image_release is not None:
|
||||
await asyncio.to_thread(self.image_release.wait)
|
||||
if self.fail is not None:
|
||||
raise self.fail
|
||||
return self.summary
|
||||
@@ -1088,10 +1250,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
|
||||
fresh_sync_status: None,
|
||||
) -> None:
|
||||
"""Idle: the three phase-64 progress keys ride along as null/0/0,
|
||||
the two phase-129 chunk keys as 0/0, the four phase-98 phase keys
|
||||
as null/0/0/0, and EVERY pre-existing key is unchanged — the full
|
||||
response dict is pinned, so the current UI and every existing
|
||||
consumer keep working."""
|
||||
the two phase-129 chunk keys as 0/0, the two phase-152 image keys
|
||||
as 0/0, the four phase-98 phase keys as null/0/0/0, and EVERY
|
||||
pre-existing key is unchanged — the full response dict is pinned,
|
||||
so the current UI and every existing consumer keep working."""
|
||||
assert sync_api.sync_status() == {
|
||||
"state": "idle",
|
||||
"started_at": None,
|
||||
@@ -1105,6 +1267,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
|
||||
# (the dataclass defaults, no special-casing).
|
||||
"current_chunk": 0,
|
||||
"chunks_total": 0,
|
||||
# Phase 152 (task 01): the image-position keys — 0/0 idle
|
||||
# (the dataclass defaults, no special-casing).
|
||||
"images_done": 0,
|
||||
"images_total": 0,
|
||||
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
|
||||
"phase": None,
|
||||
"current_summary": None,
|
||||
@@ -1145,13 +1311,19 @@ def test_mid_run_status_reports_current_file(
|
||||
assert s["files_done"] == 0 and s["files_total"] == 0
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
# Phase 152 (task 01): the image position is 0/0 in the
|
||||
# prelude too (no vision span yet).
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
clone.release.set()
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
# Import phase: the hook's file is live on the status, and the
|
||||
# phase machine names the phase (the summary span has not
|
||||
# started — its counters are still 0/0); no embed batch has
|
||||
# fired the chunk hook (this canned import passes no pairs),
|
||||
# so the chunk position stays 0/0.
|
||||
# so the chunk position stays 0/0; the image position stays
|
||||
# 0/0 too (this canned import fires no image hook — task 03
|
||||
# wires the runner's image closure, the importer's span is
|
||||
# what would feed it).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "import"
|
||||
@@ -1161,6 +1333,7 @@ def test_mid_run_status_reports_current_file(
|
||||
assert s["current_summary"] is None
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
fake_import.release.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
@@ -1178,7 +1351,8 @@ def test_mid_run_status_reports_current_file(
|
||||
# folder step never fired the summary hook, so the summary counts
|
||||
# stay 0/0 (the keep-final-counts convention keeps what fired);
|
||||
# the chunk position clears with the file (phase 129 — it is a
|
||||
# sub-position of a done file, never a keep-final counter).
|
||||
# sub-position of a done file, never a keep-final counter), and
|
||||
# so does the image position (phase 152 — same convention).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "success"
|
||||
assert s["phase"] is None
|
||||
@@ -1187,6 +1361,7 @@ def test_mid_run_status_reports_current_file(
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
assert s["error"] is None
|
||||
assert s["started_at"] is not None and s["finished_at"] is not None
|
||||
|
||||
@@ -1215,6 +1390,7 @@ def test_failed_terminal_clears_current_file_keeps_counts(
|
||||
"failed: connection refused"
|
||||
),
|
||||
chunk_pairs=[(1, 38)], # phase 129: the live pair before the raise
|
||||
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair before the raise
|
||||
)
|
||||
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
|
||||
|
||||
@@ -1231,6 +1407,13 @@ def test_failed_terminal_clears_current_file_keeps_counts(
|
||||
assert s["current_file"] == "repo/notes/deep.md"
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
assert s["current_chunk"] == 1 and s["chunks_total"] == 38
|
||||
# Phase 152 (task 04): a live image position in flight when
|
||||
# the run dies — fired through the runner's OWN _image_hook
|
||||
# closure (image_pairs[0] — the importer's span would do the
|
||||
# same); the failed terminal must clear it (the
|
||||
# sub-position of an abandoned file, like the chunk
|
||||
# position).
|
||||
assert s["images_done"] == 3 and s["images_total"] == 15
|
||||
fake_import.release.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
@@ -1238,8 +1421,9 @@ def test_failed_terminal_clears_current_file_keeps_counts(
|
||||
thread.join(10.0)
|
||||
assert not thread.is_alive()
|
||||
assert errors == []
|
||||
# Wiring: the pair fired through the runner's own closure.
|
||||
# Wiring: the pairs fired through the runner's OWN closure.
|
||||
assert fake_import.chunk_hook_calls == [(1, 38)]
|
||||
assert fake_import.image_hook_calls == [(3, 15)]
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "failed"
|
||||
assert s["current_file"] is None # cleared in the terminal state
|
||||
@@ -1251,6 +1435,11 @@ def test_failed_terminal_clears_current_file_keeps_counts(
|
||||
# terminal too — the convention split, pinned (the FILE counts
|
||||
# above keep their final values, the sub-position does not).
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
# Phase 152 (task 04): the image position clears in the FAILED
|
||||
# terminal too — the convention split, pinned (the live 3/15 fired
|
||||
# through the closure mid-run is gone, the FILE counts keep their
|
||||
# final values).
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
assert s["detail"] == {}
|
||||
error = s["error"] or ""
|
||||
assert "*****@aipi.example.com" in error # credentials masked
|
||||
@@ -1468,6 +1657,357 @@ def test_chunk_position_clears_at_phase_transitions_and_terminal(
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
|
||||
|
||||
# ---------- phase 152 (task 01 + task 04): the in-flight file's -----
|
||||
# ---------- vision-image position (images_done / images_total) -------
|
||||
#
|
||||
# Same seams as the phase-129 block above. Task 04 pins every live
|
||||
# position through the runner's OWN closures (the phase-64/129 idiom):
|
||||
# the fakes fire the image pairs through the ``on_image_progress``
|
||||
# kwarg — the ``_image_hook`` under test (task 03's wiring: the
|
||||
# importer's span's exact path) — and drive the per-file ``_hook``
|
||||
# reset (through the ``progress`` kwarg the fake import receives) and
|
||||
# the transition/terminal clears.
|
||||
|
||||
|
||||
def test_mid_run_status_reports_image_position_between_ticks(
|
||||
fresh_sync_status: None,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Phase 152 (task 04): the status carries the in-flight file's
|
||||
VISION-image position — the pairs fire through the runner's OWN
|
||||
``_image_hook`` closure (task 03's wiring — the importer's
|
||||
``on_image_progress`` span's exact path), and the position CLIMBS
|
||||
between ticks (3/15 → 15/15 — the PDF's vision span moving, the
|
||||
owner's stuck label, now moving) while ``current_file`` and the
|
||||
chunk position (14/38, a concurrent embed batch) keep their own
|
||||
values — the THREE positions are independent sub-positions of the
|
||||
file; the prelude reports 0/0, and the success terminal clears
|
||||
BOTH sub-positions (image + chunk) to 0/0 while the FILE counts
|
||||
keep their final values (the convention split)."""
|
||||
clone = _GatedClone(threading.Event(), threading.Event())
|
||||
image_release = threading.Event()
|
||||
fake_import = _GatedImport(
|
||||
ImportSummary(files=3, added=1, updated=1, unchanged=1),
|
||||
threading.Event(),
|
||||
threading.Event(),
|
||||
chunk_pairs=[(14, 38)], # a concurrent embed batch (independent)
|
||||
image_pairs=[(3, 15), (15, 15)], # two describe calls of a 15-image span
|
||||
image_release=image_release,
|
||||
)
|
||||
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
|
||||
|
||||
thread, errors = _start_run()
|
||||
try:
|
||||
assert clone.started.wait(5.0), "the run never reached the clone phase"
|
||||
# Prelude: every position 0/0 (no file, no batch, no image
|
||||
# yet — alongside the bare-label pins of the phase-64 test).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["current_file"] is None
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
clone.release.set()
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
# Import phase: the file is live (the progress hook's call),
|
||||
# the chunk position 14/38 (the embed batch), and the image
|
||||
# position 3/15 (the vision span) — the THREE positions are
|
||||
# independent sub-positions of the same file.
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "import"
|
||||
assert s["current_file"] == "repo/notes/deep.md"
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
assert s["current_chunk"] == 14 and s["chunks_total"] == 38
|
||||
assert s["images_done"] == 3 and s["images_total"] == 15
|
||||
# Release the main park: the import fires image pair two
|
||||
# (15/15 — the span's last describe call) and parks on
|
||||
# image_release.
|
||||
fake_import.release.set()
|
||||
_poll_until(
|
||||
lambda: sync_api.sync_status()["images_done"] == 15,
|
||||
what="the second image pair on the status",
|
||||
)
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "import"
|
||||
# The image position CLIMBED (3 → 15 — monotonic, no
|
||||
# backwards jump, never above the total) ... the FILE and
|
||||
# chunk positions keep their own values (independent).
|
||||
assert s["images_done"] == 15 and s["images_total"] == 15
|
||||
assert s["current_file"] == "repo/notes/deep.md"
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
assert s["current_chunk"] == 14 and s["chunks_total"] == 38
|
||||
image_release.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
fake_import.release.set()
|
||||
image_release.set()
|
||||
thread.join(10.0)
|
||||
assert not thread.is_alive()
|
||||
assert errors == [], f"the runner raised: {errors!r}"
|
||||
# Wiring: every pair fired through the runner's OWN closure (the
|
||||
# status reads above came through it; a wiring regression to
|
||||
# None would have skipped the firing branch and timed out).
|
||||
assert fake_import.chunk_hook_calls == [(14, 38)]
|
||||
assert fake_import.image_hook_calls == [(3, 15), (15, 15)]
|
||||
# Success terminal: BOTH sub-positions clear (0/0) ...
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "success"
|
||||
assert s["phase"] is None
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
# ... while the FILE counts keep their final values (the
|
||||
# convention split, pinned alongside the phase-64 terminal pin).
|
||||
assert s["current_file"] is None
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
|
||||
|
||||
class _TwoFileImport:
|
||||
"""The phase-152 (task 01 + task 04) fake: fires the runner's
|
||||
OWN progress hook for file A, then file A's image position through
|
||||
the runner's OWN ``on_image_progress`` closure (task 03's wiring —
|
||||
what the importer's span would do), parks (the test reads the
|
||||
live position on the status), fires the progress hook again for
|
||||
file B (the runner's ``_hook`` under test must reset the image
|
||||
position to 0/0 — the pairing rule, extended), parks again, and
|
||||
completes."""
|
||||
|
||||
def __init__(self, summary: ImportSummary) -> None:
|
||||
self.summary = summary
|
||||
self.parked_a = threading.Event() # parked after file A's progress
|
||||
self.release_a = threading.Event() # test releases → file B's progress
|
||||
self.parked_b = threading.Event() # parked after file B's progress
|
||||
self.release_b = threading.Event()
|
||||
self.hook_calls: list[tuple[str, str, int, int]] = []
|
||||
self.image_hook_calls: list[tuple[int, int]] = [] # phase 152 (task 04)
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
sources: list[Path],
|
||||
llm: object,
|
||||
*,
|
||||
prune: bool = False,
|
||||
limit: int | None = None,
|
||||
session: object = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
) -> ImportSummary:
|
||||
if progress is not None:
|
||||
progress("repo", "docs/a.pdf", 1, 2)
|
||||
self.hook_calls.append(("repo", "docs/a.pdf", 1, 2))
|
||||
# Phase 152 (task 04): file A's vision span is in flight —
|
||||
# its image position (3/15) fires through the runner's OWN
|
||||
# on_image_progress closure (task 03's wiring), then the fake
|
||||
# parks.
|
||||
if on_image_progress is not None:
|
||||
on_image_progress(3, 15)
|
||||
self.image_hook_calls.append((3, 15))
|
||||
self.parked_a.set()
|
||||
await asyncio.to_thread(self.release_a.wait)
|
||||
if progress is not None:
|
||||
# The SECOND progress call names file B — the closure under
|
||||
# test must reset the image position the closure fired
|
||||
# while file A was in flight (a file with no images must
|
||||
# not inherit the previous PDF's position).
|
||||
progress("repo", "docs/b.md", 2, 2)
|
||||
self.hook_calls.append(("repo", "docs/b.md", 2, 2))
|
||||
self.parked_b.set()
|
||||
await asyncio.to_thread(self.release_b.wait)
|
||||
return self.summary
|
||||
|
||||
|
||||
def test_image_position_resets_per_file(
|
||||
fresh_sync_status: None,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Phase 152 (task 01 + task 04): the pairing rule, extended — a
|
||||
``progress`` call names the file AND clears the image position;
|
||||
every image position until the next ``progress`` call belongs to
|
||||
it. File A's live position (3/15, fired through the runner's OWN
|
||||
``_image_hook`` closure while parked on file A — task 04: the
|
||||
phase-64/129 idiom, the importer's image hook's exact path) is
|
||||
0/0 the moment the runner's own ``_hook`` names file B: a file
|
||||
with no images must not inherit the previous PDF's position
|
||||
(unlike the chunk position, not every file embeds chunks). The
|
||||
success terminal keeps the position at 0/0 and the FILE counts
|
||||
keep their final values (the convention split)."""
|
||||
clone = _GatedClone(threading.Event(), threading.Event())
|
||||
fake_import = _TwoFileImport(ImportSummary(files=2, added=1, updated=1))
|
||||
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
|
||||
|
||||
thread, errors = _start_run()
|
||||
try:
|
||||
assert clone.started.wait(5.0), "the run never reached the clone phase"
|
||||
clone.release.set()
|
||||
assert fake_import.parked_a.wait(5.0), "the run never parked on file A"
|
||||
# File A is named by the runner's own _hook (a wiring
|
||||
# regression to None would leave current_file null here).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "import"
|
||||
assert s["current_file"] == "repo/docs/a.pdf"
|
||||
assert s["files_done"] == 1 and s["files_total"] == 2
|
||||
# File A's vision span is in flight: its image position is
|
||||
# live on the status — fired through the runner's OWN
|
||||
# _image_hook closure (the fake fired it before parking;
|
||||
# a wiring regression to None would leave it 0/0 here).
|
||||
assert s["images_done"] == 3 and s["images_total"] == 15
|
||||
# Release file A: the runner's _hook names file B — the image
|
||||
# position must reset to 0/0 in the same call (the pairing
|
||||
# rule, extended; the closure under test, driven through the
|
||||
# fake's progress kwarg).
|
||||
fake_import.release_a.set()
|
||||
assert fake_import.parked_b.wait(5.0), "the run never parked on file B"
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "import"
|
||||
assert s["current_file"] == "repo/docs/b.md"
|
||||
assert s["files_done"] == 2 and s["files_total"] == 2
|
||||
# RESET per file: file B (no images) does not inherit file
|
||||
# A's 3/15.
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
fake_import.release_b.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
fake_import.release_a.set()
|
||||
fake_import.release_b.set()
|
||||
thread.join(10.0)
|
||||
assert not thread.is_alive()
|
||||
assert errors == [], f"the runner raised: {errors!r}"
|
||||
# Wiring: BOTH progress calls went through the runner's own
|
||||
# closure (the reset above was its doing), and the image pair
|
||||
# went through the runner's own _image_hook closure.
|
||||
assert fake_import.hook_calls == [
|
||||
("repo", "docs/a.pdf", 1, 2),
|
||||
("repo", "docs/b.md", 2, 2),
|
||||
]
|
||||
assert fake_import.image_hook_calls == [(3, 15)]
|
||||
# Success terminal: the image position stays 0/0 (cleared in the
|
||||
# terminal too — every terminal clears it) and the FILE counts
|
||||
# keep their final values (the convention split).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "success"
|
||||
assert s["phase"] is None
|
||||
assert s["current_file"] is None
|
||||
assert s["files_done"] == 2 and s["files_total"] == 2
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
|
||||
|
||||
def test_image_position_clears_at_phase_transitions_and_terminal(
|
||||
fresh_sync_status: None,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Phase 152 (task 01 + task 04): the image position is CLEARED
|
||||
(0/0) at every point the import phase ends. This run's summary
|
||||
changes the KB, so BOTH post-import transitions run and are
|
||||
pinned: the import→``"overview"`` transition and the
|
||||
import→``"summaries"`` transition (the changed-KB branch) each
|
||||
clear the live position (3/15, fired mid-import through the
|
||||
runner's OWN ``_image_hook`` closure — task 04: the
|
||||
phase-64/129 idiom, the importer's span's exact path) to 0/0,
|
||||
and the success terminal keeps it 0/0 — through the whole span
|
||||
the FILE counts keep their final values (the convention split:
|
||||
the file counts survive for the result line's context, the image
|
||||
position is a sub-position of a file that's done)."""
|
||||
clone = _GatedClone(threading.Event(), threading.Event())
|
||||
import_release = threading.Event()
|
||||
overview_started = threading.Event()
|
||||
overview_release = threading.Event()
|
||||
folders_started = threading.Event()
|
||||
folders_release = threading.Event()
|
||||
fake_import = _GatedImport(
|
||||
ImportSummary(files=3, added=1, updated=1, unchanged=1),
|
||||
threading.Event(),
|
||||
import_release,
|
||||
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair to clear
|
||||
)
|
||||
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
|
||||
|
||||
# Park the post-import steps (re-monkeypatched over the seams
|
||||
# helper's no-ops): the test reads the status DURING the
|
||||
# "overview" phase and DURING the "summaries" phase — the two
|
||||
# import-phase-ending transitions the task pins (the same shape
|
||||
# as the phase-129 chunk-position transition test above).
|
||||
async def gated_overview(llm: object, session: object = None) -> bool:
|
||||
overview_started.set()
|
||||
await asyncio.to_thread(overview_release.wait)
|
||||
return True
|
||||
|
||||
async def gated_folders(
|
||||
db: object,
|
||||
llm: object,
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
folders_started.set()
|
||||
await asyncio.to_thread(folders_release.wait)
|
||||
return {"generated": 0, "failed": 0, "pruned": 0}
|
||||
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", gated_overview)
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
|
||||
|
||||
thread, errors = _start_run()
|
||||
try:
|
||||
assert clone.started.wait(5.0), "the run never reached the clone phase"
|
||||
clone.release.set()
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
# Mid-import: the file is live; its image position went live
|
||||
# too — fired through the runner's OWN _image_hook closure
|
||||
# (image_pairs[0] — the importer's span would do the same).
|
||||
s = sync_api.sync_status()
|
||||
assert s["phase"] == "import"
|
||||
assert s["images_done"] == 3 and s["images_total"] == 15
|
||||
# Release the import: it returns (added+updated > 0) → the
|
||||
# phase machine moves to "overview" and the position clears.
|
||||
import_release.set()
|
||||
assert overview_started.wait(5.0), "the run never reached the overview"
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "overview"
|
||||
# CLEARED at the import→overview transition (0/0) ... the file
|
||||
# counts KEEP their final values (the convention split).
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
overview_release.set()
|
||||
# ... and the changed-KB branch moves to "summaries" (the
|
||||
# second import-phase-ending transition — cleared there too,
|
||||
# idempotently) before parking.
|
||||
assert folders_started.wait(5.0), "the run never reached the summaries"
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "summaries"
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
folders_release.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
import_release.set()
|
||||
overview_release.set()
|
||||
folders_release.set()
|
||||
thread.join(10.0)
|
||||
assert not thread.is_alive()
|
||||
assert errors == [], f"the runner raised: {errors!r}"
|
||||
assert fake_import.image_hook_calls == [(3, 15)]
|
||||
# Success terminal: still 0/0 (the terminal clears too — same
|
||||
# spots as current_file), file counts kept.
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "success"
|
||||
assert s["phase"] is None
|
||||
assert s["current_file"] is None
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
|
||||
|
||||
# ---------- phase 134 (task 01): the cooperative cancel (5th state) ----------
|
||||
#
|
||||
# The cancel runs are driven through the REAL ``start_sync`` on a
|
||||
@@ -1629,11 +2169,17 @@ def test_cancel_endpoint_stops_the_in_flight_run(
|
||||
ImportSummary(files=3, added=1, updated=1, unchanged=1),
|
||||
threading.Event(),
|
||||
threading.Event(),
|
||||
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair before the cancel
|
||||
)
|
||||
_patch_sync_seams(monkeypatch, tmp_path, fake_import, _fast_clone)
|
||||
|
||||
def act(loop: asyncio.AbstractEventLoop) -> dict[str, str]:
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
# Phase 152 (task 04): the live image position in flight when
|
||||
# the cancel converts was fired through the runner's OWN
|
||||
# _image_hook closure (image_pairs[0] — the fake fired it
|
||||
# before parking, like the importer's span would); the
|
||||
# cancelled terminal must clear it.
|
||||
return loop.run_until_complete(sync_api.cancel_sync())
|
||||
|
||||
try:
|
||||
@@ -1665,6 +2211,11 @@ def test_cancel_endpoint_stops_the_in_flight_run(
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3 # final counts kept
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
||||
# Phase 152 (task 04): the image position clears in the CANCELLED
|
||||
# terminal too — every terminal clears it (the live 3/15 fired
|
||||
# through the closure in flight is gone; the FILE counts above
|
||||
# keep their final values).
|
||||
assert s["images_done"] == 0 and s["images_total"] == 0
|
||||
# PLAN §9: the ample-logging line for the converted cancel.
|
||||
assert any(
|
||||
r.getMessage()
|
||||
@@ -1879,6 +2430,7 @@ def test_event_set_during_import_cancels_before_the_lite_spans(
|
||||
session: object = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
||||
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
|
||||
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
||||
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
||||
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
||||
|
||||
Reference in New Issue
Block a user