phase: 152_sync_progress_image_pass
Build and Push Containers / build-and-push-app (push) Successful in 3m34s
Build and Push Containers / build-and-push-db (push) Successful in 23s

All criteria verified. The CLI path is hook-less (only `app/api/sync.py` passes the hook), and the byte-identity is pinned by `test_on_image_progress_none_run_is_byte_identical`.

## Final Report — Phase 152 final verification pass

- **Verified (no new code needed):** `SyncStatus.images_done/images_total` + reset/clear rules in `app/api/sync.py`; `on_image_progress` threaded through the importer's three PDF vision spans with the continuous per-file counter; the `_image_hook` closure; `sources.js` `· image n/m` label part (box + title + aria).
- **C1** status climbs 1→N + UI shows `· image n/m`: ✅ unit 3/15→15/15, integration 1/15→7/15, E2E 1/4→4/4 (title/aria ride the full string)
- **C2** never frozen: ✅ E2E asserts ≥3 distinct positions, starting at 1, non-decreasing, ≤ total, reaching it
- **C3** cleared at overview/summaries + all terminals; per-file reset via `_hook`; no stale position for image-less files: ✅ unit pins + E2E settle (image part gone in box/title/aria, endpoint 0/0)
- **C4** `/api/sync/status` additive-only: ✅ full-dict pin passes (all pre-existing keys unchanged + the two new 0/0 keys)
- **C5** hook-less caller byte-identical: ✅ CLI passes no hook; `test_on_image_progress_none_run_is_byte_identical` green
- **C6** `uv run pytest`: **3526 passed, exit 0** (95s); dedicated E2E in isolation: **2 passed** (95s); `uv run ruff check . && uv run pyright`: **clean**; coverage: **99% on `app/` (>90% gate)**
- Regression: all five neighbouring E2E suites green in isolation (progress_box 8, chunk_progress 2, label_mobile 4, pdf_support 6, image_documents 4)
- **C7** atomic commit: left to the harness (no git index/history touched per rules)
- **Docs:** README.md — sync-label paragraph already updated this phase, accurately; all `.env.example` vars documented; routes/commands verified — **no changes needed**. AGENTS.md — all references (PLAN, user_stories, test_smoke.py, .env.example, gpg config, .gitignore) verified — **no changes needed**
- **Defects found in prior work:** none (one prior env-leak defect was already fixed inside this phase's task 04, conftest)
- No live-infrastructure touched; no ntfy used. **Next pending phase:** none — `todo/` held only this phase
This commit is contained in:
2026-10-03 01:52:41 -04:00
parent c7cfcd26b5
commit 12fa69b7ea
39 changed files with 3971 additions and 133 deletions
+6
View File
@@ -48,6 +48,12 @@ from app.config import Settings as _Settings # noqa: E402
os.environ["BOR_DOCS_REPO"] = ""
os.environ["BOR_SUGGESTIONS"] = json.dumps(_Settings.model_fields["suggestions"].default)
# Phase 152 (task 04, defect fix): the same leak class for the display
# name — an operator's local ``.env`` may legitimately carry
# ``BOR_APP_NAME`` (the dev environment runs a renamed build), and the
# default-metadata pins (``/api/config`` ``app_name``, the page titles
# + header brand) must see the code default.
os.environ["BOR_APP_NAME"] = _Settings.model_fields["app_name"].default
# Phase 62: the same leak class for the new UI customization settings —
# an operator's local ``.env`` may legitimately carry
+885
View File
@@ -0,0 +1,885 @@
"""Phase 152 task 05 E2E (Playwright, mock-only): during a PDF sync whose
images take time to describe, the live label shows the image position
MOVING — the owner's "stuck for several minutes" (TODO.md:3: the label
sat on ``Syncing… … (2540/2540) · chunk 125/126`` through the PDF's
vision spans) is impossible on the fixture.
The dedicated story suite for ``152_sync_progress_image_pass`` (A16 —
one Playwright file per phase, run in isolation): the ``on_image_progress
hook (phase 152, tasks 02/03) fires immediately before each vision
describe call with ``(position, total)``, the runner's ``_image_hook``
closure lands it on ``SyncStatus.images_done/images_total``, and
``fmtSyncPhaseLabel`` renders ``· image n/m`` — the phase-149 box
(``#sync-progress``) is the surface that shows it, the button
``title``/``aria-label`` carry the UNTRUNCATED composed string (A4).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_sync_image_progress.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic mock plus the phase-64 ``tests/e2e/slow_llm.py`` delay
proxy in front of it (``SLOW_DELAY_S = 1.5 s``) that stretches EVERY
LLM call (vision describes included) so the fixture's vision span
outlives the UI's 2 s status poll and each position outlives the
recorder's ~300 ms page-sampling cadence by ~10× — the observed
positions are dense samples of a real climb, not races.
Timing fixture (the phase-129/98 rationale, re-sized for vision spans):
the fixture's ONE-PDF run is ≈ 27 LLM calls ≈ 40 s: 2 model probes +
10 in-flight embed batches (the ~4 KB-per-page text packs into 20
content chunks at the app's code-default, pinned sizing — the token
budget groups them into 10 requests of 2) + 4 image pipelines (one
vision describe + 2 content-embed batches each — the image doc's
context + description content packs into 3 chunks) + 1 PDF summary +
1 overview + 1 folder summary. The image span itself is 4 × (1.5 s
describe + 2 × 1.5 s embed) = 18 s: each of the 4 positions holds
for 4.5 s — that outlives the UI's 2 s poll (at least one tick lands
in every 4.5 s window — every position is rendered, not a race) and
the test's ~300 ms page sampling sees every rendered value.
KB fixture (the ``test_local_directory_sources.py`` / phase-94 API
registration pattern — a host temp dir, the app runs on the same host;
no git anywhere): ``imgpass/`` with EXACTLY ONE file, a PyMuPDF-built
``imgpass.pdf`` (LOCKED A1) — 10 text pages (each ~4 KB of
deterministic filler, well over the 100-char bar — the phase-140
pictorial gate never fires, and the layer is clean — the phase-133
fallback never fires: the EMBEDDED-IMAGE pass is the file's one and
only vision span) + 4 GENUINELY DISTINCT non-blank images on pages
1–4 (different sizes + seeds → different stored bytes — the phase-151
dedupe keeps all four; colorful noise, sample mean ~127 — well under
the 235 near-white floor, so the blank gate keeps all four: the
span's total is the post-dedupe/post-gate count, the work actually
about to happen). The expected chunk count is computed IN THE TEST by
calling ``extract_pdf_text`` (the SAME parser the importer uses) and
``chunk_document`` (the SAME function + arguments — the importer's
``max(400, target)`` floor + the overlap setting — as
``_replace_content_chunks``), read back from the PDF on disk — the
ground truth never drifts from the app under test. The sizing knobs
are pinned to the code defaults in the app-under-test env (the
leak-guard pattern — an operator's gitignored ``.env`` tuning
``BOR_CHUNK_TARGET_CHARS`` must not change the suite's ground truth).
The suite runs ONE module app (it binds its own port — the conftest
session app is never started in an isolated run) over the shared E2E
Postgres: the SLOW app (``BOR_LLM_BASE_URL`` = the slow proxy in front
of the mock), ``BOR_IMAGES=true`` + the ``pdf`` import token pinned
EXPLICITLY (an operator's local gitignored ``.env`` cannot leak the
scope in either direction) + ``BOR_PDF_PICTORIAL`` pinned off (the
house E2E convention — this suite's span is the embedded pass; the
fixture's ≥100-char pages make the pictorial pass a no-op even if the
knob leaked on).
Autouse cleanup (the phase-96 pattern): before each test wait for no
running sync on the module app (the run state lives in its memory),
then truncate the shared registry + KB tables (documents/chunks/
query_log/steering_notes/kb_overview/git_sources/sources_meta/
folder_summaries) — each test seeds its own source row and runs its
own sync; after each test the same truncate.
Test → observable mapping (Playwright Mapping Rule):
1. ``test_label_advances_through_the_image_span`` (the anti-freeze
proof — the UI's own 2 s poll, densely sampled at ~300 ms): after
clicking **Sync sources**, the test collects the DISTINCT
``#sync-progress-text`` values seen while the run is in flight
(first-seen order) — and asserts: (a) at least THREE distinct
labels containing ``· image `` are observed (the freeze the owner
reported is a single repeated label — "≥3 distinct positions" IS
the bug's negation), the positions START at 1, are
NON-DECREASING in observation order, never EXCEED the total (4 —
the post-gate count), and REACH it; (b) the CHUNK part appeared
EARLIER in the run (the first ``· chunk n/20`` label precedes the
first ``· image `` label — the two positions are distinct) and
every image-span label carries the chunk part WITH IT (the last
chunk position persists through the file's image span —
"… (1/1) · chunk 19/20 · image 3/4"), its total == the file's
TRUE chunk count; (c) the button ``title`` + ``aria-label`` carry
the SAME full string as the box on every running sample (A4 —
asserted per sample, atomically); (d) at settle the box shows the
counts line (``5 added`` — the PDF + its 4 image docs), the label
reads the terminal ``Synced HH:MM``, the count + chunk parts are
hidden + empty, and the image part is GONE (no stale position in a
terminal state — the endpoint's terminal carries 0/0 + the cleared
chunk position while the file counts keep their final values).
2. ``test_mid_run_reload_re_attaches_with_image_position`` (the
never-stale contract): the API starts the sync, the page's own
poll renders the image span (the FIRST observed position is
``1/4`` — the whole span is ahead), the test reloads the page
mid-run, and the load-time re-attach (``initSyncButton`` — a
running sync re-enters with the WHOLE status, phase 98) renders
the CURRENT image position with the A4 title/aria — then the run
settles to the counts line.
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import httpx
import pymupdf as fitz # PyMuPDF — the fixture-PDF builder (LOCKED A1)
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import text
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.rag.chunker import chunk_document
from app.rag.llm import LLMClient
from app.rag.pdf import extract_pdf_text, is_usable_pdf_text
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
MOCK_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module's app binds its own port
# instead (a same-port second uvicorn dies on bind and would drive the
# wrong server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_IMGPROG", "8171"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
#: The slow-LLM proxy's port (the conftest's mock LLM stays on
#: MOCK_PORT — the proxy fronts it).
SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT_IMGPROG", "8909"))
SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}"
#: Per-LLM-request delay on the proxy — sized so each in-flight VISION
#: PIPELINE's position (the describe + the content embed) outlives the
#: UI's 2 s poll (see the module docstring's timing fixture paragraph).
SLOW_DELAY_S = "1.5"
# --------------------------------------------------------------------------
# Fixture constants (deterministic, token-controlled)
# --------------------------------------------------------------------------
#: The local source — the temp directory's basename (``kind=local`` →
#: the directory's basename is the source name, phase 38).
SOURCE = "imgpass"
#: The single fixture file (the source is one file — the count part
#: reads "(1/1)" throughout the import phase).
PDF_NAME = "imgpass.pdf"
N_FILES = 1
#: The fixture PDF's shape — 10 text pages (each ~4 KB of filler —
#: 20 content chunks at the code-default sizing, 10 embed batches) +
#: 4 distinct non-blank images on pages 1–4 (the span's total —
#: post-dedupe, post-blank-gate: the work actually about to happen).
N_PAGES = 10
N_IMAGES = 4
#: The settled counts line (phase 150 — the box is the sole sync text
#: surface): the PDF + its 4 embedded-image docs, a fresh KB (nothing
#: updated/unchanged/pruned).
SETTLE_TEXT = f"{1 + N_IMAGES} added"
#: The three sizing knobs, pinned to the CODE DEFAULTS in both the
#: app-under-test env (the leak guards below) and the test process's
#: expected-value computation — an operator's gitignored ``.env``
#: (which may legitimately tune ``BOR_CHUNK_TARGET_CHARS``) cannot
#: change the suite's ground truth. Derived from the class fields, so
#: they never drift from app/config.py (the conftest leak-guard
#: pattern).
CHUNK_TARGET = _Settings.model_fields["chunk_target_chars"].default
CHUNK_OVERLAP = _Settings.model_fields["chunk_overlap_chars"].default
EMBED_BATCH_SIZE = _Settings.model_fields["embed_batch_size"].default
#: The test process's mirror of the app-under-test sizing settings
#: (explicit kwargs beat any ``.env`` for the fields the expected
#: values depend on).
_EXPECTED_SETTINGS = _Settings(
chunk_target_chars=CHUNK_TARGET,
chunk_overlap_chars=CHUNK_OVERLAP,
embed_batch_size=EMBED_BATCH_SIZE,
)
#: "Synced HH:MM" — the local-time last-result label (sources.js's
#: fmtSyncTime), any hour/minute (the test_sync_button.py pattern).
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}$")
#: The settled boot label — idle ("Sync sources") or the RE-ATTACHED
#: terminal from the previous test's sync (the module app is shared,
#: its in-memory sync state survives the test boundary, and the
#: phase-32 boot re-attach renders the last result).
IDLE_OR_SETTLED = re.compile(r"^(Sync sources|Synced \d{1,2}:\d{2})$")
#: The chunk part (the phase-129 unit-pinned shape): "· chunk n/m" —
#: ``m`` is asserted against the expected chunk count by the test.
CHUNK_PART_RE = re.compile(r"· chunk (\d+)/(\d+)")
#: The image part (the phase-152 unit-pinned shape): "· image n/m".
IMAGE_PART_RE = re.compile(r"· image (\d+)/(\d+)")
#: Generous settle budgets: a changed sync against the slowed LLM is
#: ≈ 40 s; the UI's 2 s poll settles at most one tick after the
#: terminal state lands.
SETTLE_TIMEOUT_MS = 120_000
# ---------------------------------------------------------------------------
# The fixture PDF (built with PyMuPDF — LOCKED A1)
# ---------------------------------------------------------------------------
def _noise_png(w: int, h: int, seed: int) -> bytes:
"""A deterministic colorful-noise PNG (the "real image" stand-in).
Different ``(w, h, seed)`` → different stored bytes (the phase-151
byte-dedupe keeps all four); the formula's sample mean is ~127 —
well under the phase-151 blank gate's 235 near-white floor, and
the samples scatter far outside the ±17 band — every image
survives the gate with room to spare (see the module docstring's
KB fixture paragraph)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, w, h))
for y in range(h):
for x in range(w):
pix.set_pixel(
x,
y,
((x * 7 + y * 11 + seed * 13) % 256,
(x * 3 + y * 17 + seed * 29) % 256,
(x * 5 + y * 23 + seed * 41) % 256),
)
return pix.tobytes("png")
#: The four fixture images' shapes (distinct sizes — the distinct-
#: bytes guarantee even if two seeds collided) — one per page 1–4.
_IMAGE_SHAPES = [(24, 18), (32, 24), (40, 20), (28, 28)]
def _page_text(k: int) -> str:
"""One page's deterministic ~4 KB filler (no randomness — the
expected chunk count is stable run to run)."""
base = (
f"Page {k} notes for the imgpass fixture document carry "
f"deterministic filler number {k} of the ten-page pdf, written "
f"out to size so the chunker has real content to work with "
f"across the page. "
)
return base * 11
def _build_pdf(pdf_path: Path) -> None:
"""The fixture PDF: ``N_PAGES`` text pages (``insert_textbox`` —
the return value is checked — an overfull rect would silently
drop text and skew the ground truth) + the 4 images on pages 1–4
(distinct ``insert_image`` streams — distinct xrefs, distinct
bytes). The metadata title is the stem (no title — the stem
fallback, phase 126 A2)."""
doc = fitz.open()
for k in range(1, N_PAGES + 1):
page = doc.new_page()
rc = page.insert_textbox(
fitz.Rect(48, 90, 548, 812), _page_text(k), fontsize=10
)
assert rc >= 0, f"the page {k} text did not fit the rect (rc={rc})"
if k <= N_IMAGES:
w, h = _IMAGE_SHAPES[k - 1]
page.insert_image(
fitz.Rect(72, 820 - h * 2, 72 + w * 2, 860),
stream=_noise_png(w, h, k),
)
doc.save(pdf_path)
doc.close()
def _expected_chunks(pdf_path: Path) -> list[str]:
"""The PDF's expected content chunks — the SAME function and
arguments the importer's ``_replace_content_chunks`` uses
(``chunk_document(content, rel, max(400, target), overlap)`` — the
importer's token-cap floor rule; ``rel`` = the source-relative
path) over the SAME parser the importer reads
(``extract_pdf_text`` — the ``---PAGE-N---`` page markers): the
endpoint's ``chunks_total`` must equal
``len(chunk_document(...))`` — the file's TRUE chunk count, not a
batch count."""
content = extract_pdf_text(pdf_path)
return chunk_document(
content, PDF_NAME, max(400, CHUNK_TARGET), CHUNK_OVERLAP
)
def _expected_batch_count(chunks: list[str]) -> int:
"""The expected embed-batch count — the SAME
``LLMClient._batch_texts`` partition the app's embed loop walks
(the phase-129 unit pattern — call it directly for the boundaries).
The task's sizing bar: ≥ 2 batches (the chunk part's climb is a
real sequence, not a single point)."""
return len(LLMClient(_EXPECTED_SETTINGS)._batch_texts(chunks))
# ---------------------------------------------------------------------------
# The in-browser snapshot functions (Playwright Mapping Rule — the
# rendered DOM is the observable).
#
# ``_JS_SAMPLE`` — test 1's dense sampler: the box text + the button
# title/aria read in the SAME in-browser evaluation (a 2 s poll tick
# updates the parts + the title together in one JS task, so the atomic
# check can never straddle one — the phase-128 pattern); ``running``
# marks the in-flight labels (the A4 title/aria contract applies to
# them).
#
# ``_JS_IMAGE_SNAP`` — test 2's wait: null until the box carries the
# ``· image n/<arg>`` part, then the texts (read atomically with the
# title/aria).
# ---------------------------------------------------------------------------
_JS_SAMPLE = """
() => {
const boxTextEl = document.querySelector('#sync-progress-text');
const btn = document.querySelector('#sync-btn');
if (!boxTextEl || !btn) return null;
const box = boxTextEl.textContent;
return {
box: box,
title: btn.getAttribute('title') || '',
aria: btn.getAttribute('aria-label') || '',
running: box.startsWith('Syncing…'),
};
}
"""
_JS_IMAGE_SNAP = """
(n) => {
const boxTextEl = document.querySelector('#sync-progress-text');
const btn = document.querySelector('#sync-btn');
if (!boxTextEl || !btn) return null;
const box = boxTextEl.textContent;
const m = box.match(/^Syncing… .*· image (\\d+)\\/(\\d+)$/);
if (!m || Number(m[2]) !== n) return null;
return {
box: box,
posN: Number(m[1]),
title: btn.getAttribute('title') || '',
aria: btn.getAttribute('aria-label') || '',
};
}
"""
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def imgpass_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The one-source temp tree (see the module docstring): the app
server runs on the same host, so the paths are visible to it.
The directory NAME is the source name (``kind=local``, phase 38).
EXACTLY ONE file: the PyMuPDF-built ``imgpass.pdf`` (10 text
pages + 4 distinct non-blank images — see :func:`_build_pdf`)."""
root = tmp_path_factory.mktemp("bor_imgprog")
src = root / SOURCE
src.mkdir()
_build_pdf(src / PDF_NAME)
assert len(list(src.rglob("*"))) == N_FILES
return src
def _app_env(base_url: str, tmp_path_factory: pytest.TempPathFactory) -> dict[str, str]:
"""The per-module app env (the conftest pattern, cf.
``test_oneshot_llm_retry.py`` / ``test_kb_tree.py``): the
``base_url`` picks the LLM (the slow proxy in front of the mock),
``BOR_GIT_SOURCES`` forced empty (the sync source is this suite's
own DB-registered local row), the sizing knobs pinned to the code
defaults (the leak guards — see the module constants), the
phase-126 knobs pinned EXPLICITLY (``BOR_IMPORT_EXTENSIONS`` with
the ``pdf`` token — without it the walk is blind to the fixture —
and ``BOR_IMAGES=true`` — the embedded pass is the span under
test), the E2E ``BOR_PDF_PICTORIAL`` off-pin (the house
convention — the suite's span is the embedded pass; the fixture's
≥100-char pages keep the pictorial pass a no-op even if the knob
leaked on), and the other leak-guarded code defaults."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url
)
# Mock-calibrated threshold (conftest pattern): this suite never
# asks the chat model anything — the gate is never on a path.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
# Phase 67: instant retry waits + the code-default budget (the
# conftest leak-guard pattern).
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
# reads it from cwd) — override it with an EMPTY value (the env var
# beats the .env file): the registry must hold EXACTLY the local
# directory this suite registers.
env["BOR_GIT_SOURCES"] = ""
# Leak guards (conftest pattern): an operator's local (gitignored)
# .env cannot leak corpus-specific settings into the app under
# test — including the chunking / batching knobs this suite's
# ground truth is computed from (see the module constants).
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
_Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
env["BOR_CHUNK_TARGET_CHARS"] = str(CHUNK_TARGET)
env["BOR_CHUNK_OVERLAP_CHARS"] = str(CHUNK_OVERLAP)
env["BOR_EMBED_BATCH_SIZE"] = str(EMBED_BATCH_SIZE)
# Phase 126/122: the import scope WITH the ``pdf`` token + the
# embedded-image toggle ON (the span under test) + the E2E
# pictorial off-pin + the suite-private image home (the app under
# test must not write copies into the owner's real ~/bor-sources).
env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf"
env["BOR_IMAGES"] = "true"
env["BOR_PDF_PICTORIAL"] = "0"
env["BOR_IMAGE_DIR"] = str(tmp_path_factory.mktemp("bor_imgprog_images"))
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
return env
@pytest.fixture(scope="module")
def slow_llm(mock_llm: int) -> Iterator[int]:
"""The delay-injecting reverse proxy in front of the mock LLM
(``tests/e2e/slow_llm.py``) — the timing fixture for the whole
suite: it delays EVERY LLM call, vision describes included, so
each in-flight image pipeline's position outlives the UI's 2 s
poll (see ``SLOW_DELAY_S`` + the module docstring's timing
fixture paragraph)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S
env["E2E_MOCK_PORT"] = str(MOCK_PORT)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app",
"--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{SLOW_URL}/v1/models")
yield SLOW_PORT
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_server(
mock_llm: int,
slow_llm: int,
imgpass_dir: Path,
tmp_path_factory: pytest.TempPathFactory,
) -> Iterator[str]:
"""The SLOW app under test (the whole suite):
``BOR_LLM_BASE_URL`` is the slow proxy in front of the mock (the
timing fixture — see the module docstring). ``imgpass_dir`` is a
dependency only for the fixture ordering (the temp tree exists
before the first test). Skips the module (clear instructions)
when Postgres is down — the sync needs the KB tables."""
env = _app_env(f"{SLOW_URL}/v1", tmp_path_factory)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
body = httpx.get(f"{APP_URL}/api/health", timeout=5).json()
if body["db"] != "up":
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
share one Postgres, so a leftover git_sources row or document would
pollute the file counts + the counts line the syncs and the
assertions pin exactly. ``sources_meta`` (the KB generation counter)
resets with the KB — the sync re-creates the row."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources, sources_meta, folder_summaries"
)
)
db.commit() # without the commit the TRUNCATE rolls back (the house pattern)
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
"""No background sync may leak across tests (the run state lives in
the app's memory, and a still-running run would keep importing into
the NEXT test's truncated KB): wait for the app's sync status to
be non-running BEFORE the truncate. Own admin session (the
endpoint is admin-only) — usually a no-op: the tests settle only
after their run's terminal state."""
with httpx.Client(base_url=base_url, timeout=5.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
if r.json()["state"] != "running":
return
time.sleep(0.2)
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
@pytest.fixture(autouse=True)
def _clean(app_url: str) -> Iterator[None]:
"""Per-test KB isolation (the phase-96 cleanup pattern): the
shared E2E Postgres is truncated before AND after every test —
each test seeds its own source row and runs its own sync, so the
KB rows + folder_summaries rows of the temp source are exactly
this test's own. The pre-wait keeps a leaked run from the previous
test (the module app is shared) from racing the truncate."""
_wait_sync_settled(app_url)
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _register_source(base_url: str, path: str) -> None:
"""Register the temp directory through the authenticated API (the
``test_local_directory_sources.py`` / phase-94 pattern — a plain
directory is an API/DB-only operation)."""
with httpx.Client(base_url=base_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.post(
"/api/git-sources", json={"kind": "local", "path": path}
)
assert r.status_code == 201, r.text
def _wait_terminal(app_url: str, timeout_s: float = 180.0) -> dict[str, Any]:
"""Poll the status endpoint (own admin session) until the
UI-/API-triggered run reaches a terminal state — the source of
truth for the settle (the UI's 2 s poll renders it at most one
tick later). A FAILED terminal fails the test with the terminal
body (the ``detail`` / error text) — never an opaque label
timeout."""
with httpx.Client(base_url=app_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
body = r.json()
if body["state"] in ("success", "failed"):
assert body["state"] == "success", body
return body
time.sleep(0.2)
raise AssertionError(f"the run never reached a terminal state: {body}")
def _start_sync_click(page: Page, app_url: str, sync_dir: Path) -> None:
"""Register the source, land the admin on the sources view, wait
for the settled boot label (the phase-64 pattern — also proves the
phase-32 boot re-attach completed, so no later re-render can
clobber the running state), then click **Sync sources** and pin
the click's immediate 202 state (the bare "Syncing…" — the count
AND the chunk parts hidden: no file yet, no batch in flight — and
the image part is absent too: no PDF's vision span has started,
``images_total == 0``)."""
_register_source(app_url, str(sync_dir))
login(page, app_url) # lands on /sources.html (the button's home)
btn = page.locator("#sync-btn")
expect(btn).to_be_visible(timeout=30_000)
expect(page.locator("#sync-label")).to_have_text(IDLE_OR_SETTLED)
expect(btn).to_be_enabled()
btn.click()
# The click's immediate state (the 202 moment — the bare prefix,
# the phase-64 contract): disabled, aria-busy, spinning icon, and
# the label's MAIN part alone ("Syncing…" — the 202 click carries
# no status, so the pinned parts hide, phase 128/129 writers).
# Phase 149 (task 01): the running detail rides the box — the 202's
# bare label is in #sync-progress (the button is the spinner).
expect(btn).to_be_disabled()
expect(btn).to_have_attribute("aria-busy", "true")
expect(btn.locator(".sync-icon")).to_have_class(re.compile(r"\bis-spinning\b"))
expect(page.locator("#sync-progress")).to_have_text("Syncing…")
expect(page.locator(".sync-label-count")).to_be_hidden()
expect(page.locator(".sync-label-chunk")).to_be_hidden()
expect(page.locator("#sync-error-banner")).to_be_hidden()
# ---------------------------------------------------------------------------
# 1. The anti-freeze proof — the label advances through the PDF's
# image span (≥3 distinct positions, monotonic, chunk part earlier,
# the A4 title/aria, the settled clearing)
# ---------------------------------------------------------------------------
def test_label_advances_through_the_image_span(
page: Page, app_url: str, imgpass_dir: Path
) -> None:
"""A fresh admin page on ``/sources.html`` clicks **Sync sources**;
the test then densely samples (≈ every 300 ms) the rendered
``#sync-progress-text`` — NOT the API (the rendered DOM is the
observable; the UI's own 2 s poll is the writer) — collecting the
DISTINCT values in first-seen order until the settled counts line,
and asserts the phase-152 contract end to end (see the module
docstring's test mapping): the image part climbs (≥3 distinct
positions, starting at 1, non-decreasing, never above the total,
reaching it), the chunk part appeared earlier (and persists
through the image span with the file's TRUE chunk total), the
A4 title/aria ride the full string on every running sample, and
the settle clears the image position (the counts line, the
terminal label, the cleared endpoint position — no stale
position in a terminal state)."""
page.set_default_timeout(30_000)
# The ground truth — computed from the PDF ON DISK with the SAME
# parser + function + arguments the importer uses (see the helper
# docstrings): the TRUE chunk count + the batch count (the task's
# sizing bar — ≥2 embed batches, so the chunk part's climb is a
# real sequence).
pdf_path = imgpass_dir / PDF_NAME
chunks = _expected_chunks(pdf_path)
n_chunks = len(chunks)
assert n_chunks >= EMBED_BATCH_SIZE + 1, n_chunks # 2+ embed batches
assert _expected_batch_count(chunks) >= 2 # the sizing bar
# The happy path's premise: a clean text layer (the phase-133
# fallback never fires) — the embedded-image pass is the file's
# one and only vision span (the ≥100-char pages keep the phase-140
# gate candidate-free even under the app's pinned-off knob).
assert is_usable_pdf_text(extract_pdf_text(pdf_path))
_start_sync_click(page, app_url, imgpass_dir)
# The observation loop (the task's "poll the page, not the API"):
# dense samples of the rendered box text (≈ 300 ms — the evaluate
# round-trip + the sleep) in first-seen order, each sample read
# atomically with the button's title/aria (A4 checked per running
# sample), until the settled counts line lands (or the generous
# deadline — a failed run then fails LOUDLY in _wait_terminal
# with the terminal body, never an opaque timeout).
seen: list[str] = []
last: str | None = None
deadline = time.monotonic() + 150
while time.monotonic() < deadline:
snap = page.evaluate(_JS_SAMPLE)
assert snap is not None
if snap["running"]:
# A4 — while a run is in flight, the title + the
# aria-label carry the SAME untruncated full string as the
# box (the "· image" part included).
assert snap["title"] == snap["box"], snap
assert snap["aria"] == snap["box"], snap
if snap["box"] != last:
seen.append(snap["box"])
last = snap["box"]
if snap["box"] == SETTLE_TEXT:
break
time.sleep(0.2)
# --- the settle (the UI's 2 s poll renders the terminal at most one
# tick after it lands; the endpoint is the source of truth) ---
assert last == SETTLE_TEXT, f"the run never settled to the counts line: {seen[-5:]}"
terminal = _wait_terminal(app_url)
# The counts line's ground truth: the PDF + its 4 embedded-image
# docs — one file, 5 added.
assert terminal["detail"]["added"] == 1 + N_IMAGES, terminal
# The keep-final-file-counts split (the phase-64/129 convention,
# restated for the phase-152 sub-position): the FILE counts keep
# their final values, the sub-file chunk + image positions clear
# with the file (0/0 in the terminal — no stale position), the
# phase clears.
assert terminal["files_done"] == N_FILES and terminal["files_total"] == N_FILES, terminal
assert terminal["current_file"] is None, terminal
assert terminal["current_chunk"] == 0 and terminal["chunks_total"] == 0, terminal
assert terminal["images_done"] == 0 and terminal["images_total"] == 0, terminal
assert terminal["phase"] is None, terminal
# --- the run was real (the prelude + the import span were
# sampled) ---
assert any(t == "Syncing…" for t in seen), f"no bare prelude label: {seen[:6]}"
import_labels = [t for t in seen if t.startswith("Syncing… ")]
assert any(f"({N_FILES}/{N_FILES})" in t for t in import_labels), (
f"no file count part: {import_labels[:6]}"
)
# --- the chunk part appeared EARLIER in the run (the two
# positions are distinct) ---
chunk_idx = next(
(i for i, t in enumerate(seen) if CHUNK_PART_RE.search(t)), None
)
assert chunk_idx is not None, f"no chunk part observed: {seen}"
first_chunk = CHUNK_PART_RE.search(seen[chunk_idx])
assert first_chunk is not None, seen[chunk_idx]
# The chunk part's total is the file's TRUE chunk count (not a
# batch count — the phase-129 A6 assumption, the E2E lens).
assert int(first_chunk.group(2)) == n_chunks, (first_chunk.group(0), n_chunks)
# --- the image part: ≥3 distinct positions (the anti-freeze
# proof — the owner's freeze is a single repeated label), the
# positions START at 1, are NON-DECREASING in observation order,
# never EXCEED the total (the post-dedupe/post-gate count — the
# work actually about to happen), and REACH it ---
def _image_part(t: str) -> re.Match[str]:
m = IMAGE_PART_RE.search(t)
assert m is not None, t # the list comprehension's filter guarantees it
return m
image_items = [
(i, _image_part(t))
for i, t in enumerate(seen)
if IMAGE_PART_RE.search(t)
]
assert len(image_items) >= 3, (
f"fewer than 3 distinct image positions observed: "
f"{[m.group(0) for _, m in image_items]}"
)
first_image_idx = image_items[0][0]
assert chunk_idx < first_image_idx, (chunk_idx, first_image_idx)
positions = [(int(m.group(1)), int(m.group(2))) for _, m in image_items]
assert positions[0][0] == 1, f"the climb does not start at 1: {positions}"
assert all(m == N_IMAGES for _, m in positions), positions
assert all(n <= N_IMAGES for n, _ in positions), positions
ns = [n for n, _ in positions]
assert ns == sorted(ns), f"the climb is backwards somewhere: {positions}"
assert N_IMAGES in ns, f"the climb never reached the total: {positions}"
# The two positions COEXIST (distinct and both may show — "… (1/1)
# · chunk 19/20 · image 3/4"): every image-span label carries the
# chunk part with it (the last chunk position persists through the
# file's image span — the per-file reset + the phase transitions
# clear it, not the end of the chunk span).
for i, _ in image_items:
cm = CHUNK_PART_RE.search(seen[i])
assert cm is not None, f"an image-span label lost the chunk part: {seen[i]}"
assert int(cm.group(2)) == n_chunks, (cm.group(0), n_chunks)
# --- the settle: the counts line, the terminal label, the parts
# cleared + empty, and the IMAGE part GONE (no stale position in a
# terminal state — the endpoint pin above is the server-side
# half) ---
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
btn = page.locator("#sync-btn")
expect(btn).to_be_enabled()
expect(btn).not_to_have_attribute("aria-busy")
expect(page.locator(".sync-label-count")).to_be_hidden()
expect(page.locator(".sync-label-count")).to_have_text("")
expect(page.locator(".sync-label-chunk")).to_be_hidden()
expect(page.locator(".sync-label-chunk")).to_have_text("")
# The counts line rides the box (phase 150 — the sole sync text
# surface): the image part is not in it, not in the title, not in
# the aria (the settle restores the ship aria value).
box = page.locator("#sync-progress-text").inner_text()
assert box == SETTLE_TEXT, box
assert "· image" not in box and "· image" not in (btn.get_attribute("title") or "")
assert btn.get_attribute("aria-label") == "Sync sources", btn.get_attribute("aria-label")
expect(page.locator("#sync-error-banner")).to_be_hidden()
# ---------------------------------------------------------------------------
# 2. The never-stale contract — a mid-run page reload re-attaches with
# the current image position visible
# ---------------------------------------------------------------------------
def test_mid_run_reload_re_attaches_with_image_position(
page: Page, app_url: str, imgpass_dir: Path
) -> None:
"""The admin clicks **Sync sources** (the run is started the way
the owner starts it — the click is what starts the page's 2 s
poll; the reload is the contract under test), the page's own 2 s
poll renders the image span (the FIRST observed position is
``1/4`` — the whole span is ahead, so the reload + the load-time
re-attach happen with the span in flight), the test reloads the
page mid-run, and the load-time re-attach (``initSyncButton`` —
a running sync re-enters with the WHOLE status, phase 98's
never-stale contract) renders the CURRENT ``· image n/4`` with
the A4 title/aria; the run then settles to the counts line (the
image part gone — no stale position)."""
page.set_default_timeout(30_000)
_start_sync_click(page, app_url, imgpass_dir)
# The image span, rendered by the page's own 2 s poll (the span
# starts after the prelude + the file's embed batches — ≈ 21 s
# under the slow proxy; each position holds 4.5 s — the describe
# + the image doc's 2 embed batches).
snap = page.wait_for_function(
_JS_IMAGE_SNAP, arg=N_IMAGES, timeout=60_000
).json_value()
# The FIRST observed position is 1 — the whole span is ahead (the
# reload's re-attach is guaranteed to catch the span in flight:
# ≈ 18 s remain, the local reload + the load-time status fetch
# land well inside it).
assert snap["posN"] == 1, snap
# A4 — the pre-reload running state is untruncated on the title +
# the aria (the "· image" part included).
assert snap["title"] == snap["box"] and "· image" in snap["title"], snap
assert snap["aria"] == snap["box"], snap
# The mid-run reload (the session cookie survives — the re-attach
# is the admin's, phase 98).
page.reload()
# The load-time re-attach: a running sync re-enters with the WHOLE
# status — the CURRENT image position is visible (the never-stale
# contract), the A4 title/aria ride it.
snap2 = page.wait_for_function(
_JS_IMAGE_SNAP, arg=N_IMAGES, timeout=30_000
).json_value()
assert snap2["title"] == snap2["box"] and "· image" in snap2["title"], snap2
assert snap2["aria"] == snap2["box"], snap2
# Then the settle: the counts line (the image part gone — the
# terminal clears the position, test 1 pins the endpoint half),
# the terminal label, the parts cleared + empty.
expect(page.locator("#sync-progress-text")).to_have_text(
SETTLE_TEXT, timeout=SETTLE_TIMEOUT_MS
)
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
btn = page.locator("#sync-btn")
expect(btn).to_be_enabled()
expect(btn).not_to_have_attribute("aria-busy")
expect(page.locator(".sync-label-count")).to_be_hidden()
expect(page.locator(".sync-label-chunk")).to_be_hidden()
assert "· image" not in (btn.get_attribute("title") or "")
expect(page.locator("#sync-error-banner")).to_be_hidden()
terminal = _wait_terminal(app_url)
assert terminal["detail"]["added"] == 1 + N_IMAGES, terminal
assert terminal["images_done"] == 0 and terminal["images_total"] == 0, terminal
+352 -1
View File
@@ -93,6 +93,7 @@ import asyncio
import logging
import os
import subprocess
import textwrap
import threading
import time
from collections.abc import Callable, Iterator, Sequence
@@ -100,6 +101,7 @@ from datetime import UTC, datetime
from pathlib import Path
from typing import Any
import pymupdf as fitz
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import select, text
@@ -309,6 +311,12 @@ class FakeImportSources:
# regresses) — the canned fake never fires it (the runner's
# status chunk fields stay 0/0, like a no-batch import).
self.chunk_hooks: list[object] = []
# Phase 152 (task 03): the image-position hook the runner
# passes (a live closure while wired, None if the wiring
# regresses) — the canned fake never fires it (the status
# image fields stay 0/0, like the chunk fields of a no-batch
# import).
self.image_hooks: list[object] = []
async def __call__(
self,
@@ -320,6 +328,7 @@ class FakeImportSources:
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -329,6 +338,7 @@ class FakeImportSources:
self.prune_flags.append(prune)
self.progress_hooks.append(progress)
self.chunk_hooks.append(on_chunk_progress)
self.image_hooks.append(on_image_progress)
self.ignore_maps.append(ignore_by_root or {})
self.include_hidden_maps.append(include_hidden_by_root or {})
self.doc_dates_maps.append(doc_dates_by_root or {})
@@ -500,6 +510,10 @@ def test_admin_sync_success_reports_full_detail(
# (the dataclass defaults).
"current_chunk": 0,
"chunks_total": 0,
# Phase 152 (task 01): the image-position keys — 0/0 idle
# (the dataclass defaults).
"images_done": 0,
"images_total": 0,
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
"phase": None,
"current_summary": None,
@@ -656,13 +670,23 @@ def test_double_trigger_while_running_returns_409(
class _GatedImport:
"""An import that fires the runner's progress hook once, then
parks on a threading gate — the test reads the status mid-import
(the phase ``"import"`` + the file hook)."""
(the phase ``"import"`` + the file hook).
Phase 152 (task 03): optionally ALSO fires the in-flight file's
image-position pairs (``image_pairs``) through the runner's OWN
``on_image_progress`` closure: pair one lands before the main park
(readable mid-run), every later pair after the main park + its own
park (``image_release``), so each position is readable on the
status between polls (the phase-129 chunk idiom, mirrored for the
vision-image position)."""
def __init__(
self,
summary: ImportSummary,
started: threading.Event,
release: threading.Event,
image_pairs: Sequence[tuple[int, int]] = (),
image_release: threading.Event | None = None,
) -> None:
self.summary = summary
self.started = started
@@ -672,6 +696,13 @@ class _GatedImport:
# passes (recorded for the wiring pin — the gated fake never
# fires it, so the status chunk fields stay 0/0).
self.chunk_hook: Callable[[int, int], None] | None = None
# Phase 152 (task 03): the image-position hook the runner
# passes (recorded for the wiring pin) + the optional image
# pairs (see the class docstring).
self.image_hook: Callable[[int, int], None] | None = None
self.image_pairs = list(image_pairs)
self.image_release = image_release
self.image_hook_calls: list[tuple[int, int]] = []
async def __call__(
self,
@@ -683,6 +714,7 @@ class _GatedImport:
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -691,6 +723,25 @@ class _GatedImport:
progress("repo", "notes/deep.md", 1, 3)
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
self.chunk_hook = on_chunk_progress
self.image_hook = on_image_progress
if on_image_progress is not None and self.image_pairs:
# Phase 152 (task 03): the in-flight PDF's image-position
# pairs, fired through the runner's OWN hook closure (the
# closure under test) — pair one before the main park, each
# later pair after the main park + its own park (each park
# lets the test read the status between polls). A wiring
# regression to None skips this branch and the pair test
# times out.
on_image_progress(*self.image_pairs[0])
self.image_hook_calls.append(self.image_pairs[0])
self.started.set()
await asyncio.to_thread(self.release.wait)
for pair in self.image_pairs[1:]:
on_image_progress(*pair)
self.image_hook_calls.append(pair)
if self.image_release is not None:
await asyncio.to_thread(self.image_release.wait)
return self.summary
self.started.set()
await asyncio.to_thread(self.release.wait)
return self.summary
@@ -867,6 +918,305 @@ def test_status_reports_the_phase_machine_across_the_run(
assert gated_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
def test_image_position_climbs_between_polls_mid_run(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""Phase 152 (task 03): while a PDF's vision span is in flight,
the status carries the in-flight image position — the pairs are
fired through the runner's OWN ``_image_hook`` closure (task 03's
wiring: ``import_sources(..., on_image_progress=_image_hook)``),
and the position CLIMBS between two polls of
``GET /api/sync/status`` (1/15 → 7/15 — monotonic, never above
the total, no backwards jumps — the owner's stuck label now
moves), paired with the file the ``progress`` hook named (the
hooks are paired — the position belongs to that file); the
success terminal clears the position to 0/0 while the FILE
counts keep their final values (the convention split — same as
the chunk position)."""
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
image_release = threading.Event()
gated_import = _GatedImport(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
threading.Event(),
image_pairs=[(1, 15), (7, 15)], # two describe calls of a 15-image span
image_release=image_release,
)
monkeypatch.setattr(sync_api, "import_sources", gated_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
# Idle: the image keys ride along as 0/0 (the dataclass defaults).
body = sync_client.get("/api/sync/status").json()
assert body["state"] == "idle"
assert body["images_done"] == 0 and body["images_total"] == 0
assert sync_client.post("/api/sync").status_code == 202
try:
assert gated_import.started.wait(5.0), "the run never reached the first image pair"
# Poll 1 — the import phase, pair one live: the position is
# 1/15, paired with the file the progress hook named; the
# chunk position is 0/0 (this canned file has no embed
# batches — the two positions are independent).
s = sync_client.get("/api/sync/status").json()
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 1 and s["images_total"] == 15
# Release pair one: the import fires pair two (7/15 — the
# NEXT describe call of the same 15-image span) and parks
# again.
gated_import.release.set()
deadline = time.monotonic() + 5.0
while time.monotonic() < deadline:
s = sync_client.get("/api/sync/status").json()
if s["images_done"] == 7:
break
assert s["state"] == "running", (
f"unexpected state {s['state']!r} while waiting for pair two: {s}"
)
time.sleep(0.05)
else:
pytest.fail("the second image pair never landed on the status")
# Poll 2 — the position CLIMBED (1 → 7, monotonic — no
# backwards jump, never above the total): the label is
# moving, not frozen.
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/notes/deep.md"
assert s["images_done"] == 7 and s["images_total"] == 15
assert 0 < s["images_done"] <= s["images_total"]
image_release.set()
finally:
gated_import.release.set()
image_release.set()
body = _poll(sync_client, "success")
# Terminal: the image position clears (0/0) ... the FILE counts
# KEEP their final values (the convention split — the position is
# a sub-position of a done file, like the chunk position).
assert body["images_done"] == 0 and body["images_total"] == 0
assert body["current_file"] is None
assert body["files_done"] == 1 and body["files_total"] == 3
# Wiring: BOTH pairs went through the runner's OWN live closure
# (the status reads above came through it — a wiring regression
# to None would have skipped the firing branch and timed out),
# and the file hook fired once.
assert gated_import.image_hook is not None
assert gated_import.image_hook_calls == [(1, 15), (7, 15)]
assert gated_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
# --- phase 152 (task 04): the full-run image-position monotonicity -------
def _kb_solid_png(value: int) -> bytes:
"""A 40x30 solid-color PNG (channel *value*): every value used
below stays under the phase-151 blank gate's 235 mean floor
(kept, never blank-skipped), and distinct values are distinct
images (the parser's dedupe keeps every one)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 40, 30), 0)
pix.clear_with(value)
return pix.tobytes("png")
def _kb_image_pdf(path: Path, values: Sequence[int]) -> None:
"""A REAL one-page PDF (the fixture KB's image-bearing file): a
text block comfortably over the phase-140 100-char sparseness
bar (a USABLE layer — the phase-133 gate stays closed, zero
pictorial candidates even with the toggle on — the "no other
vision span" class) carrying *values* the solid-color embedded
images (one placement each, distinct)."""
doc = fitz.open()
page = doc.new_page()
text = (
"The quadlet unit file defines the container interface, the "
"image, the network, and the restart policy. Every key maps "
"to a docker flag in the generated service file, and comments "
"use the usual semicolon form throughout the whole document."
)
for i, line in enumerate(textwrap.wrap(text, 60)):
page.insert_text((72, 72 + i * 14), line)
y = 300
for value in values:
page.insert_image(fitz.Rect(72, y, 112, y + 30), stream=_kb_solid_png(value))
y += 40
doc.save(path)
doc.close()
class _GatedVisionLLM(FakeEmbedder):
"""The phase-152 (task 04) mock LLM — the deterministic
in-process ``FakeEmbedder`` whose VISION calls (the multimodal
describe shape ``describe_image`` sends) PARK on a threading
gate: the run parks inside the PDF's embedded-image span, so the
test can read the status between the describe calls
(deterministic — no sleeps, no wall-clock dependence; the gate
holds the run until the test releases it). The hook fired BEFORE
the parked call (the task-02 contract), so the position is live
while the run is parked. Text (``lite``) calls keep the
``FakeEmbedder`` behaviour; the embeds are the deterministic
vectors."""
def __init__(self) -> None:
super().__init__()
self.vision_parks = 0 # the vision calls that parked (and released)
self.vision_release = threading.Event()
async def chat(
self, messages: list[dict[str, Any]], model: str | None = None
) -> str:
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
# One slow vision describe call — the gate (clear before
# the wait — a latching set would otherwise leak across
# calls): the test releases after reading the position.
self.vision_release.clear()
self.vision_parks += 1
await asyncio.to_thread(self.vision_release.wait)
return "A solid color swatch described." # deterministic, non-empty
return await super().chat(messages, model)
def test_image_position_monotonic_across_polls_over_a_real_pdf(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
clean_documents: None,
) -> None:
"""Phase 152 (task 04): a FULL sync run over a fixture KB with
ONE image-bearing PDF (the REAL ``import_sources`` — the
runner's ``_image_hook`` is driven by the importer's own
embedded-image span; the LLM is the deterministic mock whose
vision calls park on a gate) — polling ``GET /api/sync/status``
from the test while the run is parked between the describe
calls, the observed ``images_done`` sequence is MONOTONICALLY
NON-DECREASING, never above ``images_total``, and ends at
``images_total`` (no backwards jumps, no overshoot — the
owner's stuck label climbs one describe call at a time); the
position is paired with the file the ``progress`` hook named
(the hooks are paired), the success terminal clears it to 0/0
while the FILE counts keep their final values, and the KB
carries the PDF + its N image docs (the span really ran)."""
kb = tmp_path / "PictKB"
kb.mkdir()
(kb / "notes.md").write_text("# Notes\nthe local fixture\n", encoding="utf-8")
_kb_image_pdf(kb / "report.pdf", (90, 130, 170)) # 3 distinct images
n = 3
_seed_local(db, kb)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
clone_calls, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
llm = _GatedVisionLLM()
llm.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
import_extensions="md,txt,pdf",
images=True,
image_dir=str(tmp_path / "images"),
pdf_page_dir=str(tmp_path / "pages"),
)
monkeypatch.setattr(sync_api, "LLMClient", lambda: llm)
_stub_probe(monkeypatch) # the probe passes on the mock (no network)
_login(sync_client)
def status() -> dict:
return sync_client.get("/api/sync/status").json()
def wait_for(predicate, what: str, timeout: float = 10.0) -> None:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if predicate():
return
time.sleep(0.02)
raise AssertionError(f"timed out waiting for {what}")
# Idle: the image keys ride along as 0/0 (the dataclass
# defaults).
body = status()
assert body["state"] == "idle"
assert body["images_done"] == 0 and body["images_total"] == 0
assert sync_client.post("/api/sync").status_code == 202
try:
# The run reaches the PDF's vision span: the first hook fired
# (total = the span's N) and the first describe call parked
# the run (the gate).
wait_for(
lambda: status()["state"] == "running"
and status()["phase"] == "import"
and status()["images_total"] == n,
"the run to reach the PDF's vision span",
)
# Each describe call: the hook fired BEFORE the call, so the
# position is live while the run is parked in the call —
# observe it, release the call, let the next position climb.
observed: list[tuple[int, int]] = []
for expected in range(1, n + 1):
def position_ready(target: int = expected) -> bool:
# the default argument binds the loop variable (B023)
return status()["images_done"] == target
wait_for(position_ready, f"images_done to climb to {expected}")
s = status()
assert s["state"] == "running" and s["phase"] == "import"
# The position is paired with the file the progress hook
# named (the two hooks are paired — the position belongs
# to that file).
assert s["current_file"] == "PictKB/report.pdf"
observed.append((s["images_done"], s["images_total"]))
llm.vision_release.set()
# The run finishes the rest of the pipeline (the span is
# done — no more vision calls) and settles.
assert llm.vision_parks == n, "one park per describe call"
finally:
llm.vision_release.set()
body = _poll(sync_client, "success")
# The observed sequence: monotonically non-decreasing, never
# above the total, ending at the total (the gate made it exact —
# 1..N, no backwards jumps, no overshoot).
dones = [d for d, _ in observed]
assert dones == list(range(1, n + 1)), f"the sequence: {observed}"
assert all(d <= t for d, t in observed), "no overshoot"
assert dones[-1] == n, "ends at images_total"
assert {t for _, t in observed} == {n}, "one total for the whole span"
# Terminal: the image position clears (0/0) ... the FILE counts
# KEEP their final values (the convention split — the position
# is a sub-position of a done file, like the chunk position).
assert body["images_done"] == 0 and body["images_total"] == 0
assert body["current_file"] is None
assert body["files_done"] == 2 and body["files_total"] == 2
assert clone_calls == [] # nothing to clone — local is walked directly
# The KB carries the fixture md + the PDF + its N image docs —
# the span really ran (N new image documents, no failures).
assert body["detail"]["added"] == 2 + n
assert body["detail"]["errors"] == 0
docs = {(d["source"], d["path"])
for d in sync_client.get("/api/docs").json()["documents"]}
assert ("PictKB", "report.pdf") in docs
assert sum(1 for _, p in docs if "#page=" in p) == n
def test_failed_terminal_after_the_summary_hook_clears_phase_keeps_counts(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
@@ -1497,6 +1847,7 @@ def test_import_error_is_reported_with_credentials_masked(
session: Session | None = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None,
include_hidden_by_root: dict[str, bool] | None = None,
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
+242 -37
View File
@@ -127,6 +127,28 @@ setSyncProgress and pins the result mode behaviorally (a settle
result line reveals the box, "" empties + hides it); the bare
settle's row now shows the box LEFT ALONE (the settle label rides
the button only — A2).
Phase 152 (task 03) adds the label's FOURTH part — the vision-image
position: fmtSyncPhaseLabel's import fall-through gains "· image
n/m" (rendered while images_total > 0 — a PDF's vision span is in
flight; n the 1-based image about to be described, m the file's
total images, ONE continuous counter across the PDF's three vision
spans), the overview/summaries branches return image: "" (the
position belongs to the import phase only — the status's image
fields are cleared at those transitions, so the branches never read
them), and the bare builder stays image-free (the upload label and
the 202/409 click stay bare). setSyncLabel(mainText, countText =
"", chunkText = "", running = false, imageText = "") appends the
image part LAST to the composed full string — so the box, the
button title, and the running aria-label all carry
"Syncing… <file> (2/3) · chunk 14/38 · image 3/15". The image part
gets NO span of its own (the running button is the spinner — its
parts are hidden, phase 149): it rides ONLY the composed string,
while the chunk part keeps its .sync-label-chunk span; the two
positions are distinct and each disappears independently. The node
harness gains the two in-flight image rows (both parts at once —
the owner's stuck label — and the image-only pictorial PDF) and
pins the composed full string byte-for-byte.
"""
from __future__ import annotations
@@ -298,8 +320,9 @@ def test_fmt_sync_label_returns_the_parts_contract() -> None:
def test_set_sync_label_is_the_single_parts_writer() -> None:
"""setSyncLabel(mainText, countText = "", chunkText = "",
running = false) — Phase 128 (task 01) + Phase 129 (task 04):
the SINGLE point that writes #sync-label. The MAIN part gets
running = false, imageText = "") — Phase 128 (task 01) + Phase
129 (task 04) + Phase 152 (task 03): the SINGLE point that writes
#sync-label. The MAIN part gets
mainText (textContent only — the house rule, no HTML strings);
the COUNT part gets countText when present (textContent +
hidden=false — the "(n/m)" position is visible, NEVER clipped)
@@ -309,11 +332,18 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
hidden=true — the part renders only while a file's embed batch
is in flight). Then the UNTRUNCATED full label — mainText +
(countText ? " " + countText : "") + (chunkText ? " " + chunkText
: "") — rides the button title (A4 — the screen reader hears the
whole "Syncing… <file> (2/3) · chunk 14/38" there, and in the
box — see below). Phase 149 (task 01): the fourth parameter
: "") + (imageText ? " " + imageText : "") — rides the button
title (A4 — the screen reader hears the
whole "Syncing… <file> (2/3) · chunk 14/38 · image 3/15" there,
and in the box — see below). The image part (phase 152) gets NO
span of its own (the running button is the spinner — its parts
are hidden, phase 149): it rides only the composed full string.
Phase 149 (task 01): the fourth parameter
`running` (the in-flight flag — the running entry passes true,
the settle the default false). Phase 150 (task 02): while
the settle the default false); phase 152 (task 03) adds the
fifth — `imageText` (the bare builders and the non-running calls
pass nothing, so it defaults "" and the part stays out of the
string). Phase 150 (task 02): while
running, the SAME full label goes into the detail box
(#sync-progress — the SOLE aria-live region, the announcer the
screen reader hears) through the single box writer
@@ -324,10 +354,20 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
running / the ship value "Sync sources" otherwise
(overwrites/restores only — never removes)."""
body = _fn(_js(), "setSyncLabel")
assert (
'function setSyncLabel(mainText, countText = "", chunkText = "", running = false)'
in body
), "phase 149: the in-flight flag is the fourth parameter"
# the exact signature, byte-for-byte (concatenated — the literal
# is 105 columns, over the line cap): main, count, chunk, the
# phase-149 `running` flag (fourth), the phase-152 `imageText`
# (fifth — the bare builders + non-running calls pass nothing,
# so it defaults to the empty string).
expected_sig = (
'function setSyncLabel(mainText, countText = "", chunkText = "",'
' running = false, imageText = "")'
)
assert expected_sig in body, (
"phase 149: the in-flight flag is the fourth parameter; phase"
" 152: the image part is the fifth, defaulting to the empty"
" string (the part stays out of the composed full string)"
)
# the parts: textContent only (the house rule), null-guarded refs
assert "syncLabelMain.textContent = mainText" in body
assert "syncLabelCount.textContent = countText" in body
@@ -353,8 +393,12 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
"const full =\n"
" mainText +\n"
" (countText ? ` ${countText}` : \"\") +\n"
" (chunkText ? ` ${chunkText}` : \"\");"
) in body, ("the full label is main + ' ' + count + ' ' + chunk (spaces = the CSS gaps)")
" (chunkText ? ` ${chunkText}` : \"\") +\n"
" (imageText ? ` ${imageText}` : \"\");"
) in body, (
"the full label is main + ' ' + count + ' ' + chunk + ' ' +"
" image — the image part LAST (spaces = the CSS gaps)"
)
i_full = body.find("const full")
i_title = body.find("syncBtn.title = full")
i_box = body.find("if (running) setSyncProgress(full);")
@@ -375,7 +419,8 @@ def test_set_sync_label_is_the_single_parts_writer() -> None:
def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
"""fmtSyncPhaseLabel(status) — the SYNC job's phase-aware label
(phase 98 task 02, D2; phase 128 task 01: the PARTS). "overview"
(phase 98 task 02, D2; phase 128 task 01: the PARTS; phase 152
task 03: the IMAGE part). "overview"
→ main exactly "Writing KB overview…" (no folder, no count part);
"summaries" → main "Summarizing folders… <current_summary>" +
count "(n/m)" — the folder part (one leading space +
@@ -385,7 +430,14 @@ def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
a total); anything else (the null prelude, "import") → the
phase-64 fmtSyncLabel parts, byte-identical (same builder, same
fields, same order). The UPLOAD job never enters here — its status
has no phase (D2)."""
has no phase (D2). Phase 152 (task 03): the import fall-through
is the ONLY branch that reads the status's image fields — "·
image n/m" renders while images_total > 0 (a PDF's vision span is
in flight — n the 1-based image about to be described, m the
file's total images) and hides at 0/0; the overview/summaries
branches return image: "" and never read the fields (the position
belongs to the import phase only — the runner clears it at those
transitions)."""
body = _fn(_js(), "fmtSyncPhaseLabel")
assert "function fmtSyncPhaseLabel(status)" in body
# "overview": the exact copy, no folder, no counts.
@@ -421,14 +473,30 @@ def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
)
assert "chunks_total" not in ov, "the overview branch never reads the chunk fields"
assert "chunks_total" not in sum_, "the summaries branch never reads the chunk fields"
# phase 152 (task 03): the overview/summaries labels carry NO
# image part either (the position belongs to the import phase
# only) and never read the status's image fields (the runner
# clears them at those transitions).
assert 'image: ""' in ov, "the overview label carries no image part"
assert 'image: ""' in sum_, (
"the summaries label carries no image part (its (n/m) is its position)"
)
assert "images_total" not in ov, "the overview branch never reads the image fields"
assert "images_total" not in sum_, "the summaries branch never reads the image fields"
# anything else (null prelude, "import"): byte-identical to today's
# sync label — same builder, same fields, same order — plus the
# phase-129 chunk part: "· chunk n/m" ONLY while chunks_total > 0
# (a file's embed batch is in flight — n the 1-based first chunk
# of the batch, m the file's total content chunks, batch
# granularity), "" otherwise (0/0 before the first batch — the
# part hides). The fall-through is the ONLY branch that reads the
# status's chunk fields.
# part hides) + the phase-152 image part: "· image n/m" ONLY
# while images_total > 0 (a PDF's vision span is in flight — n
# the 1-based image about to be described, m the file's total
# images — ONE continuous counter across the PDF's three vision
# spans), "" otherwise (0/0 — the part hides). The fall-through
# is the ONLY branch that reads the status's chunk + image
# fields, and the two parts are independent (both may show,
# each disappears on its own 0/0).
i_destr = body.find("const { main, count } = fmtSyncLabel(")
assert -1 < i_destr < i_fall, "the fall-through destructures the phase-64 parts"
fall = body[i_destr:]
@@ -444,11 +512,22 @@ def test_fmt_sync_phase_label_is_the_exact_d2_parts_contract() -> None:
i_guard = fall.find("status.chunks_total > 0")
i_else = fall.find(": \"\";", i_guard)
assert -1 < i_guard < i_else, "0/0 (before the first batch) → the empty chunk part"
assert "return { main, count, chunk }" in fall
# phase 152 (task 03): the image part — "· image n/m" ONLY while
# images_total > 0, the exact template + the 0/0 → "" guard, and
# the return gains the fourth part.
assert "status.images_total > 0" in fall, (
"the image part renders only while a vision span is in flight"
)
assert "`· image ${status.images_done}/${status.images_total}`" in fall
i_iguard = fall.find("status.images_total > 0")
i_ielse = fall.find(": \"\";", i_iguard)
assert -1 < i_iguard < i_ielse, "0/0 (before the first image) → the empty image part"
assert "return { main, count, chunk, image }" in fall
# the bare builder (the upload job + the 202/409 click) returns NO
# chunk part — those labels stay bare.
# chunk OR image part — those labels stay bare.
bare = _fn(_js(), "fmtSyncLabel")
assert "chunk" not in bare, "fmtSyncLabel stays chunk-free (upload + the click stay bare)"
assert "image" not in bare, "fmtSyncLabel stays image-free (upload + the click stay bare)"
def test_enter_running_state_is_phase_aware_and_writes_the_parts() -> None:
@@ -460,13 +539,16 @@ def test_enter_running_state_is_phase_aware_and_writes_the_parts() -> None:
endpoint has no phase) and the 202/409 click keep the bare
fmtSyncLabel parts, byte-identical. Phase 128 (task 01): the entry
itself no longer writes the label — it hands the parts to the
SINGLE writer (setSyncLabel(main, count, chunk) — phase 129:
the chunk part destructures with a "" default), which lands the
SINGLE writer (setSyncLabel(main, count, chunk, running, image) —
phase 129: the chunk part destructures with a "" default, phase
152: the image part does too), which lands the
main part, the pinned count part (hidden when absent), the pinned
chunk part (phase 129 — hidden when absent), and the FULL
untruncated label on the button title + #sync-progress (A4 — set
for EVERY running label; CSS ellipsizes the main part only).
Phase 150 (task 02): the box write rides in the writer too
for EVERY running label; CSS ellipsizes the main part only —
phase 152: with the image part appended last when a PDF's vision
span is in flight; it gets no span of its own). Phase 150 (task
02): the box write rides in the writer too
(setSyncLabel → setSyncProgress, running mode)."""
body = _fn(_js(), "enterSyncRunningState")
assert (
@@ -487,13 +569,16 @@ def test_enter_running_state_is_phase_aware_and_writes_the_parts() -> None:
# the parts are handed to the single writer (A4 lives in it) —
# phase 129: the chunk part destructures with a "" default (the
# bare builders — the upload job, the 202/409 click — return no
# chunk part, so the part stays hidden there); phase 149: the
# chunk part, so the part stays hidden there); phase 152: the
# image part does too (it rides only the composed full string —
# no span); phase 149: the
# call passes `running = true` (the detail box reveals + the
# full-label aria-label ride in the writer).
assert "const { main, count, chunk = \"\" } =" in body, (
"the chunk part destructures with the '' default (bare builders return none)"
assert 'const { main, count, chunk = "", image = "" } =' in body, (
"the chunk + image parts destructure with the '' defaults"
" (bare builders return none)"
)
i_set = body.find("setSyncLabel(main, count, chunk, true)")
i_set = body.find("setSyncLabel(main, count, chunk, true, image)")
assert -1 < i_fall < i_set, "the parts are written after the split (running = true)"
assert body.count("setSyncLabel(") == 1, "one writer call per entry"
assert "syncBtn.title =" not in body, ("the title write lives in setSyncLabel (A4)")
@@ -713,6 +798,25 @@ enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
current_chunk: 14, chunks_total: 38,
});
show("import-inflight");
// 1b. Phase 152 (task 03): import, a PDF's vision span in flight
// (the owner's stuck label — the last chunk position 125/126 is
// STILL on the status + the climbing image position 3/15): the
// composed full string carries BOTH parts (chunk keeps its span,
// the image part rides the string only — no span):
enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 3,
current_chunk: 125, chunks_total: 126, images_done: 3, images_total: 15,
});
show("import-image-span");
// 1c. Phase 152 (task 03): import, a purely pictorial PDF (no text
// chunks — the chunk position never left 0/0) in its vision
// span: the image part shows ALONE (the chunk part stays hidden
// + empty — the two parts are independent):
enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 3,
current_chunk: 0, chunks_total: 0, images_done: 7, images_total: 7,
});
show("import-image-only");
// 2. import before the first batch (0/0) — the chunk part hides:
enterSyncRunningState("sync", "src/a/b.md", 2, 3, {
phase: "import", current_file: "src/a/b.md", files_done: 2, files_total: 3,
@@ -893,6 +997,96 @@ test-document-viewer pattern). The pinned "· chunk n/m" part (the
)
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_image_part_renders_only_in_the_import_phase_under_node() -> None:
"""Phase 152 (task 03): behavioral check (node) of the REAL label
path — the extracted fmtSyncPhaseLabel / setSyncLabel /
enterSyncRunningState run against the stub element refs. The
pinned "· image n/m" part (the task's fixture values):
* import, a PDF's vision span in flight (images_total 15 > 0)
WITH the last chunk position still on the status (125/126):
the composed full string carries BOTH parts — the box, the
button title, and the running aria-label all read exactly
"Syncing… src/a/b.md (2/3) · chunk 125/126 · image 3/15"
(the owner's stuck label, now moving); the chunk part KEEPS
its span ("· chunk 125/126" visible) and the image part gets
NO span of its own (it rides the composed string only — the
running button is the spinner, phase 149);
* import, a purely pictorial PDF (the chunk position never left
0/0) in its vision span: the image part shows ALONE —
"Syncing… src/a/b.md (2/3) · image 7/7" — and the chunk part
stays hidden + empty (the two parts are distinct and each
disappears independently on its own 0/0);
* every OTHER running state (the pre-batch import, the prelude,
the 202/409 click, the upload run, the overview, the
summaries — the status's image fields are 0/0 or cleared at
those transitions) leaves the composed string image-free.
The box / Cancel-control / is-running contracts of the running
state apply to the new rows exactly as to every other running
state (the harness's other pins cover them per tag)."""
rows = _chunk_harness_rows()
by_tag = {row[0]: row for row in rows}
# row = [tag, title, announcer (the box's text — the sole live
# region, phase 150), main, count, countHidden, chunk,
# chunkHidden, cancelHidden, cancelDisabled, progressText,
# progressHidden, btnRunning, iconSpinning, ariaLabel]
# 1b. both parts at once — the composed full string, byte-for-byte
# (the image part LAST, after the chunk part), on the box, the
# title, and the running aria-label (A4) — the chunk span
# keeps its own part.
full = "Syncing… src/a/b.md (2/3) · chunk 125/126 · image 3/15"
row = by_tag["import-image-span"]
assert row[1] == full, row[1]
assert row[2] == full, "the announcer (the box) hears the whole thing (A4)"
assert row[14] == full, "the running button's accessible name is the full label"
assert row[10] == full, "the detail box carries the EXACT full label"
assert row[6] == "· chunk 125/126" and row[7] is False, (
"the chunk part keeps its own span alongside the image part"
)
# 1c. the image part alone (the chunk position never left 0/0 —
# the two parts are independent).
row = by_tag["import-image-only"]
assert row[1] == "Syncing… src/a/b.md (2/3) · image 7/7", row[1]
assert row[2] == row[1] and row[14] == row[1], row
assert row[6] == "" and row[7] is True, (
"the chunk part stays hidden + empty (its own 0/0)"
)
# The pre-batch import (0/0 on BOTH positions) and every other
# running state: the composed string is image-free (the part
# hides on its own 0/0 — the overview/summaries transitions
# clear the position server-side, the bare builders carry none).
for tag, expected_title in (
("import-inflight", "Syncing… src/a/b.md (2/3) · chunk 14/38"),
("import-prebatch", "Syncing… src/a/b.md (2/3)"),
("prelude", "Syncing…"),
("click", "Syncing…"),
("upload", "Importing"),
("overview", "Writing KB overview…"),
("summaries", "Summarizing folders… src/alpha (2/3)"),
):
row = by_tag[tag]
assert row[1] == expected_title, (tag, row)
assert "· image" not in row[1], (
f"{tag}: the image part shows only while images_total > 0"
)
# The running-state contracts apply to the new rows too: box
# visible + is-running + spinning, the Cancel control shown +
# enabled (a sync run in flight).
for tag in ("import-image-span", "import-image-only"):
row = by_tag[tag]
assert row[11] is False, f"{tag}: the box must be visible while in flight"
assert row[12] is True and row[13] is True, f"{tag}: is-running + spinning"
assert row[8] is False and row[9] is False, (
f"{tag}: the Cancel control is shown + enabled while a sync runs"
)
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_progress_box_and_running_button_contract_under_node() -> None:
"""Phase 149 (task 01) — the detail box + the icon-only running
@@ -995,14 +1189,16 @@ def test_progress_box_and_running_button_contract_under_node() -> None:
def test_every_other_state_passes_no_chunk_text() -> None:
"""Source-level pin of the render-only-while-chunks_total>0
contract: the import fall-through is the ONLY place in sources.js
that reads the status's chunk fields (the summaries/overview
branches and the bare builders never touch them), and every
NON-running label emission passes no chunk text (the part hides):
the settle/idle labels go through setSyncLabel(label) — the only
running-flagged call site (phase 149: `running = true`) is the
running entry."""
"""Source-level pin of the render-only-while-chunks_total>0 +
render-only-while-images_total>0 contracts: the import
fall-through is the ONLY place in sources.js that reads the
status's chunk OR image fields (the summaries/overview branches
and the bare builders never touch them), and every
NON-running label emission passes no chunk/image text (both
parts hide): the settle/idle labels go through setSyncLabel(label)
— the only running-flagged call site (phase 149: `running = true`)
is the running entry (phase 152: passing the image part as its
fifth argument)."""
js = _js()
body = _fn(js, "fmtSyncPhaseLabel")
i_fall = body.find("fmtSyncLabel(")
@@ -1010,12 +1206,21 @@ def test_every_other_state_passes_no_chunk_text() -> None:
assert "chunks_total" not in pre and "current_chunk" not in pre, (
"the overview/summaries branches never read the chunk fields"
)
assert "images_total" not in pre and "images_done" not in pre, (
"the overview/summaries branches never read the image fields (phase 152)"
)
fall = body[i_fall:]
assert "status.chunks_total" in fall and "status.current_chunk" in fall
assert js.count("setSyncLabel(main, count, chunk, true)") == 1, (
"the ONLY running-flagged call site is the running entry (phase 149)"
assert "status.images_total" in fall and "status.images_done" in fall, (
"the import fall-through is the ONLY reader of the image fields (phase 152)"
)
assert js.count("setSyncLabel(main, count, chunk, true, image)") == 1, (
"the ONLY running-flagged call site is the running entry (phase 149"
" — phase 152: with the image part)"
)
assert "setSyncLabel(label)" in js, (
"the settle/idle labels stay two-arg (no chunk/image)"
)
assert "setSyncLabel(label)" in js, "the settle/idle labels stay two-arg (no chunk)"
# ---------- the two-job tick (startSyncPolling) ----------
+508 -1
View File
@@ -18,7 +18,8 @@ import base64
import hashlib
import logging
import os
from collections.abc import Callable
import textwrap
from collections.abc import Callable, Sequence
from datetime import UTC, datetime
from pathlib import Path
@@ -41,6 +42,11 @@ from app.rag.importer import (
match_extension,
)
from app.rag.llm import EmbeddingError, LLMError
from app.rag.pdf import (
extract_pdf_images_with_stats,
extract_pdf_page_texts,
pictorial_pages,
)
from tests.fakes import FakeEmbedder
#: The original seven A9 formats as dotted suffixes (pre-phase-47 default
@@ -2088,3 +2094,504 @@ def test_fallback_embedded_images_still_join_the_pipeline(
assert image_doc.content != d1 and image_doc.content != d2
finally:
_cleanup_source(db, "both")
# ---------- phase 152 (task 02): on_image_progress plumbing ----------
#: A text block comfortably over the phase-140 100-char sparseness bar
#: (a USABLE layer, zero pictorial candidates even with the toggle on
#: — the "no vision span" class). Inserted wrapped at 60 chars: a
#: single long line would be CLIPPED at the page edge on extraction,
#: while the wrapped multi-line read round-trips (the phase-140
#: fixture's class property).
_P152_DENSE_TEXT = (
"The quadlet unit file defines the container interface, the image, "
"the network, and the restart policy. Every key maps to a docker "
"flag in the generated service file, and comments use the usual "
"semicolon form throughout the whole document."
)
assert len(_P152_DENSE_TEXT.strip()) > 100 # the class property, not a typo
def _p152_solid_png(value: int) -> bytes:
"""A 40x30 solid-color PNG (channel *value*): every value used
below stays under the phase-151 blank gate's 235 mean floor (kept,
never blank-skipped), and distinct values are distinct images (the
dedupe keeps every one)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 40, 30), 0)
pix.clear_with(value)
return pix.tobytes("png")
def _build_image_pdf(
path: Path,
values: Sequence[int],
dense: bool = True,
repeat_first: bool = False,
) -> None:
"""A REAL one-page PDF: *dense* a text block over the 100-char bar
(a USABLE layer — the fallback never fires; zero pictorial
candidates even with the toggle on) carrying *values* the
solid-color embedded images (one placement each, distinct).
``dense=False`` inserts the sparse ``BESTA`` marker instead (a
pictorial candidate under the default 100-char bar).
``repeat_first`` adds a SECOND placement of the first image (the
same stream — one xref, the parser's repeated-placement class,
phase 151: the dedupe drops it, the position count is
post-dedupe)."""
doc = fitz.open()
page = doc.new_page()
text = _P152_DENSE_TEXT if dense else "BESTA"
for i, line in enumerate(textwrap.wrap(text, 60)):
page.insert_text((72, 72 + i * 14), line)
y = 300
for value in values:
page.insert_image(fitz.Rect(72, y, 112, y + 30), stream=_p152_solid_png(value))
y += 40
if repeat_first:
page.insert_image(
fitz.Rect(150, 300, 190, 330), stream=_p152_solid_png(values[0])
)
doc.save(path)
doc.close()
def _build_pictorial_image_pdf(path: Path, values: Sequence[int]) -> None:
"""A REAL two-page PDF — the ONE continuous counter's two-span
case: page 1 is SPARSE (``BESTA`` — a pictorial candidate under
the default 100-char bar), page 2 is dense (never a candidate) and
carries *values* the solid-color embedded images. Both the
phase-140 pictorial pass (before the document commit) and the
phase-126 embedded-image pass (after it) fire on it."""
doc = fitz.open()
doc.new_page().insert_text((72, 72), "BESTA")
page = doc.new_page()
for i, line in enumerate(textwrap.wrap(_P152_DENSE_TEXT, 60)):
page.insert_text((72, 72 + i * 14), line)
y = 300
for value in values:
page.insert_image(fitz.Rect(72, y, 112, y + 30), stream=_p152_solid_png(value))
y += 40
doc.save(path)
doc.close()
def _build_garbage_image_pdf(path: Path, values: Sequence[int]) -> None:
"""A REAL two-page garbage-text PDF (every page XML-ish — the
phase-133 gate fires, the page-screenshot fallback runs) with
*values* the solid-color embedded images on page 2 — the fallback
span + the embedded span, ONE continuous counter."""
doc = fitz.open()
for p in range(1, 3):
page = doc.new_page()
for i in range(10):
page.insert_text((72, 72 + i * 14), f"<foo:p{p}>{i}</foo:p{p}>")
y = 400
for value in values:
doc[1].insert_image(
fitz.Rect(72, y, 112, y + 30), stream=_p152_solid_png(value)
)
y += 40
doc.save(path)
doc.close()
def _p152_summary_fields(s: ImportSummary) -> tuple:
"""One run's observable outcome (the byte-identity pin's left
side): every count a hook could plausibly move + the changed
paths' RELS (the source names are equal by construction — the two
roots carry the same directory name)."""
return (
s.files, s.added, s.updated, s.unchanged, s.pruned, s.errors,
s.chunks, s.embed_batches, s.summaries, s.summary_errors,
s.images_failed, s.pdfs_screenshotted, s.pdfs_pictorial,
s.pdf_pictorial_pages, s.pdf_images_deduped, s.pdf_images_blank_skipped,
tuple(rel for _, rel in s.changed_paths),
)
def _p152_rows(db, source: str) -> list[tuple]:
"""A source's document rows (the byte-identity pin's right side),
identity-blind: the fields a hook could plausibly move."""
docs = db.scalars(select(Document).where(Document.source == source)).all()
return sorted(
(d.path, d.title, d.content, d.content_hash, d.is_image, d.summary,
d.image_context)
for d in docs
)
def test_on_image_progress_embedded_pass_climbs_one_per_image(
db, tmp_path: Path
) -> None:
"""The phase-152 core pin: a PDF with N DISTINCT embedded images
(one placement each — post-dedupe, post-blank-gate: the work
actually about to happen) fires the hook ONCE per image, 1-based,
with the span's total, at the START of each image's pipeline step:
``(1, N) … (N, N)``. A repeated placement of the first image (the
parser's dedupe, phase 151) adds NO position (the total is the
post-dedupe count), a clean usable text layer burns no fallback or
pictorial span (the embedded pass's totals stand alone), and the
phase-147 fast path on a re-sync (raw bytes unchanged) burns no
vision call and fires the hook never."""
root = tmp_path / "imgprog"
root.mkdir()
values = (90, 130, 170, 210) # 4 distinct images, all under the blank floor
_build_image_pdf(root / "report.pdf", values, repeat_first=True)
n = len(values)
llm = _PageVisionLLM()
llm.settings = _pdf_settings(tmp_path, images=True)
events: list[tuple[int, int]] = []
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
# The PDF + its N image docs; the repeat placement deduped.
assert (summary.files, summary.added) == (1, 1 + n)
assert summary.pdf_images_deduped == 1
assert (summary.images_failed, summary.pdfs_screenshotted,
summary.pdfs_pictorial) == (0, 0, 0)
# The embedded pass is the ONLY vision span: N slow calls.
assert llm.vision_calls == n
# The core pin: once per image, 1-based, the span's total.
assert events == [(i, n) for i in range(1, n + 1)]
# The phase-147 fast path: a re-sync (bytes unchanged) burns no
# vision call and fires the hook never — the events list grows
# not at all.
summary2 = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
assert summary2.unchanged == 1
assert llm.vision_calls == n
assert len(events) == n
finally:
_cleanup_source(db, root.name)
def test_on_image_progress_pictorial_then_embedded_is_one_counter(
db, tmp_path: Path
) -> None:
"""ASSUMPTION A3: the spans of ONE file are ONE continuous counter
— the embedded pass starts where the pictorial pass ended (``base
= images_done_so_far``): the positions climb ``1…C+N`` with NO
restart and never above the combined total (the pictorial span's
total is its candidate-page count, the embedded span's total the
combined count)."""
root = tmp_path / "imgprog-two"
root.mkdir()
values = (90, 130, 170)
_build_pictorial_image_pdf(root / "ikea.pdf", values)
# The span inputs, computed IN the test (no hardcoded lists — the
# phase-140 idiom): the deterministic gate's candidates and the
# post-dedupe/post-gate image count.
pdf = root / "ikea.pdf"
page_texts = extract_pdf_page_texts(pdf)
n_cand = len(pictorial_pages(page_texts, min_chars=100))
images, _ = extract_pdf_images_with_stats(pdf)
n_img = len(images)
assert (n_cand, n_img) == (1, len(values))
llm = _PageVisionLLM()
llm.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
import_extensions="md,txt,pdf",
images=True,
image_dir=str(tmp_path / "images"),
pdf_pictorial=True, # default on — the pass runs on the sparse page
)
events: list[tuple[int, int]] = []
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
assert (summary.files, summary.added) == (1, 1 + n_img)
assert (summary.pdfs_pictorial, summary.pdf_pictorial_pages) == (1, 1)
# The continuous counter: 1…C (total C) then C+1…C+N (total
# C+N) — no restart, never above the combined total.
assert events == (
[(i, n_cand) for i in range(1, n_cand + 1)]
+ [(n_cand + i, n_cand + n_img) for i in range(1, n_img + 1)]
)
finally:
_cleanup_source(db, root.name)
def test_on_image_progress_fallback_span_climbs_into_embedded(
db, tmp_path: Path
) -> None:
"""The owner's stuck-label class: a garbage-text PDF's per-page
screenshot describes (the phase-133 fallback span — the span that
froze the label for minutes) report the rendered-page count
(``1…M`` with total ``M``) and hand their reported count to the
embedded pass, which continues the SAME counter (``M+1`` with
total ``M+N``) — every slow call has a position before it."""
root = tmp_path / "imgprog-fb"
root.mkdir()
_build_garbage_image_pdf(root / "garbage.pdf", (90,))
llm = _PageVisionLLM()
llm.settings = _pdf_settings(tmp_path, images=True)
events: list[tuple[int, int]] = []
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
assert summary.pdfs_screenshotted == 1
assert (summary.files, summary.added, summary.images_failed) == (1, 2, 0)
# 2 fallback pages (total 2) + 1 embedded image continuing at
# 3 (total 3) — ONE counter across the two spans.
assert events == [(1, 2), (2, 2), (3, 3)]
# Every slow call (2 pages + 1 image) was preceded by a
# position — the label never sits frozen on a busy run.
assert llm.vision_calls == 3
finally:
_cleanup_source(db, root.name)
def test_on_image_progress_text_only_pdf_never_fires(
db, tmp_path: Path
) -> None:
"""A clean-text PDF with NO embedded images fires the hook NEVER,
with BOTH default-on toggles: the garbage gate stays closed (no
fallback span), the pictorial pass's deterministic gate finds zero
candidates (a no-op — the clean-text byte-identity class), and the
embedded pass's list is empty. The skipped spans report nothing
and advance the counter not at all (there is no vision call at
all)."""
root = tmp_path / "imgprog-none"
root.mkdir()
_build_image_pdf(root / "clean.pdf", ()) # dense text, zero images
llm = _PageVisionLLM()
llm.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
import_extensions="md,txt,pdf",
images=True,
image_dir=str(tmp_path / "images"),
pdf_pictorial=True, # default on — the pass runs and no-ops
)
events: list[tuple[int, int]] = []
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
assert (summary.files, summary.added) == (1, 1)
assert (summary.pdfs_screenshotted, summary.pdfs_pictorial) == (0, 0)
assert llm.vision_calls == 0 # no span ran — zero slow calls
assert events == []
finally:
_cleanup_source(db, root.name)
def test_on_image_progress_toggle_off_reports_nothing(
db, tmp_path: Path
) -> None:
"""The ``images`` toggle OFF skips the embedded span entirely (the
prune guard keeps any existing image docs) — a PDF that WOULD
carry N embedded images reports nothing (a skipped span advances
the counter not at all) and indexes as text only."""
root = tmp_path / "imgprog-off"
root.mkdir()
_build_image_pdf(root / "off.pdf", (90, 130))
llm = _PageVisionLLM()
llm.settings = _pdf_settings(tmp_path, images=False)
events: list[tuple[int, int]] = []
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
assert (summary.files, summary.added) == (1, 1)
assert llm.vision_calls == 0
assert events == []
finally:
_cleanup_source(db, root.name)
def test_on_image_progress_none_run_is_byte_identical(
db, tmp_path: Path
) -> None:
"""The regression pin: a hook-less run (the CLI/chat default —
``on_image_progress=None``, every call site a guarded no-op) lands
byte-identical to a run WITH the hook: the same documents (path,
title, content, digest, image flag, summary, context) and the same
summary counters. The hook observes; it never changes the run."""
# Both roots carry the SAME directory name — the source (the
# walk's root.name) is the document's provenance and part of the
# stored image context, so the comparison needs it equal.
root_a = tmp_path / "a" / "same_src"
root_a.mkdir(parents=True)
root_b = tmp_path / "b" / "same_src"
root_b.mkdir(parents=True)
values = (90, 130, 170, 210)
_build_image_pdf(root_a / "same.pdf", values)
_build_image_pdf(root_b / "same.pdf", values)
llm_a, llm_b = _PageVisionLLM(), _PageVisionLLM()
llm_a.settings = _pdf_settings(tmp_path, images=True)
llm_b.settings = _pdf_settings(tmp_path, images=True)
events: list[tuple[int, int]] = []
try:
# Run A (hook-less) first, snapshot its rows, clear the shared
# source name, then run B (hooked) — the rows line up 1:1.
summary_a = asyncio.run(import_sources([root_a], llm_a, session=db))
rows_a = _p152_rows(db, "same_src")
_cleanup_source(db, "same_src")
summary_b = asyncio.run(
import_sources(
[root_b], llm_b, session=db,
on_image_progress=lambda pos, total: events.append((pos, total)),
)
)
rows_b = _p152_rows(db, "same_src")
# The hook ran in run B (N image positions, the span's total)
# — and changed nothing observable.
assert events == [(i, len(values)) for i in range(1, len(values) + 1)]
assert _p152_summary_fields(summary_a) == _p152_summary_fields(summary_b)
assert rows_a == rows_b
finally:
_cleanup_source(db, "same_src")
class _InterleavingVisionLLM(_PageVisionLLM):
"""A ``_PageVisionLLM`` variant that records each vision call in
a SHARED event sequence (phase 152, task 04): the importer fires
the ``on_image_progress`` hook BEFORE the
``_index_one_embedded_image`` describe seam, so the sequence reads
hook, vision, hook, vision, … — the interleaving pin (the label
moves at the START of each slow call, never after it)."""
def __init__(self, events: list[tuple]) -> None:
super().__init__()
self.events = events
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
reply = await super().chat(messages, model)
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
self.events.append(("vision", self.vision_calls))
return reply
def test_on_image_progress_hook_fires_before_each_describe_call(
db, tmp_path: Path
) -> None:
"""Phase 152 (task 04): the hook fires BEFORE the corresponding
describe call (the contract: at the START of each slow call, never
after it) — the interleaving pinned by recording events in the
fake: for a PDF with N embedded images the sequence reads
hook(1,N), vision(1), hook(2,N), vision(2), …, hook(N,N),
vision(N) (the fake's vision call IS the describe seam the hook
precedes — the label moves while the describe is about to start,
so a busy run never reads frozen)."""
root = tmp_path / "imgprog-order"
root.mkdir()
values = (90, 130, 170)
_build_image_pdf(root / "order.pdf", values)
n = len(values)
events: list[tuple] = []
llm = _InterleavingVisionLLM(events)
llm.settings = _pdf_settings(tmp_path, images=True)
try:
summary = asyncio.run(
import_sources(
[root], llm, session=db,
on_image_progress=lambda pos, total: events.append(("image", pos, total)),
)
)
assert (summary.files, summary.added) == (1, 1 + n)
assert llm.vision_calls == n # one describe per image
# The interleaving: hook BEFORE each describe call, in order.
expected: list[tuple] = []
for i in range(1, n + 1):
expected.append(("image", i, n))
expected.append(("vision", i))
assert events == expected
finally:
_cleanup_source(db, root.name)
def test_on_image_progress_blank_gate_excludes_skipped_images_from_the_total(
db, tmp_path: Path
) -> None:
"""Phase 152 (task 04, the phase-151 interaction): images skipped
by the dedupe/blank gate are NOT counted in the hook's total —
the total is the work actually about to happen (the pass's
post-dedupe, post-gate list). A PDF with 2 real + 1 near-white
image (250 — the blank gate's class, mean ≥ 235): the gate
(default ON) drops the near-white one BEFORE the pass — the hook
fires for the 2 kept with total 2 (the skipped image burns no
vision call, no position); with the kill switch OFF
(``pdf_image_skip_blank=False`` — the pre-phase-151 behavior) the
near-white image is indexed too — the hook fires for all 3 with
total 3 (the total is the work about to happen under the knob's
state)."""
values_a = (90, 130, 250) # 250 = near-white (the blank gate's class)
values_b = (110, 150, 250) # distinct bytes — a different document set
# Both roots carry the SAME directory name — the source is part of
# the stored image context, so the runs line up 1:1 (the house
# cleanup idiom separates them).
root_a = tmp_path / "a" / "same_src"
root_a.mkdir(parents=True)
root_b = tmp_path / "b" / "same_src"
root_b.mkdir(parents=True)
_build_image_pdf(root_a / "blank.pdf", values_a)
_build_image_pdf(root_b / "blank.pdf", values_b)
llm_a = _PageVisionLLM()
llm_a.settings = _pdf_settings(tmp_path, images=True) # skip_blank default ON
events_a: list[tuple[int, int]] = []
try:
summary_a = asyncio.run(
import_sources(
[root_a], llm_a, session=db,
on_image_progress=lambda pos, total: events_a.append((pos, total)),
)
)
# The blank gate dropped the near-white image BEFORE the
# pass — the total is the 2 actually about to happen.
assert summary_a.pdf_images_blank_skipped == 1
assert (summary_a.files, summary_a.added) == (1, 3)
assert llm_a.vision_calls == 2
assert events_a == [(1, 2), (2, 2)]
# The knob OFF: the near-white image is indexed too — the
# total is all 3 (the pre-phase behavior, the work about to
# happen under the knob's state). Fresh source (the house
# cleanup idiom — run B's rows would otherwise be updates of
# run A's rows, same synthetic image paths).
_cleanup_source(db, "same_src")
llm_b = _PageVisionLLM()
llm_b.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
import_extensions="md,txt,pdf",
images=True,
image_dir=str(tmp_path / "images"),
pdf_pictorial=False,
pdf_image_skip_blank=False,
)
events_b: list[tuple[int, int]] = []
summary_b = asyncio.run(
import_sources(
[root_b], llm_b, session=db,
on_image_progress=lambda pos, total: events_b.append((pos, total)),
)
)
assert summary_b.pdf_images_blank_skipped == 0
assert (summary_b.files, summary_b.added) == (1, 4)
assert llm_b.vision_calls == 3
assert events_b == [(1, 3), (2, 3), (3, 3)]
finally:
_cleanup_source(db, "same_src")
+567 -15
View File
@@ -110,6 +110,56 @@ poll tree's new 5th branch (after failed, before the upload branches
— the 7-branch tree is now 8), the neutral ``applySyncCancelled``
(no modal, no banner, no error styling), and the
``initSyncButton`` re-attach of a ``cancelled`` run.
Phase 152 (task 01): the status contract gains the in-flight file's
VISION-image position — ``images_done`` (the 1-based index of the
image about to be described) + ``images_total`` (the file's total
images for this run — the three PDF vision spans share ONE continuous
counter per file, the importer's ``on_image_progress`` hook that
task 02/03 wire). The idle dict pins them 0/0; the per-file reset is
the pairing rule, extended — a ``progress`` call names the file AND
clears the image position, pinned through the runner's own ``_hook``
closure (the two-file fake below: the position fired through the
runner's own ``_image_hook`` closure while file A is in flight is
0/0 when file B is named — a file with no images must not inherit
the previous PDF's position); the clear rule mirrors the chunk
position — 0/0 at the overview/summaries transitions and in EVERY
terminal (success, failed, and cancelled — each pinned with a live
position in flight; task 04 moves every live position below onto the
runner's OWN closures, the phase-64/129 idiom), while the FILE
counts keep their final values (the convention split: a
sub-position of a file that's done clears with the file).
Phase 152 (task 03): the runner's THIRD paired closure —
``_image_hook`` (beside ``_hook`` and ``_chunk_hook``) — lands the
importer's ``on_image_progress`` pairs on the status, and the import
call now passes ``on_image_progress=_image_hook`` (the wiring pin:
the fakes receive the kwarg and record the live closure). The
frontend gains the "· image n/m" label PART: the running entry
destructures ``image = ""`` and the writer call becomes
``setSyncLabel(main, count, chunk, true, image)`` — the image part
rides ONLY the composed full string (box + title + running
aria-label — no new button span; the running button is the
spinner), the chunk part keeps its own span, and the bare builders
+ non-running calls pass nothing (both parts default "").
Phase 152 (task 04): the position pins move onto the runner's OWN
closures (the phase-64/129 idiom — the fakes fire the image pairs
through the ``on_image_progress`` kwarg, the ``_image_hook`` under
test, instead of standing in on the status object): a mid-run pair
(3/15, then 15/15) climbs on the status BETWEEN ticks while
``current_file`` and the chunk position keep their own values (the
THREE positions are independent sub-positions of the file), the
per-file reset, the overview/summaries transition clears, and the
FAILED + CANCELLED terminals all fire their live positions through
the closure too. The frontend source pins the ``fmtSyncPhaseLabel``
image part (present iff ``images_total > 0`` — the exact guard +
template), the ``overview`` / ``summaries`` branches' ``image: ""``
(unchanged labels), and ``setSyncLabel``'s composed ``full`` string
byte-for-byte (a status carrying both positions reads "… (2540/2540)
· chunk 125/126 · image 3/15" on the box + title + running
aria-label — the runtime string is behaviorally pinned in
tests/unit/test_frontend_sync_upload.py's node check).
"""
from __future__ import annotations
@@ -393,13 +443,16 @@ def test_sources_js_running_state_is_never_stale() -> None:
Phase 128 (task 01): the entry hands the label PARTS (the
phase-aware split — sync + status → fmtSyncPhaseLabel, the
fall-through → the byte-identical fmtSyncLabel) to the SINGLE
writer, setSyncLabel(main, count, chunk — phase 129: the chunk
part destructures with a "" default), which lands the main part,
writer, setSyncLabel(main, count, chunk, running, image — phase
129: the chunk part destructures with a "" default, phase 152:
the image part does too), which lands the main part,
the pinned count part (hidden when absent), the pinned chunk part
(phase 129 — hidden when absent), and the FULL untruncated label
on the button title + #sync-progress (A4) — the direct span /
title / announcer writes are GONE from the entry (they live in
setSyncLabel + setSyncProgress). Phase 149 (task 01): the writer call passes
on the button title + #sync-progress (A4 — phase 152: with the
image part appended after the chunk part when a PDF's vision span
is in flight; the image part gets no span of its own) — the
direct span / title / announcer writes are GONE from the entry
(they live in setSyncLabel + setSyncProgress). Phase 149 (task 01): the writer call passes
`running = true` (the detail box reveals with the full label),
and the button's accessible name rides in the writer too (the
full live label — the icon-only running button keeps a name);
@@ -414,8 +467,10 @@ def test_sources_js_running_state_is_never_stale() -> None:
assert "fmtSyncLabel(kind, currentFile, done, total)" in body
# …and the single writer lands the parts (A4 in it — the title is
# set for EVERY running label: the file-only rule is adjusted
# away) with the phase-149 `running` flag (the box reveals).
assert "setSyncLabel(main, count, chunk, true)" in body
# away) with the phase-149 `running` flag (the box reveals) +
# the phase-152 image part (rides only the composed full string —
# no span).
assert "setSyncLabel(main, count, chunk, true, image)" in body
assert body.count("setSyncLabel(") == 1, "the entry writes ONLY through the parts writer"
assert "syncBtn.title =" not in body, ("the title write lives in setSyncLabel (A4)")
assert "syncLabel.textContent" not in body, ("no direct span write — the parts")
@@ -433,6 +488,70 @@ def test_sources_js_running_state_is_never_stale() -> None:
)
def test_fmt_sync_phase_label_image_part_is_pinned_source_level() -> None:
"""Phase 152 (task 04): the frontend SOURCE pins the "· image
n/m" label part (the browser behavior is pinned in
tests/unit/test_frontend_sync_upload.py's node check — the REAL
extracted functions run against stub refs; this pin keeps the
template itself from drifting without a browser):
* the import fall-through is the ONLY reader of the status's
image fields, guarded on ``images_total > 0`` (the part is
present IFF ``images_total > 0`` — 0/0 hides it, the exact
guard + template + empty false-branch);
* the ``overview`` / ``summaries`` branches return ``image: ""``
(the position belongs to the import phase only — those labels
stay unchanged by the phase);
* the running entry destructures the part with a `""` default and
hands it to the single writer as the FIFTH argument (after the
`running` flag);
* ``setSyncLabel``'s composed ``full`` string appends the image
part LAST (after the chunk part) — so a status carrying both
positions reads exactly "Syncing… <file> (2540/2540) · chunk
125/126 · image 3/15" on the box, the title, and the running
aria-label (byte-for-byte; the docstring's worked example
carries the shape).
"""
js = _text(SOURCES_JS)
builder = _body(js, "fmtSyncPhaseLabel")
# The import fall-through: present IFF images_total > 0 — the
# exact guard + template (the phase-129 chunk part's sibling,
# same shape).
assert (
"status.images_total > 0\n"
" ? `· image ${status.images_done}/${status.images_total}`\n"
" : \"\";"
) in builder
assert "return { main, count, chunk, image };" in builder
# The overview branch: the unchanged label + the part absent.
assert (
'return { main: "Writing KB overview…", count: "", chunk: "", image: "" };'
) in builder
# The summaries branch: the unchanged label + the part absent
# (the phase's own (n/m) is its position).
assert "image: \"\", // the image position belongs to the import phase only" in builder
assert "`Summarizing folders…${folder}`" in builder
# The running entry: the part destructures with a `""` default
# and rides the writer call as the fifth argument.
entry = _body(js, "enterSyncRunningState")
assert 'const { main, count, chunk = "", image = "" }' in entry
assert "setSyncLabel(main, count, chunk, true, image)" in entry
# The writer: the composed full string — the image part LAST,
# after the chunk part (the "… · chunk 125/126 · image 3/15"
# shape the docstring's worked example carries).
writer = _body(js, "setSyncLabel")
assert (
"const full =\n"
" mainText +\n"
" (countText ? ` ${countText}` : \"\") +\n"
" (chunkText ? ` ${chunkText}` : \"\") +\n"
" (imageText ? ` ${imageText}` : \"\");"
) in writer
assert "· chunk 125/126 · image 3/15" in js, (
"the worked example (both positions at once) is documented"
)
def test_sources_js_terminal_states() -> None:
"""Terminal rendering: success → enabled + 'Synced HH:MM' (local
time of finished_at); failed → enabled + retry-ready 'Sync sources'
@@ -927,7 +1046,20 @@ class _GatedImport:
``_chunk_hook`` closure: pair one lands before the main park
(readable mid-run), and every later pair lands after the main
park + its own park (``chunk_release``), so each position is
readable on the status between ticks."""
readable on the status between ticks.
Phase 152 (task 03): the runner now ALWAYS passes the
image-position closure (task 03's wiring) — recorded for the
wiring pins.
Phase 152 (task 04): optionally ALSO fires the in-flight file's
image-position pairs (``image_pairs``) through the runner's OWN
``_image_hook`` closure (pair one before the main park —
readable mid-run; every later pair after the main park + its own
park ``image_release`` — the phase-129 chunk idiom, mirrored for
the vision-image position). With no pairs the fake never fires
it (the status image fields stay 0/0, like the chunk fields of a
no-batch import)."""
def __init__(
self,
@@ -937,6 +1069,8 @@ class _GatedImport:
fail: BaseException | None = None,
chunk_pairs: Sequence[tuple[int, int]] = (),
chunk_release: threading.Event | None = None,
image_pairs: Sequence[tuple[int, int]] = (), # phase 152 (task 04)
image_release: threading.Event | None = None, # phase 152 (task 04)
) -> None:
self.summary = summary
self.started = started
@@ -947,6 +1081,12 @@ class _GatedImport:
self.chunk_pairs = list(chunk_pairs)
self.chunk_release = chunk_release
self.chunk_hook_calls: list[tuple[int, int]] = []
# Phase 152 (task 03): the image-position hook the runner
# passes (recorded for the wiring pins).
self.image_hook: Callable[[int, int], None] | None = None
self.image_pairs = list(image_pairs)
self.image_release = image_release
self.image_hook_calls: list[tuple[int, int]] = []
async def __call__(
self,
@@ -958,6 +1098,7 @@ class _GatedImport:
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
@@ -966,6 +1107,16 @@ class _GatedImport:
if progress is not None:
progress("repo", "notes/deep.md", 1, 3)
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
# Phase 152 (task 03): the runner now ALWAYS passes the
# image-position closure (task 03's wiring) — recorded for the
# wiring pins.
self.image_hook = on_image_progress
# Phase 152 (task 04): image pair one lands BEFORE the main
# park (readable mid-run — the phase-129 chunk-pair-one
# idiom, mirrored for the vision-image position).
if on_image_progress is not None and self.image_pairs:
on_image_progress(*self.image_pairs[0])
self.image_hook_calls.append(self.image_pairs[0])
if on_chunk_progress is not None and self.chunk_pairs:
# Phase 129 (task 03): the file's chunk-position pairs,
# fired through the runner's OWN hook closure (the closure
@@ -986,6 +1137,17 @@ class _GatedImport:
else:
self.started.set()
await asyncio.to_thread(self.release.wait) # park without freezing the worker loop
# Phase 152 (task 04): every later image pair lands after the
# main park + its own park (``image_release``), so each
# position is readable on the status between ticks. A wiring
# regression to None skips this branch and the pair tests
# time out.
if on_image_progress is not None:
for pair in self.image_pairs[1:]:
on_image_progress(*pair)
self.image_hook_calls.append(pair)
if self.image_release is not None:
await asyncio.to_thread(self.image_release.wait)
if self.fail is not None:
raise self.fail
return self.summary
@@ -1088,10 +1250,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
fresh_sync_status: None,
) -> None:
"""Idle: the three phase-64 progress keys ride along as null/0/0,
the two phase-129 chunk keys as 0/0, the four phase-98 phase keys
as null/0/0/0, and EVERY pre-existing key is unchanged — the full
response dict is pinned, so the current UI and every existing
consumer keep working."""
the two phase-129 chunk keys as 0/0, the two phase-152 image keys
as 0/0, the four phase-98 phase keys as null/0/0/0, and EVERY
pre-existing key is unchanged — the full response dict is pinned,
so the current UI and every existing consumer keep working."""
assert sync_api.sync_status() == {
"state": "idle",
"started_at": None,
@@ -1105,6 +1267,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
# (the dataclass defaults, no special-casing).
"current_chunk": 0,
"chunks_total": 0,
# Phase 152 (task 01): the image-position keys — 0/0 idle
# (the dataclass defaults, no special-casing).
"images_done": 0,
"images_total": 0,
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
"phase": None,
"current_summary": None,
@@ -1145,13 +1311,19 @@ def test_mid_run_status_reports_current_file(
assert s["files_done"] == 0 and s["files_total"] == 0
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
# Phase 152 (task 01): the image position is 0/0 in the
# prelude too (no vision span yet).
assert s["images_done"] == 0 and s["images_total"] == 0
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Import phase: the hook's file is live on the status, and the
# phase machine names the phase (the summary span has not
# started — its counters are still 0/0); no embed batch has
# fired the chunk hook (this canned import passes no pairs),
# so the chunk position stays 0/0.
# so the chunk position stays 0/0; the image position stays
# 0/0 too (this canned import fires no image hook — task 03
# wires the runner's image closure, the importer's span is
# what would feed it).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
@@ -1161,6 +1333,7 @@ def test_mid_run_status_reports_current_file(
assert s["current_summary"] is None
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 0 and s["images_total"] == 0
fake_import.release.set()
finally:
clone.release.set()
@@ -1178,7 +1351,8 @@ def test_mid_run_status_reports_current_file(
# folder step never fired the summary hook, so the summary counts
# stay 0/0 (the keep-final-counts convention keeps what fired);
# the chunk position clears with the file (phase 129 — it is a
# sub-position of a done file, never a keep-final counter).
# sub-position of a done file, never a keep-final counter), and
# so does the image position (phase 152 — same convention).
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
@@ -1187,6 +1361,7 @@ def test_mid_run_status_reports_current_file(
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["error"] is None
assert s["started_at"] is not None and s["finished_at"] is not None
@@ -1215,6 +1390,7 @@ def test_failed_terminal_clears_current_file_keeps_counts(
"failed: connection refused"
),
chunk_pairs=[(1, 38)], # phase 129: the live pair before the raise
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair before the raise
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
@@ -1231,6 +1407,13 @@ def test_failed_terminal_clears_current_file_keeps_counts(
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 1 and s["chunks_total"] == 38
# Phase 152 (task 04): a live image position in flight when
# the run dies — fired through the runner's OWN _image_hook
# closure (image_pairs[0] — the importer's span would do the
# same); the failed terminal must clear it (the
# sub-position of an abandoned file, like the chunk
# position).
assert s["images_done"] == 3 and s["images_total"] == 15
fake_import.release.set()
finally:
clone.release.set()
@@ -1238,8 +1421,9 @@ def test_failed_terminal_clears_current_file_keeps_counts(
thread.join(10.0)
assert not thread.is_alive()
assert errors == []
# Wiring: the pair fired through the runner's own closure.
# Wiring: the pairs fired through the runner's OWN closure.
assert fake_import.chunk_hook_calls == [(1, 38)]
assert fake_import.image_hook_calls == [(3, 15)]
s = sync_api.sync_status()
assert s["state"] == "failed"
assert s["current_file"] is None # cleared in the terminal state
@@ -1251,6 +1435,11 @@ def test_failed_terminal_clears_current_file_keeps_counts(
# terminal too — the convention split, pinned (the FILE counts
# above keep their final values, the sub-position does not).
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
# Phase 152 (task 04): the image position clears in the FAILED
# terminal too — the convention split, pinned (the live 3/15 fired
# through the closure mid-run is gone, the FILE counts keep their
# final values).
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["detail"] == {}
error = s["error"] or ""
assert "*****@aipi.example.com" in error # credentials masked
@@ -1468,6 +1657,357 @@ def test_chunk_position_clears_at_phase_transitions_and_terminal(
assert s["files_done"] == 1 and s["files_total"] == 3
# ---------- phase 152 (task 01 + task 04): the in-flight file's -----
# ---------- vision-image position (images_done / images_total) -------
#
# Same seams as the phase-129 block above. Task 04 pins every live
# position through the runner's OWN closures (the phase-64/129 idiom):
# the fakes fire the image pairs through the ``on_image_progress``
# kwarg — the ``_image_hook`` under test (task 03's wiring: the
# importer's span's exact path) — and drive the per-file ``_hook``
# reset (through the ``progress`` kwarg the fake import receives) and
# the transition/terminal clears.
def test_mid_run_status_reports_image_position_between_ticks(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""Phase 152 (task 04): the status carries the in-flight file's
VISION-image position — the pairs fire through the runner's OWN
``_image_hook`` closure (task 03's wiring — the importer's
``on_image_progress`` span's exact path), and the position CLIMBS
between ticks (3/15 → 15/15 — the PDF's vision span moving, the
owner's stuck label, now moving) while ``current_file`` and the
chunk position (14/38, a concurrent embed batch) keep their own
values — the THREE positions are independent sub-positions of the
file; the prelude reports 0/0, and the success terminal clears
BOTH sub-positions (image + chunk) to 0/0 while the FILE counts
keep their final values (the convention split)."""
clone = _GatedClone(threading.Event(), threading.Event())
image_release = threading.Event()
fake_import = _GatedImport(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
threading.Event(),
chunk_pairs=[(14, 38)], # a concurrent embed batch (independent)
image_pairs=[(3, 15), (15, 15)], # two describe calls of a 15-image span
image_release=image_release,
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
# Prelude: every position 0/0 (no file, no batch, no image
# yet — alongside the bare-label pins of the phase-64 test).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["current_file"] is None
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 0 and s["images_total"] == 0
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Import phase: the file is live (the progress hook's call),
# the chunk position 14/38 (the embed batch), and the image
# position 3/15 (the vision span) — the THREE positions are
# independent sub-positions of the same file.
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 14 and s["chunks_total"] == 38
assert s["images_done"] == 3 and s["images_total"] == 15
# Release the main park: the import fires image pair two
# (15/15 — the span's last describe call) and parks on
# image_release.
fake_import.release.set()
_poll_until(
lambda: sync_api.sync_status()["images_done"] == 15,
what="the second image pair on the status",
)
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
# The image position CLIMBED (3 → 15 — monotonic, no
# backwards jump, never above the total) ... the FILE and
# chunk positions keep their own values (independent).
assert s["images_done"] == 15 and s["images_total"] == 15
assert s["current_file"] == "repo/notes/deep.md"
assert s["files_done"] == 1 and s["files_total"] == 3
assert s["current_chunk"] == 14 and s["chunks_total"] == 38
image_release.set()
finally:
clone.release.set()
fake_import.release.set()
image_release.set()
thread.join(10.0)
assert not thread.is_alive()
assert errors == [], f"the runner raised: {errors!r}"
# Wiring: every pair fired through the runner's OWN closure (the
# status reads above came through it; a wiring regression to
# None would have skipped the firing branch and timed out).
assert fake_import.chunk_hook_calls == [(14, 38)]
assert fake_import.image_hook_calls == [(3, 15), (15, 15)]
# Success terminal: BOTH sub-positions clear (0/0) ...
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
assert s["images_done"] == 0 and s["images_total"] == 0
# ... while the FILE counts keep their final values (the
# convention split, pinned alongside the phase-64 terminal pin).
assert s["current_file"] is None
assert s["files_done"] == 1 and s["files_total"] == 3
class _TwoFileImport:
"""The phase-152 (task 01 + task 04) fake: fires the runner's
OWN progress hook for file A, then file A's image position through
the runner's OWN ``on_image_progress`` closure (task 03's wiring —
what the importer's span would do), parks (the test reads the
live position on the status), fires the progress hook again for
file B (the runner's ``_hook`` under test must reset the image
position to 0/0 — the pairing rule, extended), parks again, and
completes."""
def __init__(self, summary: ImportSummary) -> None:
self.summary = summary
self.parked_a = threading.Event() # parked after file A's progress
self.release_a = threading.Event() # test releases → file B's progress
self.parked_b = threading.Event() # parked after file B's progress
self.release_b = threading.Event()
self.hook_calls: list[tuple[str, str, int, int]] = []
self.image_hook_calls: list[tuple[int, int]] = [] # phase 152 (task 04)
async def __call__(
self,
sources: list[Path],
llm: object,
*,
prune: bool = False,
limit: int | None = None,
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
) -> ImportSummary:
if progress is not None:
progress("repo", "docs/a.pdf", 1, 2)
self.hook_calls.append(("repo", "docs/a.pdf", 1, 2))
# Phase 152 (task 04): file A's vision span is in flight —
# its image position (3/15) fires through the runner's OWN
# on_image_progress closure (task 03's wiring), then the fake
# parks.
if on_image_progress is not None:
on_image_progress(3, 15)
self.image_hook_calls.append((3, 15))
self.parked_a.set()
await asyncio.to_thread(self.release_a.wait)
if progress is not None:
# The SECOND progress call names file B — the closure under
# test must reset the image position the closure fired
# while file A was in flight (a file with no images must
# not inherit the previous PDF's position).
progress("repo", "docs/b.md", 2, 2)
self.hook_calls.append(("repo", "docs/b.md", 2, 2))
self.parked_b.set()
await asyncio.to_thread(self.release_b.wait)
return self.summary
def test_image_position_resets_per_file(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""Phase 152 (task 01 + task 04): the pairing rule, extended — a
``progress`` call names the file AND clears the image position;
every image position until the next ``progress`` call belongs to
it. File A's live position (3/15, fired through the runner's OWN
``_image_hook`` closure while parked on file A — task 04: the
phase-64/129 idiom, the importer's image hook's exact path) is
0/0 the moment the runner's own ``_hook`` names file B: a file
with no images must not inherit the previous PDF's position
(unlike the chunk position, not every file embeds chunks). The
success terminal keeps the position at 0/0 and the FILE counts
keep their final values (the convention split)."""
clone = _GatedClone(threading.Event(), threading.Event())
fake_import = _TwoFileImport(ImportSummary(files=2, added=1, updated=1))
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
clone.release.set()
assert fake_import.parked_a.wait(5.0), "the run never parked on file A"
# File A is named by the runner's own _hook (a wiring
# regression to None would leave current_file null here).
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/docs/a.pdf"
assert s["files_done"] == 1 and s["files_total"] == 2
# File A's vision span is in flight: its image position is
# live on the status — fired through the runner's OWN
# _image_hook closure (the fake fired it before parking;
# a wiring regression to None would leave it 0/0 here).
assert s["images_done"] == 3 and s["images_total"] == 15
# Release file A: the runner's _hook names file B — the image
# position must reset to 0/0 in the same call (the pairing
# rule, extended; the closure under test, driven through the
# fake's progress kwarg).
fake_import.release_a.set()
assert fake_import.parked_b.wait(5.0), "the run never parked on file B"
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "import"
assert s["current_file"] == "repo/docs/b.md"
assert s["files_done"] == 2 and s["files_total"] == 2
# RESET per file: file B (no images) does not inherit file
# A's 3/15.
assert s["images_done"] == 0 and s["images_total"] == 0
fake_import.release_b.set()
finally:
clone.release.set()
fake_import.release_a.set()
fake_import.release_b.set()
thread.join(10.0)
assert not thread.is_alive()
assert errors == [], f"the runner raised: {errors!r}"
# Wiring: BOTH progress calls went through the runner's own
# closure (the reset above was its doing), and the image pair
# went through the runner's own _image_hook closure.
assert fake_import.hook_calls == [
("repo", "docs/a.pdf", 1, 2),
("repo", "docs/b.md", 2, 2),
]
assert fake_import.image_hook_calls == [(3, 15)]
# Success terminal: the image position stays 0/0 (cleared in the
# terminal too — every terminal clears it) and the FILE counts
# keep their final values (the convention split).
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
assert s["current_file"] is None
assert s["files_done"] == 2 and s["files_total"] == 2
assert s["images_done"] == 0 and s["images_total"] == 0
def test_image_position_clears_at_phase_transitions_and_terminal(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""Phase 152 (task 01 + task 04): the image position is CLEARED
(0/0) at every point the import phase ends. This run's summary
changes the KB, so BOTH post-import transitions run and are
pinned: the import→``"overview"`` transition and the
import→``"summaries"`` transition (the changed-KB branch) each
clear the live position (3/15, fired mid-import through the
runner's OWN ``_image_hook`` closure — task 04: the
phase-64/129 idiom, the importer's span's exact path) to 0/0,
and the success terminal keeps it 0/0 — through the whole span
the FILE counts keep their final values (the convention split:
the file counts survive for the result line's context, the image
position is a sub-position of a file that's done)."""
clone = _GatedClone(threading.Event(), threading.Event())
import_release = threading.Event()
overview_started = threading.Event()
overview_release = threading.Event()
folders_started = threading.Event()
folders_release = threading.Event()
fake_import = _GatedImport(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
import_release,
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair to clear
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
# Park the post-import steps (re-monkeypatched over the seams
# helper's no-ops): the test reads the status DURING the
# "overview" phase and DURING the "summaries" phase — the two
# import-phase-ending transitions the task pins (the same shape
# as the phase-129 chunk-position transition test above).
async def gated_overview(llm: object, session: object = None) -> bool:
overview_started.set()
await asyncio.to_thread(overview_release.wait)
return True
async def gated_folders(
db: object,
llm: object,
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
folders_started.set()
await asyncio.to_thread(folders_release.wait)
return {"generated": 0, "failed": 0, "pruned": 0}
monkeypatch.setattr(sync_api, "regenerate_overview", gated_overview)
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Mid-import: the file is live; its image position went live
# too — fired through the runner's OWN _image_hook closure
# (image_pairs[0] — the importer's span would do the same).
s = sync_api.sync_status()
assert s["phase"] == "import"
assert s["images_done"] == 3 and s["images_total"] == 15
# Release the import: it returns (added+updated > 0) → the
# phase machine moves to "overview" and the position clears.
import_release.set()
assert overview_started.wait(5.0), "the run never reached the overview"
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "overview"
# CLEARED at the import→overview transition (0/0) ... the file
# counts KEEP their final values (the convention split).
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["files_done"] == 1 and s["files_total"] == 3
overview_release.set()
# ... and the changed-KB branch moves to "summaries" (the
# second import-phase-ending transition — cleared there too,
# idempotently) before parking.
assert folders_started.wait(5.0), "the run never reached the summaries"
s = sync_api.sync_status()
assert s["state"] == "running"
assert s["phase"] == "summaries"
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["files_done"] == 1 and s["files_total"] == 3
folders_release.set()
finally:
clone.release.set()
import_release.set()
overview_release.set()
folders_release.set()
thread.join(10.0)
assert not thread.is_alive()
assert errors == [], f"the runner raised: {errors!r}"
assert fake_import.image_hook_calls == [(3, 15)]
# Success terminal: still 0/0 (the terminal clears too — same
# spots as current_file), file counts kept.
s = sync_api.sync_status()
assert s["state"] == "success"
assert s["phase"] is None
assert s["current_file"] is None
assert s["images_done"] == 0 and s["images_total"] == 0
assert s["files_done"] == 1 and s["files_total"] == 3
# ---------- phase 134 (task 01): the cooperative cancel (5th state) ----------
#
# The cancel runs are driven through the REAL ``start_sync`` on a
@@ -1629,11 +2169,17 @@ def test_cancel_endpoint_stops_the_in_flight_run(
ImportSummary(files=3, added=1, updated=1, unchanged=1),
threading.Event(),
threading.Event(),
image_pairs=[(3, 15)], # phase 152 (task 04): the live pair before the cancel
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, _fast_clone)
def act(loop: asyncio.AbstractEventLoop) -> dict[str, str]:
assert fake_import.started.wait(5.0), "the run never reached the import"
# Phase 152 (task 04): the live image position in flight when
# the cancel converts was fired through the runner's OWN
# _image_hook closure (image_pairs[0] — the fake fired it
# before parking, like the importer's span would); the
# cancelled terminal must clear it.
return loop.run_until_complete(sync_api.cancel_sync())
try:
@@ -1665,6 +2211,11 @@ def test_cancel_endpoint_stops_the_in_flight_run(
assert s["files_done"] == 1 and s["files_total"] == 3 # final counts kept
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
# Phase 152 (task 04): the image position clears in the CANCELLED
# terminal too — every terminal clears it (the live 3/15 fired
# through the closure in flight is gone; the FILE counts above
# keep their final values).
assert s["images_done"] == 0 and s["images_total"] == 0
# PLAN §9: the ample-logging line for the converted cancel.
assert any(
r.getMessage()
@@ -1879,6 +2430,7 @@ def test_event_set_during_import_cancels_before_the_lite_spans(
session: object = None,
progress: Callable[[str, str, int, int], None] | None = None,
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
on_image_progress: Callable[[int, int], None] | None = None, # phase 152
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106