ducoterra
2d0f608246
All green. Final report:
**Phase 129 final verification pass — 100% green** (previous work intact; this pass re-verified everything and fixed two pre-existing README inconsistencies)
- Verified on-spec: `LLMClient.embed` per-batch `on_progress` hook; importer `on_chunk_progress` plumbing (text/image/PDF branches; summary embed excluded); `SyncStatus.current_chunk/chunks_total` reset+clear contract; UI `.sync-label-chunk` pinned part with untruncated `title`/`#sync-result`
- `uv run pytest --cov=app -q`: exit 0, ~3041 passed, 0 failed, `app/` coverage **99%** (>90% gate)
- `uv run pytest tests/e2e/test_sync_chunk_progress.py -v --no-cov` (isolated, DB up): **2 passed** (endpoint chunk climb with true `chunk_document` total + UI `· chunk n/m` part)
- `uv run ruff check .`: all checks passed · `uv run pyright`: 0 errors, 0 warnings
- Completion criteria: all 5 ✅ (incl. no behavior change in completed phases — full suite green, CLI/chat no-op paths byte-identical)
- Docs: README.md fixed 2 pre-existing inconsistencies (upload section claimed "unpacks + scans immediately" — phase 90 made it unpack+register only, scan deferred to Sync; troubleshooting claimed "120s client timeout" — actual guard is the 300 s pre-token `TURN_TIMEOUT_MS`). Phase-129 label behavior already documented. AGENTS.md re-verified consistent — no changes
- Removed stray untracked `NOTES.md` (8-byte artifact); no live-infra changes; no commits — all changes left in working tree for the harness
- Next pending phase: `130_chat_answer_clusters`
1336 lines
59 KiB
Python
1336 lines
59 KiB
Python
"""Unit: the admin "Sync sources" button contract (phase 32, task 02).
|
|
|
|
The browser behavior is E2E-covered (tests/e2e/test_sync_button.py,
|
|
task 03); here we pin the source-level wiring — the anonymous-safe
|
|
ship-hidden button markup, the Sources-page admin reveal on the SAME
|
|
cached whoami (no extra fetch), the sync state machine that now lives
|
|
in sources.js (owner rework 2026-08-28: the button is Sources-page
|
|
only, so the page owns the machine it drives — 2 s poll, 202 start
|
|
/ 409 adoption / 403 hide, terminal labels, the "bor:sync-status"
|
|
event with the status object as detail, the single-poll-loop guard, no
|
|
client-side hard timeout), the page's #sync-result line +
|
|
#sync-error-banner, the lazily created error modal (phase 41), and the
|
|
§7.4 never-stale CSS (spin + reduced-motion opt-out,
|
|
disabled state, 44px floor, contrast pair) — so a silent regression is
|
|
caught without a browser.
|
|
|
|
Phase 64 (task 02): unit coverage of the ``app/api/sync.py`` status
|
|
contract — the idle response dict is pinned in full (every pre-existing
|
|
key unchanged, plus ``current_file``/``files_done``/``files_total`` as
|
|
null/0/0); mid-run the status reports the file the (mock) import is
|
|
processing, through the runner's own hook closure (no file yet during
|
|
the clone/pull phase — A4); the terminal states clear
|
|
``current_file`` while keeping the run's final counts.
|
|
|
|
Phase 98 (task 01): the status's phase machine joins the pins — the
|
|
four new keys (``phase`` / ``current_summary`` / ``summaries_done`` /
|
|
``summaries_total``) ride along as null/0/0/0 idle, the prelude keeps
|
|
``phase: null``, the import reports ``phase: "import"``, and the
|
|
terminal states clear ``phase`` + ``current_summary`` while keeping
|
|
the run's final summary counts.
|
|
|
|
Phase 98 (task 02): the running-state title rule is adjusted — the
|
|
FULL untruncated LABEL (the file label, "Writing KB overview…", or
|
|
"Summarizing folders… <folder> (n/m)") rides the button title for
|
|
EVERY running label; the phase-64 "title only when there is a current
|
|
file" (remove-when-null) rule is gone.
|
|
|
|
Phase 128 (task 01): the label becomes PARTS — #sync-label ships a
|
|
main span (the idle text pre-JS) + a hidden count span (the pinned
|
|
"(n/m)"), the builders return { main, count }, and the single writer
|
|
setSyncLabel(main, count, chunk — the chunk part is phase 129) lands
|
|
the parts + the untruncated full label (main + " " + count +
|
|
" " + chunk) on the title + #sync-result; the running-state /
|
|
settle pins move onto that shape (their labels stay byte-identical
|
|
as full strings). Phase 129 (task 04): #sync-label ships a THIRD
|
|
hidden span (the pinned "· chunk n/m" — the in-flight file's
|
|
embed-batch position), rendered only while a file's embed batch is in
|
|
flight (the import phase after the first batch — hidden in every
|
|
other state); the markup pin below pins the exact three-part ship
|
|
shape.
|
|
|
|
Phase 129 (task 03): the status contract gains the in-flight file's
|
|
EMBED-chunk position — ``current_chunk`` (the 1-based first chunk of
|
|
the in-flight token-budgeted batch) + ``chunks_total`` (the file's
|
|
total content chunks): the idle dict pins them 0/0, the running
|
|
dict's pins gain them, and the new mid-run pair test reads the
|
|
positions between ticks (1/38 → 13/38) through the runner's own
|
|
``_chunk_hook`` closure; the convention split is pinned — the chunk
|
|
position clears at every point the import phase ends (the
|
|
overview/summaries transitions + both terminals) while the FILE
|
|
counts keep their final values.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import re
|
|
import threading
|
|
import time
|
|
from collections.abc import Callable, Iterator, Sequence
|
|
from datetime import datetime
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
from app.api import sync as sync_api
|
|
from app.config import Settings
|
|
from app.models import GitSource
|
|
from app.rag.importer import ImportSummary
|
|
from app.rag.llm import EmbeddingError
|
|
|
|
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
|
|
ASSETS = FRONTEND / "assets"
|
|
|
|
HEADER_JS = ASSETS / "header.js"
|
|
SOURCES_JS = ASSETS / "sources.js"
|
|
STYLES_CSS = ASSETS / "styles.css"
|
|
# Phase 76 (task 02): sources.html is folded into the ONE-document shell
|
|
# — the sync markup now lives in the RAG view section of index.html.
|
|
SHELL_HTML = FRONTEND / "index.html"
|
|
|
|
|
|
def _text(path: Path) -> str:
|
|
assert path.is_file(), f"missing frontend file: {path}"
|
|
return path.read_text(encoding="utf-8")
|
|
|
|
|
|
def _rag_view(html: str) -> str:
|
|
"""The RAG view section of the shell (view-scoped page-sub scope —
|
|
the shell carries one .page-sub per view, so a whole-file match
|
|
would hit the earlier views first)."""
|
|
i = html.find('<section class="view" id="view-rag"')
|
|
assert i != -1, "the RAG view section must be in the shell"
|
|
j = html.find('<section class="view" id="view-git-sources"', i)
|
|
assert j != -1, "the Sources view section must follow the RAG view"
|
|
return html[i:j]
|
|
|
|
|
|
def _body(js: str, fn_name: str) -> str:
|
|
"""The source of the first `function <fn_name>` in js (brace
|
|
balanced — since phase 76 task 02 the functions live inside
|
|
mount(root), so the closing brace is indented, not column 0)."""
|
|
fn = js.find(f"function {fn_name}")
|
|
assert fn != -1, f"{fn_name} must be defined"
|
|
open_idx = js.find("{", fn)
|
|
depth = 0
|
|
for i in range(open_idx, len(js)):
|
|
c = js[i]
|
|
if c == "{":
|
|
depth += 1
|
|
elif c == "}":
|
|
depth -= 1
|
|
if depth == 0:
|
|
return js[fn : i + 1]
|
|
raise AssertionError(f"unbalanced braces in {fn_name}")
|
|
|
|
|
|
# ---------- the shell's RAG view: anonymous-safe ship-hidden markup ----------
|
|
|
|
|
|
def test_sync_button_ships_hidden_and_labeled() -> None:
|
|
"""#sync-btn SHIPS with the hidden attribute (anonymous-safe —
|
|
sources.js reveals it for the admin at view boot), is a real
|
|
<button type="button">, and carries aria-label="Sync sources" so
|
|
the accessible name stays stable across the label states."""
|
|
tag = re.search(r"<button[^>]*id=\"sync-btn\"[^>]*>", _text(SHELL_HTML))
|
|
assert tag, "the shell must carry the #sync-btn button (RAG view)"
|
|
attrs = tag.group(0)
|
|
assert 'class="sync-btn"' in attrs
|
|
assert 'type="button"' in attrs
|
|
assert re.search(r"\bhidden\b", attrs), "#sync-btn must ship hidden"
|
|
assert 'aria-label="Sync sources"' in attrs
|
|
|
|
|
|
def test_sync_button_has_icon_and_label_parts() -> None:
|
|
"""The button body is a refresh-cycle svg (aria-hidden — decorative,
|
|
the spin is the visible running state) + the .sync-label span with
|
|
the idle text, so the label can be swapped by sources.js.
|
|
|
|
Phase 128 (task 01): #sync-label is PARTS — the MAIN span
|
|
(.sync-label-main) carries the idle text (the pre-JS label renders
|
|
from the static markup; the part CSS may ellipsize it), the
|
|
COUNT span (.sync-label-count — the pinned "(n/m)" position,
|
|
never clipped) ships hidden + empty (there is no count at idle),
|
|
and — phase 129 (task 04) — the CHUNK span (.sync-label-chunk —
|
|
the pinned "· chunk n/m" embed-batch position) ships hidden +
|
|
empty too (no chunk position at idle — the part renders only
|
|
while a file's embed batch is in flight). The class/id stay (the
|
|
module query + the E2E label assertions)."""
|
|
text = _text(SHELL_HTML)
|
|
btn = text[text.find('id="sync-btn"') : text.find("</button>", text.find('id="sync-btn"'))]
|
|
assert re.search(r'<svg[^>]*class="sync-icon"[^>]*aria-hidden="true"', btn)
|
|
# Phase 128 (task 01) + Phase 129 (task 04): the parts markup —
|
|
# the exact three-part shape (the main span holds the idle text,
|
|
# the count + chunk spans are hidden + empty).
|
|
label = re.search(
|
|
r'<span\b[^>]*id="sync-label"[^>]*>'
|
|
r'<span class="sync-label-main">Sync sources</span>'
|
|
r'<span class="sync-label-count" hidden></span>'
|
|
r'<span class="sync-label-chunk" hidden></span>'
|
|
r"</span>",
|
|
btn,
|
|
)
|
|
assert label, (
|
|
"the #sync-label parts markup (main + hidden count + hidden chunk) is missing"
|
|
)
|
|
assert 'class="sync-label"' in label.group(0)
|
|
|
|
|
|
def test_sync_result_is_the_aria_live_announcer() -> None:
|
|
"""#sync-result sits right after the button and is a polite live
|
|
region (role="status" + aria-live="polite") — the last-result /
|
|
counts announcement for screen readers."""
|
|
text = _text(SHELL_HTML)
|
|
tag = re.search(r'<span[^>]*id="sync-result"[^>]*>', text)
|
|
assert tag, "the shell must carry the #sync-result announcer (RAG view)"
|
|
attrs = tag.group(0)
|
|
assert 'role="status"' in attrs
|
|
assert 'aria-live="polite"' in attrs
|
|
assert text.find('id="sync-result"') > text.find("</button>", text.find('id="sync-btn"'))
|
|
|
|
|
|
def test_sync_error_banner_is_a_hidden_alert() -> None:
|
|
"""The failure banner uses the chat error-banner markup style
|
|
(kb-banner + is-error) and role="alert", shipping hidden —
|
|
sources.js un-hides it with the error text on a failed run."""
|
|
text = _text(SHELL_HTML)
|
|
tag = re.search(r'<div[^>]*id="sync-error-banner"[^>]*>', text)
|
|
assert tag, "the shell must carry the #sync-error-banner (RAG view)"
|
|
attrs = tag.group(0)
|
|
assert "kb-banner" in attrs and "is-error" in attrs
|
|
assert 'role="alert"' in attrs
|
|
assert re.search(r"\bhidden\b", attrs)
|
|
assert 'id="sync-error-text"' in text
|
|
# The banner lives in the page content, not the 64px header bar.
|
|
assert text.find('id="sync-error-banner"') > text.find('<main id="main"')
|
|
|
|
|
|
def test_page_sub_copy_mentions_the_button() -> None:
|
|
"""The page-sub copy names the button as the one-click way to pull
|
|
the latest and re-import (the import CLI docs live elsewhere).
|
|
Phase 61: the copy describes the current source model (git repos +
|
|
local directories + uploaded archives), not the old ~/Homelab +
|
|
~/Deployments clone. Phase 76 (task 02): scoped to the RAG view —
|
|
the shell carries one .page-sub per view."""
|
|
sub = re.search(r'<p class="page-sub">(.*?)</p>', _rag_view(_text(SHELL_HTML)), re.DOTALL)
|
|
assert sub, "the RAG view must keep the .page-sub copy"
|
|
copy = re.sub(r"\s+", " ", sub.group(1)) # the markup wraps lines
|
|
assert "Press <strong>Sync sources</strong>" in copy
|
|
assert "pull the latest and re-import" in copy
|
|
assert "git repositories," in copy
|
|
assert "local directories, and uploaded archives" in copy
|
|
|
|
|
|
def test_sources_page_stays_cdn_free() -> None:
|
|
"""No-CDN rule (PLAN §7.3, A11): the new button markup adds no
|
|
external references — same-origin assets only (the integration
|
|
test_index_html_served_locally re-checks this on the served page)."""
|
|
text = _text(SHELL_HTML)
|
|
assert 'src="https://' not in text
|
|
assert 'href="https://' not in text
|
|
|
|
|
|
# ---------- sources.js: the admin reveal (view boot) ----------
|
|
|
|
|
|
def test_sources_js_reveals_sync_btn_on_the_admin_branch() -> None:
|
|
"""The view boot (mount's tail, phase 76 task 02) reveals #sync-btn
|
|
for the admin on the SAME cached whoami fetchIsAdmin() reads (no
|
|
extra fetch — header.js keeps the single /api/whoami call site; the
|
|
header itself is booted exactly once, by the chat module at shell
|
|
boot); anonymous users never leave the ship-hidden default."""
|
|
js = _text(SOURCES_JS)
|
|
boot = js[js.find("view boot (phase 76 task 02)"):]
|
|
assert "const admin = await fetchIsAdmin()" in boot
|
|
assert "syncBtn.hidden = !admin" in boot, "#sync-btn must join the admin reveal"
|
|
# The reveal must not introduce a second whoami call site.
|
|
header = _text(HEADER_JS)
|
|
assert header.count('fetch("/api/whoami")') == 1
|
|
assert 'fetch("/api/whoami")' not in js
|
|
|
|
|
|
# ---------- sources.js: the sync state machine (moved here from
|
|
# ---------- header.js in the owner rework 2026-08-28) ----------
|
|
|
|
|
|
def test_sources_js_owns_the_sync_button_elements() -> None:
|
|
"""The button refs are module-level and null-safe: #sync-btn, the
|
|
.sync-label inside the button, the .sync-icon inside the button — a
|
|
page without the markup is a complete no-op."""
|
|
js = _text(SOURCES_JS)
|
|
assert 'querySelector("#sync-btn")' in js
|
|
assert 'syncBtn.querySelector(".sync-label")' in js
|
|
assert 'syncBtn.querySelector(".sync-icon")' in js
|
|
|
|
|
|
def test_sources_js_calls_the_sync_api() -> None:
|
|
"""The click posts to POST /api/sync and the poll loop GETs
|
|
/api/sync/status — both through the same-origin API (A10)."""
|
|
js = _text(SOURCES_JS)
|
|
assert 'fetch("/api/sync", { method: "POST" })' in js
|
|
assert 'fetch("/api/sync/status")' in js
|
|
|
|
|
|
def test_sources_js_polls_every_2000ms() -> None:
|
|
"""The feedback loop is a 2000 ms poll of the status endpoint,
|
|
re-scheduled one tick at a time (setTimeout, not setInterval — an
|
|
in-flight fetch can never overlap the next tick)."""
|
|
js = _text(SOURCES_JS)
|
|
assert "SYNC_POLL_MS = 2000" in js
|
|
assert "setTimeout(tick, SYNC_POLL_MS)" in js
|
|
assert "setInterval" not in js
|
|
|
|
|
|
def test_sources_js_adopts_409_and_starts_on_202() -> None:
|
|
"""202 (started) and 409 (a run started elsewhere — e.g. a second
|
|
tab) both enter the running state and start polling: the UI never
|
|
starts a second run, it adopts the in-flight one.
|
|
|
|
Phase 64 (task 04): the entry is the sync-kind live label with no
|
|
file yet (the run is just starting — bare "Syncing…", A4)."""
|
|
js = _text(SOURCES_JS)
|
|
assert "r.status === 202 || r.status === 409" in js
|
|
idx = js.find("r.status === 202 || r.status === 409")
|
|
branch = js[idx : idx + 600]
|
|
assert "enterSyncRunningState(\"sync\", null, 0, 0)" in branch
|
|
assert "startSyncPolling()" in branch
|
|
|
|
|
|
def test_sources_js_hides_the_button_on_403() -> None:
|
|
"""A 403 anywhere (POST, the status poll, the load re-attach) is
|
|
treated as not-admin: the button hides — defense in depth behind
|
|
the whoami reveal (the primary gate)."""
|
|
js = _text(SOURCES_JS)
|
|
assert len(re.findall(r"r\.status === 403", js)) >= 3, (
|
|
"POST, the status poll, and the load re-attach must all handle 403"
|
|
)
|
|
assert js.count("syncBtn.hidden = true") >= 3, (
|
|
"every 403 branch must hide the button"
|
|
)
|
|
|
|
|
|
def test_sources_js_running_state_is_never_stale() -> None:
|
|
"""Entering the running state disables the button, sets aria-busy,
|
|
spins the icon, and swaps the label to the live file label
|
|
(fmtSyncLabel — the §7.4 feedback while the poll waits) — and a
|
|
fresh run starts clean: the previous failure's title / aria-label /
|
|
.is-error come off NOW, not when the run settles.
|
|
|
|
Phase 64 (task 04): #sync-result (the aria-live announcer) carries
|
|
the FULL untruncated label. Phase 98 (task 02): the title rule is
|
|
adjusted — the FULL untruncated LABEL (the file label, the overview
|
|
label, or the summaries label — whatever the running form is) rides
|
|
the button title for EVERY running label; the "title only when
|
|
there is a current file" (remove-when-null) rule is gone.
|
|
Phase 128 (task 01): the entry hands the label PARTS (the
|
|
phase-aware split — sync + status → fmtSyncPhaseLabel, the
|
|
fall-through → the byte-identical fmtSyncLabel) to the SINGLE
|
|
writer, setSyncLabel(main, count, chunk — phase 129: the chunk
|
|
part destructures with a "" default), which lands the main part,
|
|
the pinned count part (hidden when absent), the pinned chunk part
|
|
(phase 129 — hidden when absent), and the FULL untruncated label
|
|
on the button title + #sync-result (A4) — the direct span /
|
|
title / announcer writes are GONE from the entry (they live in
|
|
setSyncLabel)."""
|
|
js = _text(SOURCES_JS)
|
|
body = _body(js, "enterSyncRunningState")
|
|
assert "syncBtn.disabled = true" in body
|
|
assert 'syncBtn.setAttribute("aria-busy", "true")' in body
|
|
# the parts split (phase 98) is intact…
|
|
assert "fmtSyncPhaseLabel(status)" in body
|
|
assert "fmtSyncLabel(kind, currentFile, done, total)" in body
|
|
# …and the single writer lands the parts (A4 in it — the title is
|
|
# set for EVERY running label: the file-only rule is adjusted away).
|
|
assert "setSyncLabel(main, count, chunk)" in body
|
|
assert body.count("setSyncLabel(") == 1, "the entry writes ONLY through the parts writer"
|
|
assert "syncBtn.title =" not in body, ("the title write lives in setSyncLabel (A4)")
|
|
assert "syncLabel.textContent" not in body, ("no direct span write — the parts")
|
|
assert "syncResult.textContent" not in body, ("the announcer write lives in setSyncLabel (A4)")
|
|
assert 'syncBtn.setAttribute("aria-label", "Sync sources")' in body
|
|
assert "syncBtn.classList.remove(\"is-error\")" in body
|
|
assert "syncIcon.classList.add(\"is-spinning\")" in body
|
|
|
|
|
|
def test_sources_js_terminal_states() -> None:
|
|
"""Terminal rendering: success → enabled + 'Synced HH:MM' (local
|
|
time of finished_at); failed → enabled + retry-ready 'Sync sources'
|
|
label + the sanitized error in the button's title + aria-label +
|
|
the .is-error class. The counts formatting (fmtSyncResult) lives
|
|
here ('added' always announced, zero terms omitted — a no-op
|
|
re-sync reads '0 added · 1 unchanged')."""
|
|
js = _text(SOURCES_JS)
|
|
|
|
success = _body(js, "applySyncSuccess")
|
|
assert '"Synced"' in success and "fmtSyncTime(status.finished_at)" in success
|
|
|
|
failure = _body(js, "applySyncFailure")
|
|
assert 'settleSyncButton("Sync sources")' in failure # retry-ready
|
|
assert "syncBtn.title = error" in failure
|
|
assert 'syncBtn.setAttribute("aria-label", error)' in failure
|
|
assert "syncBtn.classList.add(\"is-error\")" in failure
|
|
assert "sanitizeSyncError(status.error)" in failure
|
|
|
|
result = _body(js, "fmtSyncResult")
|
|
assert "added" in result and "unchanged" in result
|
|
assert " · " in result
|
|
assert "> 0" in result, "zero terms must be omitted"
|
|
|
|
time = _body(js, "fmtSyncTime")
|
|
assert "getHours()" in time and "getMinutes()" in time, "local HH:MM of finished_at"
|
|
|
|
|
|
def test_sources_js_failed_error_is_sanitized() -> None:
|
|
"""The button's title/aria-label error is sanitized for the
|
|
attributes: the server already masks credentials (sync.py
|
|
_sanitize_error); the page collapses whitespace to a single line
|
|
and caps the length, and a missing error still names a failure."""
|
|
js = _text(SOURCES_JS)
|
|
body = _body(js, "sanitizeSyncError")
|
|
assert "replace(/\\s+/g, \" \")" in body, "single line for the attributes"
|
|
assert "200" in body, "long errors (chatty git stderr) are capped"
|
|
assert '"The sync failed."' in body
|
|
|
|
|
|
def test_sources_js_settles_the_button_on_terminal() -> None:
|
|
"""settleSyncButton re-enables the control, drops aria-busy, the
|
|
title, and the failed affordances, and un-spins the icon — the
|
|
button can never sit disabled after a run reaches a terminal state
|
|
(failed included: retry-ready). Phase 128 (task 01): the settle /
|
|
idle label ("Sync sources", "Synced HH:MM") goes through the parts
|
|
writer — setSyncLabel(label), count hidden (absent), the full
|
|
string riding the title + #sync-result (A4, every label state)."""
|
|
js = _text(SOURCES_JS)
|
|
body = _body(js, "settleSyncButton")
|
|
assert "syncBtn.disabled = false" in body
|
|
assert 'syncBtn.removeAttribute("aria-busy")' in body
|
|
assert 'syncBtn.removeAttribute("title")' in body
|
|
assert 'syncBtn.setAttribute("aria-label", "Sync sources")' in body
|
|
assert "syncIcon.classList.remove(\"is-spinning\")" in body
|
|
assert "setSyncLabel(label)" in body, ("the settle label goes through the parts writer")
|
|
assert "syncLabel.textContent" not in body, ("no direct span write — the parts")
|
|
|
|
|
|
def test_sources_js_never_starts_a_second_poll_loop() -> None:
|
|
"""startSyncPolling is guarded by the module-level timer: a 409
|
|
adoption, a reload re-attach, or a stray call can never run two
|
|
poll loops at once (phase completion criterion)."""
|
|
js = _text(SOURCES_JS)
|
|
fn = js.find("function startSyncPolling")
|
|
head = js[fn : js.find("const tick", fn)]
|
|
assert re.search(r"if\s*\(\s*syncPollTimer\s*!==\s*null\s*\)\s*return", head), (
|
|
"the single-loop guard must be the first statement"
|
|
)
|
|
assert "clearTimeout(syncPollTimer)" in _body(js, "stopSyncPolling")
|
|
|
|
|
|
def test_sources_js_has_no_client_side_hard_timeout() -> None:
|
|
"""Phase locked decision: a sync can legitimately run for minutes
|
|
(and outlive the page), so there is NO client-side hard timeout —
|
|
the 2 s poll is the feedback loop and the server state is
|
|
authoritative (the 120 s LLM-turn guard must not leak into the
|
|
sync path)."""
|
|
js = _text(SOURCES_JS)
|
|
assert "TURN_TIMEOUT" not in js
|
|
sync_start = js.find("Sync sources button (Sources page only)")
|
|
assert sync_start != -1, "the sync section marker comment"
|
|
assert "120" not in js[sync_start:]
|
|
|
|
|
|
def test_sources_js_reattaches_on_load_admin_only() -> None:
|
|
"""initSyncButton (run at page boot, after the click binding)
|
|
awaits the SAME cached whoami — ADMIN ONLY (non-admins never poll,
|
|
the status endpoint is admin-only): a running run re-enters the
|
|
running state (reload mid-sync), a terminal run renders its last
|
|
result, idle settles retry-ready; the click binding wires
|
|
startSync to the button."""
|
|
js = _text(SOURCES_JS)
|
|
body = _body(js, "initSyncButton")
|
|
assert "await fetchIsAdmin()" in body, "admin-only boot (no extra fetch)"
|
|
assert 'fetch("/api/sync/status")' in body
|
|
assert 'status.state === "running"' in body
|
|
assert 'status.state === "success"' in body
|
|
assert 'status.state === "failed"' in body
|
|
assert (
|
|
'syncBtn.addEventListener("click", startSync);\n initSyncButton();'
|
|
in js
|
|
), "the click binding and the boot re-attach ship together, guarded on syncBtn"
|
|
|
|
|
|
def test_sources_js_emits_bor_sync_status_on_state_changes() -> None:
|
|
"""Every state change dispatches window 'bor:sync-status' with the
|
|
status object as detail — the channel other page scripts (or
|
|
future ones) can subscribe to. The click path emits the synthetic
|
|
running frame IMMEDIATELY (no 2 s poll lag)."""
|
|
js = _text(SOURCES_JS)
|
|
assert (
|
|
'window.dispatchEvent(new CustomEvent("bor:sync-status", { detail: status }))'
|
|
in js
|
|
)
|
|
for fn in ("applySyncSuccess", "applySyncFailure", "applySyncIdle"):
|
|
assert "emitSyncStatus" in _body(js, fn), f"{fn} must emit its frame"
|
|
# the running frame: synthetic on click, the real object on boot
|
|
assert 'emitSyncStatus({ state: "running" })' in js
|
|
assert "emitSyncStatus(status)" in _body(js, "initSyncButton")
|
|
|
|
|
|
# ---------- sources.js: the sync failure modal (phase 41, TODO.md L4) ----------
|
|
|
|
|
|
def test_sources_js_owns_a_lazily_created_sync_modal() -> None:
|
|
"""The modal is page-owned and created lazily ONCE (module-level
|
|
`let syncModal = null`): a .sync-modal-backdrop holding a
|
|
.sync-modal panel with role="alertdialog" + aria-modal +
|
|
labelled/described ids + the close button, appended to
|
|
document.body — no page-markup changes needed (the section marker
|
|
records that the former header.js modal was recreated here)."""
|
|
js = _text(SOURCES_JS)
|
|
assert "let syncModal = null" in js, "module-level once-only modal ref"
|
|
assert 'role="alertdialog"' in js
|
|
assert 'aria-modal="true"' in js
|
|
assert 'aria-labelledby="sync-modal-title"' in js
|
|
assert 'aria-describedby="sync-modal-error"' in js
|
|
assert "sync failure modal" in js, "the modal section marker"
|
|
create = _body(js, "createSyncModal")
|
|
assert "sync-modal-backdrop" in create
|
|
assert 'document.body.appendChild(backdrop)' in create
|
|
assert "return backdrop" in create, "the module ref must hold the created element"
|
|
assert "Sync failed" in create, "the dialog title"
|
|
assert 'class="sync-modal-close"' in create
|
|
assert 'aria-label="Close error dialog"' in create
|
|
|
|
|
|
def test_sync_modal_error_is_rendered_via_text_content() -> None:
|
|
"""The error text is ALWAYS set via textContent (XSS-safe — no
|
|
innerHTML with user data in the open path), and it is set BEFORE
|
|
the already-open check, so a second failure while open updates the
|
|
text IN PLACE (no stacking, no focus jump)."""
|
|
js = _text(SOURCES_JS)
|
|
body = _body(js, "showSyncModal")
|
|
assert 'querySelector("#sync-modal-error").textContent' in body
|
|
assert "innerHTML" not in body, "the open path never touches innerHTML"
|
|
text = body.find("textContent")
|
|
open_check = body.find('contains("is-open")')
|
|
assert text != -1 and open_check != -1 and text < open_check, (
|
|
"the in-place update happens while the modal is already open"
|
|
)
|
|
|
|
|
|
def test_sync_modal_focus_goes_in_and_out_to_sync_btn() -> None:
|
|
"""On open: document.activeElement is remembered and focus moves
|
|
to the close button — with a fallback to #sync-btn when the active
|
|
element is <body> (the run's disabled button dropped focus there;
|
|
the close must still land on the control that started the run);
|
|
on close: focus returns to the remembered element (guarded by
|
|
document.contains — a detached target is a no-op)."""
|
|
js = _text(SOURCES_JS)
|
|
open_body = _body(js, "showSyncModal")
|
|
assert "const active = document.activeElement" in open_body
|
|
assert "active !== document.body" in open_body, (
|
|
"the disabled-button window leaves focus on <body> — the fallback"
|
|
)
|
|
assert "? active : syncBtn" in open_body, ("the fallback remembers #sync-btn")
|
|
assert 'querySelector(".sync-modal-close").focus()' in open_body
|
|
close_body = _body(js, "closeSyncModal")
|
|
assert "document.contains(target)" in close_body
|
|
assert "target.focus()" in close_body
|
|
assert 'contains("is-open")' in close_body, "closing a closed modal is a no-op"
|
|
|
|
|
|
def test_sync_modal_closes_via_button_esc_and_backdrop() -> None:
|
|
"""All three dismissal paths call the SAME close function: the
|
|
close button, Esc (ONE document keydown binding, acting only while
|
|
the modal is open), and a click on the backdrop element itself —
|
|
the event.target check keeps clicks bubbling from the panel from
|
|
closing it."""
|
|
js = _text(SOURCES_JS)
|
|
create = _body(js, "createSyncModal")
|
|
assert 'addEventListener("click", closeSyncModal)' in create
|
|
assert 'e.key === "Escape"' in create
|
|
assert 'addEventListener("keydown"' in create
|
|
assert 'backdrop.classList.contains("is-open")' in create
|
|
assert "e.target === backdrop" in create
|
|
|
|
|
|
def test_sources_js_apply_sync_failure_opens_the_modal_after_the_event() -> None:
|
|
"""applySyncFailure opens the modal with the SANITIZED error, and
|
|
the call comes AFTER the existing button title/aria/.is-error
|
|
lines + emitSyncStatus (the bor:sync-status event — those lines
|
|
stay byte-identical; the modal is additive)."""
|
|
js = _text(SOURCES_JS)
|
|
body = _body(js, "applySyncFailure")
|
|
call = body.find("showSyncModal(")
|
|
emit = body.find("emitSyncStatus(status)")
|
|
assert call != -1, "the modal must open from the failure path"
|
|
assert emit != -1 and call > emit, "the event still emits first (byte-identical)"
|
|
assert "showSyncModal(error)" in body, "the sanitized error goes to the modal"
|
|
# The pre-existing affordances stay (the Sources banner contract).
|
|
assert "syncBtn.title = error" in body
|
|
assert 'syncBtn.setAttribute("aria-label", error)' in body
|
|
assert 'syncBtn.classList.add("is-error")' in body
|
|
assert "emitSyncStatus(status)" in body
|
|
|
|
|
|
# ---------- sources.js: the page-specific result line + banner ----------
|
|
|
|
|
|
def test_sources_js_renders_result_line_and_banner_directly() -> None:
|
|
"""sources.js owns the page-specific rendering on the Sources page:
|
|
#sync-result (aria-live) + #sync-error-banner (role=alert), driven
|
|
directly by the state appliers: success → the counts (fmtSyncResult)
|
|
+ a live catalog refresh; failed → the banner with the sanitized
|
|
error; every state change still emits 'bor:sync-status' (the status
|
|
object as detail) for other subscribers."""
|
|
js = _text(SOURCES_JS)
|
|
success = _body(js, "applySyncSuccess")
|
|
assert "syncResult.textContent = fmtSyncResult(status.detail)" in success, (
|
|
"the counts live in status.detail — a bare status renders all zeros"
|
|
)
|
|
assert "loadTree()" in success, (
|
|
"the catalog re-fetches live on a successful sync (phase 97: the tree)"
|
|
)
|
|
failure = _body(js, "applySyncFailure")
|
|
assert "showSyncError(error)" in failure
|
|
assert "emitSyncStatus" in _body(js, "applySyncIdle")
|
|
assert (
|
|
'window.dispatchEvent(new CustomEvent("bor:sync-status", { detail: status }))'
|
|
in _body(js, "emitSyncStatus")
|
|
)
|
|
|
|
|
|
def test_header_js_no_longer_owns_the_sync_machine() -> None:
|
|
"""The state machine is GONE from header.js (sources.js owns it —
|
|
the button is Sources-page only, so the page drives it): no button
|
|
refs, no POST, no status poll, no button-state helpers, no modal.
|
|
sources.js keeps every one of them."""
|
|
header = _text(HEADER_JS)
|
|
for gone in (
|
|
"SYNC_POLL_MS",
|
|
"syncPollTimer",
|
|
"startSyncPolling",
|
|
"stopSyncPolling",
|
|
"enterSyncRunningState",
|
|
"settleSyncButton",
|
|
"applySyncSuccess",
|
|
"applySyncFailure",
|
|
"applySyncIdle",
|
|
"initSyncButton",
|
|
"syncModal",
|
|
"syncBtn",
|
|
'fetch("/api/sync", { method: "POST" })',
|
|
'fetch("/api/sync/status")',
|
|
'querySelector("#sync-btn")',
|
|
):
|
|
assert gone not in header, f"{gone!r} must be gone from header.js (sources.js owns it)"
|
|
js = _text(SOURCES_JS)
|
|
for kept in (
|
|
"SYNC_POLL_MS = 2000",
|
|
"syncPollTimer",
|
|
"startSyncPolling",
|
|
"enterSyncRunningState",
|
|
"initSyncButton",
|
|
'fetch("/api/sync", { method: "POST" })',
|
|
'fetch("/api/sync/status")',
|
|
'querySelector("#sync-btn")',
|
|
):
|
|
assert kept in js, f"{kept!r} must be in sources.js (the owner)"
|
|
|
|
|
|
# ---------- styles.css: the §7.4 states ----------
|
|
|
|
|
|
def test_sync_button_css_ghost_pill_and_disabled_state() -> None:
|
|
""".sync-btn is a pill in the shared button family (contrast pair
|
|
ink-soft on surface ≥ 4.5:1), with a ≥44px touch floor and a
|
|
:disabled state (never stale — the busy look is visible)."""
|
|
css = _text(STYLES_CSS)
|
|
block = re.search(r"\.sync-btn\s*\{([^}]*)\}", css)
|
|
assert block, "styles.css must define .sync-btn"
|
|
body = block.group(1)
|
|
assert "min-height: 44px" in body
|
|
assert "color: var(--ink-soft)" in body
|
|
assert "border-radius: 999px" in body
|
|
disabled = re.search(r"\.sync-btn:disabled\s*\{([^}]*)\}", css)
|
|
assert disabled, ".sync-btn:disabled must be styled"
|
|
assert "cursor: wait" in disabled.group(1)
|
|
|
|
|
|
def test_sync_icon_spins_and_respects_reduced_motion() -> None:
|
|
"""The running state spins the refresh icon on the shared spin
|
|
keyframes (1s linear infinite), and prefers-reduced-motion stills
|
|
it — the existing opt-out pattern."""
|
|
css = _text(STYLES_CSS)
|
|
spin = re.search(r"\.sync-btn \.sync-icon\.is-spinning\s*\{([^}]*)\}", css)
|
|
assert spin, "the .is-spinning state must be styled"
|
|
assert "animation: spin 1s linear infinite" in spin.group(1)
|
|
spinner = r"\.sync-btn \.sync-icon\.is-spinning\s*\{([^}]*)\}"
|
|
reduced = re.search(r"@media \(prefers-reduced-motion: reduce\)\s*\{\s*" + spinner, css)
|
|
assert reduced, "the spin must opt out under prefers-reduced-motion"
|
|
assert "animation: none" in reduced.group(1)
|
|
assert "@keyframes spin" in css, "the spin keyframes are shared (pre-existing)"
|
|
|
|
|
|
def test_sync_result_is_styled() -> None:
|
|
"""#sync-result (the aria-live last-result line) is styled in the
|
|
theme tokens — soft ink, small mono, no wrap in the header bar."""
|
|
css = _text(STYLES_CSS)
|
|
block = re.search(r"\.sync-result\s*\{([^}]*)\}", css)
|
|
assert block, "styles.css must define .sync-result"
|
|
assert "var(--ink-soft)" in block.group(1)
|
|
|
|
|
|
# ---------- styles.css: the sync failure modal (phase 41, TODO.md L4) ----------
|
|
|
|
|
|
def test_sync_modal_css_error_palette_and_stacking() -> None:
|
|
""".sync-modal-backdrop: fixed, full-viewport, the --bg-82% dim
|
|
(phase 92: color-mix of the identity variable), z-index above the
|
|
sticky header; .sync-modal: the centered ≈28rem panel on
|
|
the phase-08 error palette (panel on --err-bg, 1px --err-line
|
|
border, --err-ink error text, --ink title — all computed ≥4.5:1);
|
|
open/close via .is-open (visibility/opacity)."""
|
|
css = _text(STYLES_CSS)
|
|
backdrop = re.search(r"\.sync-modal-backdrop\s*\{([^}]*)\}", css)
|
|
assert backdrop, "styles.css must define .sync-modal-backdrop"
|
|
b = backdrop.group(1)
|
|
assert "position: fixed" in b
|
|
assert "inset: 0" in b
|
|
assert "z-index: 1000" in b, "above the sticky header (20) + skip-link (100)"
|
|
assert "color-mix(in srgb, var(--bg) 82%, transparent)" in b, (
|
|
"the dim over the page (phase 92: --bg at 82%)"
|
|
)
|
|
open_state = re.search(r"\.sync-modal-backdrop\.is-open\s*\{([^}]*)\}", css)
|
|
assert open_state, ".is-open must be the open state"
|
|
assert "visibility: visible" in open_state.group(1)
|
|
assert "opacity: 1" in open_state.group(1)
|
|
panel = re.search(r"\.sync-modal\s*\{([^}]*)\}", css)
|
|
assert panel, "styles.css must define .sync-modal"
|
|
p = panel.group(1)
|
|
assert "max-width: 28rem" in p
|
|
assert "var(--err-bg)" in p
|
|
assert "var(--err-line)" in p
|
|
title = re.search(r"#sync-modal-title\s*\{([^}]*)\}", css)
|
|
assert title, "the modal title must be styled"
|
|
assert "var(--ink)" in title.group(1)
|
|
error = re.search(r"#sync-modal-error\s*\{([^}]*)\}", css)
|
|
assert error, "the modal error line must be styled"
|
|
assert "var(--err-ink)" in error.group(1)
|
|
|
|
|
|
def test_sync_modal_close_button_touch_floor() -> None:
|
|
"""The close button keeps the 44px touch floor at every width (the
|
|
global 3px :focus-visible outline applies — no per-button
|
|
override)."""
|
|
css = _text(STYLES_CSS)
|
|
block = re.search(r"\.sync-modal-close\s*\{([^}]*)\}", css)
|
|
assert block, "styles.css must define .sync-modal-close"
|
|
assert "min-width: 44px" in block.group(1)
|
|
assert "min-height: 44px" in block.group(1)
|
|
|
|
|
|
def test_sync_modal_respects_reduced_motion() -> None:
|
|
"""The open/close fade is stilled under prefers-reduced-motion —
|
|
the phase-25 / doc-modal opt-out pattern."""
|
|
css = _text(STYLES_CSS)
|
|
reduced = re.search(
|
|
r"@media \(prefers-reduced-motion: reduce\)\s*\{\s*\.sync-modal-backdrop\s*\{([^}]*)\}",
|
|
css,
|
|
)
|
|
assert reduced, "the backdrop fade must opt out under prefers-reduced-motion"
|
|
assert "transition: none" in reduced.group(1)
|
|
|
|
|
|
# ---------- phase 64 (task 02): per-file progress on the sync status ----------
|
|
#
|
|
# The GET /api/sync/status contract in app/api/sync.py (task 04 renders
|
|
# the live file label on the button — its pins live in
|
|
# tests/unit/test_frontend_sync_upload.py). The runner's seams are
|
|
# monkeypatched on ``app.api.sync`` (the house mock-import pattern from
|
|
# tests/integration/test_sync_api.py); the background task runs on a
|
|
# worker thread's own loop so the test can read the status mid-run.
|
|
# No DB, no HTTP: every seam is faked.
|
|
|
|
|
|
@pytest.fixture()
|
|
def fresh_sync_status() -> Iterator[None]:
|
|
"""The module-level status object + task are process-global: reset
|
|
them before AND after every progress test (the integration suite's
|
|
``_fresh_sync_state`` pattern)."""
|
|
sync_api._status = sync_api.SyncStatus()
|
|
sync_api._task = None
|
|
yield
|
|
sync_api._status = sync_api.SyncStatus()
|
|
sync_api._task = None
|
|
|
|
|
|
class _GatedClone:
|
|
"""A ``clone_or_pull`` that parks between start and finish on a
|
|
threading gate, so the test can read the status during the
|
|
clone/pull phase (A4: no file yet)."""
|
|
|
|
def __init__(self, started: threading.Event, release: threading.Event) -> None:
|
|
self.started = started
|
|
self.release = release
|
|
self.calls: list[tuple[str, Path]] = []
|
|
|
|
def __call__(self, url: str, dest: Path | str) -> Path:
|
|
dest = Path(dest)
|
|
self.calls.append((url, dest))
|
|
dest.mkdir(parents=True, exist_ok=True)
|
|
(dest / "notes.md").write_text("# repo\ncontent for the KB\n", encoding="utf-8")
|
|
self.started.set()
|
|
self.release.wait() # blocking is fine: the clone is a sync call
|
|
return dest
|
|
|
|
|
|
class _GatedImport:
|
|
"""The mock import: fires the runner's OWN progress hook once (the
|
|
progress-shaped call goes through the real hook closure — the
|
|
closure under test), parks on a threading gate so the test can read
|
|
the status mid-run, then returns the canned summary (or raises
|
|
``fail``).
|
|
|
|
Phase 129 (task 03): optionally ALSO fires the in-flight file's
|
|
chunk-position pairs (``chunk_pairs``) through the runner's OWN
|
|
``_chunk_hook`` closure: pair one lands before the main park
|
|
(readable mid-run), and every later pair lands after the main
|
|
park + its own park (``chunk_release``), so each position is
|
|
readable on the status between ticks."""
|
|
|
|
def __init__(
|
|
self,
|
|
summary: ImportSummary,
|
|
started: threading.Event,
|
|
release: threading.Event,
|
|
fail: BaseException | None = None,
|
|
chunk_pairs: Sequence[tuple[int, int]] = (),
|
|
chunk_release: threading.Event | None = None,
|
|
) -> None:
|
|
self.summary = summary
|
|
self.started = started
|
|
self.release = release
|
|
self.fail = fail
|
|
self.hook_calls: list[tuple[str, str, int, int]] = []
|
|
self.prune_flags: list[bool] = []
|
|
self.chunk_pairs = list(chunk_pairs)
|
|
self.chunk_release = chunk_release
|
|
self.chunk_hook_calls: list[tuple[int, int]] = []
|
|
|
|
async def __call__(
|
|
self,
|
|
sources: list[Path],
|
|
llm: object,
|
|
*,
|
|
prune: bool = False,
|
|
limit: int | None = None,
|
|
session: object = None,
|
|
progress: Callable[[str, str, int, int], None] | None = None,
|
|
on_chunk_progress: Callable[[int, int], None] | None = None, # phase 129
|
|
ignore_by_root: dict[str, list[str]] | None = None, # phase 89
|
|
include_hidden_by_root: dict[str, bool] | None = None, # phase 105
|
|
doc_dates_by_root: dict[str, dict[str, datetime]] | None = None, # phase 106
|
|
) -> ImportSummary:
|
|
self.prune_flags.append(prune)
|
|
if progress is not None:
|
|
progress("repo", "notes/deep.md", 1, 3)
|
|
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
|
|
if on_chunk_progress is not None and self.chunk_pairs:
|
|
# Phase 129 (task 03): the file's chunk-position pairs,
|
|
# fired through the runner's OWN hook closure (the closure
|
|
# under test) — pair one before the main park, each later
|
|
# pair after the main park + its own park (each park lets
|
|
# the test read the status mid-run). The runner passes the
|
|
# hook only when the wiring is intact — a regression to
|
|
# None skips this branch and the pair tests time out.
|
|
on_chunk_progress(*self.chunk_pairs[0])
|
|
self.chunk_hook_calls.append(self.chunk_pairs[0])
|
|
self.started.set()
|
|
await asyncio.to_thread(self.release.wait)
|
|
for pair in self.chunk_pairs[1:]:
|
|
on_chunk_progress(*pair)
|
|
self.chunk_hook_calls.append(pair)
|
|
if self.chunk_release is not None:
|
|
await asyncio.to_thread(self.chunk_release.wait)
|
|
else:
|
|
self.started.set()
|
|
await asyncio.to_thread(self.release.wait) # park without freezing the worker loop
|
|
if self.fail is not None:
|
|
raise self.fail
|
|
return self.summary
|
|
|
|
|
|
def _patch_sync_seams(
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
tmp_path: Path,
|
|
fake_import: _GatedImport,
|
|
fake_clone: _GatedClone,
|
|
) -> None:
|
|
"""The runner's seams, monkeypatched on ``app.api.sync`` (the house
|
|
mock-import pattern): fresh settings (no ``.env`` leak), a no-op
|
|
model probe, a sentinel LLM client, one git row, the gated clone +
|
|
import, a no-op overview, a no-op folder-summary step (phase 94 —
|
|
the sentinel LLM client has no ``chat``), and the DB-free
|
|
sources-version step (dummy session + pinned counters)."""
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"get_settings",
|
|
lambda: Settings(_env_file=None, sources_dir=str(tmp_path / "bor")), # pyright: ignore[reportCallIssue]
|
|
)
|
|
|
|
async def fake_probe(llm: object) -> None:
|
|
pass
|
|
|
|
monkeypatch.setattr(sync_api, "check_models", fake_probe)
|
|
monkeypatch.setattr(sync_api, "LLMClient", lambda: object())
|
|
monkeypatch.setattr(
|
|
sync_api,
|
|
"effective_sources",
|
|
lambda session: (
|
|
# kind explicit: the Python-side default applies at INSERT
|
|
# flush, not on an in-memory instance
|
|
[GitSource(url="https://git.example.com/repo.git", kind="git")],
|
|
"db",
|
|
),
|
|
)
|
|
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
|
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
|
|
|
async def fake_overview(llm: object, session: object = None) -> bool:
|
|
return False
|
|
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
|
|
|
async def fake_folder_summaries(
|
|
db: object,
|
|
llm: object,
|
|
*,
|
|
skip: bool = False,
|
|
only_missing: bool = False,
|
|
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
|
|
on_progress: Callable[[int, int, str, str], None] | None = None,
|
|
) -> dict[str, int]:
|
|
# Phase 98 (task 01): the runner wires the progress hook into
|
|
# the generation branches — the fake accepts it (the state-
|
|
# machine tests' canned run never fires it: zero summary
|
|
# counters, like a no-gap skip). Phase 127 (task 03): the
|
|
# changed-KB branch passes the change set as changed_docs= —
|
|
# the fake accepts it (these canned runs record nothing).
|
|
return {"generated": 0, "failed": 0, "pruned": 0}
|
|
|
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folder_summaries)
|
|
# Phase 96 (task 03): the unchanged-walk gap probe is DB-free in
|
|
# these state-machine tests — no gap, so the folder step stays
|
|
# skipped exactly as before the gate change.
|
|
monkeypatch.setattr(sync_api, "missing_folder_summaries", lambda session: [])
|
|
|
|
class _DummySession:
|
|
def close(self) -> None:
|
|
pass
|
|
|
|
def commit(self) -> None:
|
|
pass
|
|
|
|
monkeypatch.setattr(sync_api, "SessionLocal", _DummySession)
|
|
monkeypatch.setattr(sync_api, "bump_sources_version", lambda session: 1)
|
|
monkeypatch.setattr(sync_api, "current_sources_version", lambda session: 1)
|
|
|
|
|
|
def _start_run() -> tuple[threading.Thread, list[BaseException]]:
|
|
"""Run the module-level runner on a worker thread's own event loop
|
|
(the house background-task pattern), capturing any unexpected
|
|
exception — the runner is supposed to die in state, never raise."""
|
|
errors: list[BaseException] = []
|
|
|
|
def _run() -> None:
|
|
try:
|
|
asyncio.run(sync_api._run_sync())
|
|
except BaseException as e: # noqa: BLE001 — surfaced to the test
|
|
errors.append(e)
|
|
|
|
thread = threading.Thread(target=_run, daemon=True)
|
|
thread.start()
|
|
return thread, errors
|
|
|
|
|
|
def test_idle_status_pins_full_shape_including_progress_keys(
|
|
fresh_sync_status: None,
|
|
) -> None:
|
|
"""Idle: the three phase-64 progress keys ride along as null/0/0,
|
|
the two phase-129 chunk keys as 0/0, the four phase-98 phase keys
|
|
as null/0/0/0, and EVERY pre-existing key is unchanged — the full
|
|
response dict is pinned, so the current UI and every existing
|
|
consumer keep working."""
|
|
assert sync_api.sync_status() == {
|
|
"state": "idle",
|
|
"started_at": None,
|
|
"finished_at": None,
|
|
"detail": {},
|
|
"error": None,
|
|
"current_file": None,
|
|
"files_done": 0,
|
|
"files_total": 0,
|
|
# Phase 129 (task 03): the chunk-position keys — 0/0 idle
|
|
# (the dataclass defaults, no special-casing).
|
|
"current_chunk": 0,
|
|
"chunks_total": 0,
|
|
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
|
|
"phase": None,
|
|
"current_summary": None,
|
|
"summaries_done": 0,
|
|
"summaries_total": 0,
|
|
}
|
|
|
|
|
|
def test_mid_run_status_reports_current_file(
|
|
fresh_sync_status: None,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
tmp_path: Path,
|
|
) -> None:
|
|
"""Mid-run: the status carries the file the (mock) import is
|
|
processing — assigned through the runner's own hook closure; during
|
|
the clone/pull phase (A4) no file is reported yet. The success
|
|
terminal clears ``current_file`` but keeps the final counts."""
|
|
clone = _GatedClone(threading.Event(), threading.Event())
|
|
fake_import = _GatedImport(
|
|
ImportSummary(files=3, added=1, updated=1, unchanged=1),
|
|
threading.Event(),
|
|
threading.Event(),
|
|
)
|
|
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
|
|
|
|
thread, errors = _start_run()
|
|
try:
|
|
assert clone.started.wait(5.0), "the run never reached the clone phase"
|
|
# Clone/pull prelude: running — but no file yet (A4: bare
|
|
# "Syncing…") and no phase yet either (D1: the prelude stays
|
|
# null — the bare label's pin), and no chunk position yet
|
|
# (phase 129: 0/0 before the first embed batch).
|
|
s = sync_api.sync_status()
|
|
assert s["state"] == "running"
|
|
assert s["phase"] is None
|
|
assert s["current_file"] is None
|
|
assert s["current_summary"] is None
|
|
assert s["files_done"] == 0 and s["files_total"] == 0
|
|
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
|
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
|
clone.release.set()
|
|
assert fake_import.started.wait(5.0), "the run never reached the import"
|
|
# Import phase: the hook's file is live on the status, and the
|
|
# phase machine names the phase (the summary span has not
|
|
# started — its counters are still 0/0); no embed batch has
|
|
# fired the chunk hook (this canned import passes no pairs),
|
|
# so the chunk position stays 0/0.
|
|
s = sync_api.sync_status()
|
|
assert s["state"] == "running"
|
|
assert s["phase"] == "import"
|
|
assert s["current_file"] == "repo/notes/deep.md"
|
|
assert s["files_done"] == 1
|
|
assert s["files_total"] == 3
|
|
assert s["current_summary"] is None
|
|
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
|
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
|
fake_import.release.set()
|
|
finally:
|
|
clone.release.set()
|
|
fake_import.release.set()
|
|
thread.join(10.0)
|
|
assert not thread.is_alive()
|
|
assert errors == [], f"the runner raised: {errors!r}"
|
|
# Wiring: prune=True is preserved, and the hook fired through the
|
|
# runner's own closure (the recorded call is what the closure
|
|
# assigned to the status above).
|
|
assert fake_import.prune_flags == [True]
|
|
assert fake_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
|
|
# Success terminal: current_file null, final file counts retained;
|
|
# the phase machine clears phase + current_summary — the fake
|
|
# folder step never fired the summary hook, so the summary counts
|
|
# stay 0/0 (the keep-final-counts convention keeps what fired);
|
|
# the chunk position clears with the file (phase 129 — it is a
|
|
# sub-position of a done file, never a keep-final counter).
|
|
s = sync_api.sync_status()
|
|
assert s["state"] == "success"
|
|
assert s["phase"] is None
|
|
assert s["current_summary"] is None
|
|
assert s["current_file"] is None
|
|
assert s["files_done"] == 1 and s["files_total"] == 3
|
|
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
|
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
|
assert s["error"] is None
|
|
assert s["started_at"] is not None and s["finished_at"] is not None
|
|
|
|
|
|
def test_failed_terminal_clears_current_file_keeps_counts(
|
|
fresh_sync_status: None,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
tmp_path: Path,
|
|
) -> None:
|
|
"""Terminal (failed): a run that dies inside the import clears
|
|
``current_file`` but keeps the hook's final counts — the last
|
|
position is useful context next to the (sanitized) error.
|
|
|
|
Phase 129 (task 03): the convention split extends to the chunk
|
|
position — the (about-to-fail) import reports a live pair
|
|
(1/38), the failed terminal clears it to 0/0 (a sub-position of a
|
|
file that's done/abandoned), while the FILE counts keep their
|
|
final values next to the error."""
|
|
clone = _GatedClone(threading.Event(), threading.Event())
|
|
fake_import = _GatedImport(
|
|
ImportSummary(),
|
|
threading.Event(),
|
|
threading.Event(),
|
|
fail=EmbeddingError(
|
|
"embeddings request to https://u:p@aipi.example.com/v1 "
|
|
"failed: connection refused"
|
|
),
|
|
chunk_pairs=[(1, 38)], # phase 129: the live pair before the raise
|
|
)
|
|
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
|
|
|
|
thread, errors = _start_run()
|
|
try:
|
|
assert clone.started.wait(5.0), "the run never reached the clone phase"
|
|
clone.release.set()
|
|
assert fake_import.started.wait(5.0), "the chunk pair never landed"
|
|
# While the (about-to-fail) import is parked: the file is live,
|
|
# and so is its chunk position (the pair the fake fired through
|
|
# the runner's own closure).
|
|
s = sync_api.sync_status()
|
|
assert s["state"] == "running"
|
|
assert s["current_file"] == "repo/notes/deep.md"
|
|
assert s["files_done"] == 1 and s["files_total"] == 3
|
|
assert s["current_chunk"] == 1 and s["chunks_total"] == 38
|
|
fake_import.release.set()
|
|
finally:
|
|
clone.release.set()
|
|
fake_import.release.set()
|
|
thread.join(10.0)
|
|
assert not thread.is_alive()
|
|
assert errors == []
|
|
# Wiring: the pair fired through the runner's own closure.
|
|
assert fake_import.chunk_hook_calls == [(1, 38)]
|
|
s = sync_api.sync_status()
|
|
assert s["state"] == "failed"
|
|
assert s["current_file"] is None # cleared in the terminal state
|
|
assert s["phase"] is None # phase 98: cleared in the terminal state
|
|
assert s["current_summary"] is None
|
|
assert s["files_done"] == 1 and s["files_total"] == 3 # final counts kept
|
|
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
|
# Phase 129 (task 03): the chunk position clears in the FAILED
|
|
# terminal too — the convention split, pinned (the FILE counts
|
|
# above keep their final values, the sub-position does not).
|
|
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
|
assert s["detail"] == {}
|
|
error = s["error"] or ""
|
|
assert "*****@aipi.example.com" in error # credentials masked
|
|
assert "u:p" not in error
|
|
assert "connection refused" in error # the reason survives
|
|
|
|
|
|
# ---------- phase 129 (task 03): the in-flight file's embed-chunk ----------
|
|
# ---------- position (current_chunk / chunks_total) ----------------------
|
|
#
|
|
# Same seams as the phase-64 block above: the fake import fires the
|
|
# runner's OWN ``_chunk_hook`` closure (the task under test) with the
|
|
# file's chunk-position pairs, parking between pairs so the test reads
|
|
# the status BETWEEN ticks (the A6 batch-granularity contract: 14 =
|
|
# the first chunk of the in-flight token-budgeted batch).
|
|
|
|
|
|
def _poll_until(
|
|
predicate: Callable[[], bool],
|
|
timeout: float = 5.0,
|
|
what: str = "condition",
|
|
) -> None:
|
|
"""Poll (main thread) until *predicate* — the worker-thread run's
|
|
state is only observable between its parks."""
|
|
deadline = time.monotonic() + timeout
|
|
while not predicate():
|
|
if time.monotonic() >= deadline:
|
|
raise AssertionError(f"timed out waiting for {what}")
|
|
time.sleep(0.01)
|
|
|
|
|
|
def test_mid_run_status_reports_chunk_position_between_ticks(
|
|
fresh_sync_status: None,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
tmp_path: Path,
|
|
) -> None:
|
|
"""Phase 129 (task 03): the status carries the in-flight file's
|
|
embed-chunk position — the hook's (first, total) pair climbs
|
|
through the file's batches (1/38 → 13/38, read between ticks),
|
|
paired with the file the ``progress`` hook named; the prelude
|
|
reports 0/0 (no embed batch yet), and the success terminal clears
|
|
the position to 0/0 while the FILE counts keep their final values
|
|
(the convention split — the chunk position is a sub-position of a
|
|
done file, never a keep-final counter)."""
|
|
clone = _GatedClone(threading.Event(), threading.Event())
|
|
import_release = threading.Event()
|
|
chunk_release = threading.Event()
|
|
fake_import = _GatedImport(
|
|
ImportSummary(files=3, added=1, updated=1, unchanged=1),
|
|
threading.Event(),
|
|
import_release,
|
|
chunk_pairs=[(1, 38), (13, 38)], # two batches of a 38-chunk file
|
|
chunk_release=chunk_release,
|
|
)
|
|
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
|
|
|
|
thread, errors = _start_run()
|
|
try:
|
|
assert clone.started.wait(5.0), "the run never reached the clone phase"
|
|
# Prelude: the chunk position is 0/0 (no file, no batch yet —
|
|
# alongside the bare-label pins of the phase-64 test above).
|
|
s = sync_api.sync_status()
|
|
assert s["state"] == "running"
|
|
assert s["current_file"] is None
|
|
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
|
clone.release.set()
|
|
assert fake_import.started.wait(5.0), "the first chunk pair never landed"
|
|
# Import phase, between ticks: pair one is live on the status,
|
|
# paired with the file the progress hook named (the two hooks
|
|
# are paired — the position belongs to that file).
|
|
s = sync_api.sync_status()
|
|
assert s["state"] == "running"
|
|
assert s["phase"] == "import"
|
|
assert s["current_file"] == "repo/notes/deep.md"
|
|
assert s["files_done"] == 1 and s["files_total"] == 3
|
|
assert s["current_chunk"] == 1 and s["chunks_total"] == 38
|
|
# Release pair one: the import fires pair two (13/38 — the
|
|
# second batch of the 38-chunk file) and parks again.
|
|
import_release.set()
|
|
_poll_until(
|
|
lambda: sync_api.sync_status()["current_chunk"] == 13,
|
|
what="the second chunk pair on the status",
|
|
)
|
|
s = sync_api.sync_status()
|
|
assert s["state"] == "running"
|
|
assert s["phase"] == "import"
|
|
# The position CLIMBED (batch granularity) — total unchanged
|
|
# (the file's chunk count, not the batch size).
|
|
assert s["current_chunk"] == 13 and s["chunks_total"] == 38
|
|
assert s["current_file"] == "repo/notes/deep.md"
|
|
assert s["files_done"] == 1 and s["files_total"] == 3
|
|
chunk_release.set()
|
|
finally:
|
|
clone.release.set()
|
|
import_release.set()
|
|
chunk_release.set()
|
|
thread.join(10.0)
|
|
assert not thread.is_alive()
|
|
assert errors == [], f"the runner raised: {errors!r}"
|
|
# Wiring: the pairs fired through the runner's OWN closure (the
|
|
# status reads above came through it; a wiring regression to
|
|
# None would have skipped the firing branch and timed out).
|
|
assert fake_import.chunk_hook_calls == [(1, 38), (13, 38)]
|
|
# Success terminal: the chunk position clears (0/0) ...
|
|
s = sync_api.sync_status()
|
|
assert s["state"] == "success"
|
|
assert s["phase"] is None
|
|
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
|
# ... while the FILE counts keep their final values (the
|
|
# convention split, pinned alongside the phase-64 terminal pin).
|
|
assert s["current_file"] is None
|
|
assert s["files_done"] == 1 and s["files_total"] == 3
|
|
|
|
|
|
def test_chunk_position_clears_at_phase_transitions_and_terminal(
|
|
fresh_sync_status: None,
|
|
monkeypatch: pytest.MonkeyPatch,
|
|
tmp_path: Path,
|
|
) -> None:
|
|
"""Phase 129 (task 03): the chunk position is CLEARED (0/0) at
|
|
every point the import phase ends. This run's summary changes the
|
|
KB, so BOTH post-import transitions run and are pinned: the
|
|
import→``"overview"`` transition and the import→``"summaries"``
|
|
transition (the changed-KB branch) each clear it to 0/0, and the
|
|
success terminal keeps it 0/0 — through the whole span the FILE
|
|
counts keep their final values (the convention split: the file
|
|
counts survive for the result line's context, the chunk position
|
|
is a sub-position of a file that's done)."""
|
|
clone = _GatedClone(threading.Event(), threading.Event())
|
|
import_release = threading.Event()
|
|
overview_started = threading.Event()
|
|
overview_release = threading.Event()
|
|
folders_started = threading.Event()
|
|
folders_release = threading.Event()
|
|
fake_import = _GatedImport(
|
|
ImportSummary(files=3, added=1, updated=1, unchanged=1),
|
|
threading.Event(),
|
|
import_release,
|
|
chunk_pairs=[(7, 38)], # a live pair to clear at each transition
|
|
)
|
|
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
|
|
|
|
# Park the post-import steps (re-monkeypatched over the seams
|
|
# helper's no-ops): the test reads the status DURING the
|
|
# "overview" phase and DURING the "summaries" phase — the two
|
|
# import-phase-ending transitions the task pins.
|
|
async def gated_overview(llm: object, session: object = None) -> bool:
|
|
overview_started.set()
|
|
await asyncio.to_thread(overview_release.wait)
|
|
return True
|
|
|
|
async def gated_folders(
|
|
db: object,
|
|
llm: object,
|
|
*,
|
|
skip: bool = False,
|
|
only_missing: bool = False,
|
|
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
|
|
on_progress: Callable[[int, int, str, str], None] | None = None,
|
|
) -> dict[str, int]:
|
|
folders_started.set()
|
|
await asyncio.to_thread(folders_release.wait)
|
|
return {"generated": 0, "failed": 0, "pruned": 0}
|
|
|
|
monkeypatch.setattr(sync_api, "regenerate_overview", gated_overview)
|
|
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
|
|
|
|
thread, errors = _start_run()
|
|
try:
|
|
assert clone.started.wait(5.0), "the run never reached the clone phase"
|
|
clone.release.set()
|
|
assert fake_import.started.wait(5.0), "the chunk pair never landed"
|
|
# Mid-import: the live pair (7/38) is on the status.
|
|
s = sync_api.sync_status()
|
|
assert s["phase"] == "import"
|
|
assert s["current_chunk"] == 7 and s["chunks_total"] == 38
|
|
# Release the import: it returns (added+updated > 0) → the
|
|
# phase machine moves to "overview" and the position clears.
|
|
import_release.set()
|
|
assert overview_started.wait(5.0), "the run never reached the overview"
|
|
s = sync_api.sync_status()
|
|
assert s["state"] == "running"
|
|
assert s["phase"] == "overview"
|
|
# CLEARED at the import→overview transition (0/0) ... the file
|
|
# counts KEEP their final values (the convention split).
|
|
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
|
assert s["files_done"] == 1 and s["files_total"] == 3
|
|
overview_release.set()
|
|
# ... and the changed-KB branch moves to "summaries" (the
|
|
# second import-phase-ending transition — cleared there too,
|
|
# idempotently) before parking.
|
|
assert folders_started.wait(5.0), "the run never reached the summaries"
|
|
s = sync_api.sync_status()
|
|
assert s["state"] == "running"
|
|
assert s["phase"] == "summaries"
|
|
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
|
assert s["files_done"] == 1 and s["files_total"] == 3
|
|
folders_release.set()
|
|
finally:
|
|
clone.release.set()
|
|
import_release.set()
|
|
overview_release.set()
|
|
folders_release.set()
|
|
thread.join(10.0)
|
|
assert not thread.is_alive()
|
|
assert errors == [], f"the runner raised: {errors!r}"
|
|
assert fake_import.chunk_hook_calls == [(7, 38)]
|
|
# Success terminal: still 0/0 (the terminal clears too — same
|
|
# spots as current_file), file counts kept.
|
|
s = sync_api.sync_status()
|
|
assert s["state"] == "success"
|
|
assert s["phase"] is None
|
|
assert s["current_file"] is None
|
|
assert s["current_chunk"] == 0 and s["chunks_total"] == 0
|
|
assert s["files_done"] == 1 and s["files_total"] == 3
|