phase: 98_sync_summary_visibility
All verification complete. Final report: **Phase 98 — Sync summary visibility: final verification pass** (all 5 tasks already complete; implementation verified against the design, no defects found, no code changes needed) - **Implementation checked:** `SyncStatus` phase machine (4 new keys, terminal-keep counts), `on_progress` hook in `generate_folder_summaries`, `summary_pending` on `KbTreeSource`/`KbTreeFolder` + D3 rule in `build_kb_tree`, phase-aware sync labels + pending UI in `sources.js`, `.kb-summary-pending` CSS — all match decisions D1–D5. - **Unit + integration:** `uv run pytest` → 2184 tests, 0 failed/errors (exit 0) - **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **99%** on `app/` (criterion >90% ✓; `app/api/sync.py` and `app/rag/folder_summaries.py` at 100%) - **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings - **Phase E2E (isolation):** `uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov` → **3 passed** (phase machine, live label, pending markers + gap-fill self-heal) - **Regression suites (each isolated, `--no-cov`):** test_kb_tree ✓, test_ls_tree_drilldown 3 ✓, test_sync_button 3 ✓, test_sync_upload_progress 4 ✓, test_oneshot_llm_retry 2 ✓, test_local_directory_sources 3 ✓ - **Completion criteria:** all 7 verified green — status phase fields + terminal semantics; `Writing KB overview…`/`Summarizing folders… (n/m)` labels (title + aria-live); pending set == `missing_folder_summaries` (integration cross-check pinned at `test_docs_api.py:428`); CLI/`ls` byte-identity (no changes to those paths, pins green); suite/coverage/lint gates; dedicated + regression E2E. Commit left to the harness per protocol (no `git add`/`commit` run). - **Decisions/deviations:** none — no fixes were required this pass. - **Next pending phase:** `99_kb_tree_table_and_back_nav`.
This commit is contained in:
@@ -0,0 +1,958 @@
|
||||
"""Phase 98 task 05 E2E (Playwright, mock-only): the sync's summary
|
||||
phases are visible live — endpoint + button label — and missing folder
|
||||
summaries read "waiting to generate" (the "Summary pending" markers)
|
||||
until a manual save or the next sync's gap-fill makes them go away.
|
||||
|
||||
The dedicated story suite for ``98_sync_summary_visibility`` (A16 — one
|
||||
Playwright file per phase, run in isolation): the owner's report was
|
||||
that while a sync runs "the number pauses for a really long time"
|
||||
(after the import, the KB-overview + the per-folder summary span burn
|
||||
minutes of ``lite`` calls with nothing on the wire) and that a folder
|
||||
whose summary is due but missing looks "missed" in the catalog.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||
deterministic mock (the phase-94 ``FOLDER_SUMMARY_MODE`` canned
|
||||
``Fixture folder summary for <folder>.`` lines) plus the phase-64
|
||||
``tests/e2e/slow_llm.py`` delay proxy that stretches the run past both
|
||||
the ~100 ms recorder cadence and the UI's 2 s poll.
|
||||
|
||||
Timing fixture (the phase-64 sizing rationale, re-sized for the phase
|
||||
labels): ``SLOW_DELAY_S = 1.5`` — ONE LLM call outlives the recorder's
|
||||
~100 ms cadence by ~15×, so every phase span (the overview's one call
|
||||
≈ 1.5 s, the summaries' three ≈ 4.5 s) is densely sampled by the
|
||||
recorder AND outlives the UI's 2 s status poll (any window ≥ the poll
|
||||
period catches a tick — the rendered-label assertions are
|
||||
deterministic, not races). The whole changed sync is ≈ 10 LLM calls
|
||||
(2 model probes + 4 file embeds + 1 overview + 3 folder summaries)
|
||||
≈ 15–18 s — well inside the generous settle budgets.
|
||||
|
||||
The suite runs TWO module apps (the phase-79 full-inventory pattern —
|
||||
each binds its own port) over the shared E2E Postgres:
|
||||
|
||||
* the SLOW app — ``BOR_LLM_BASE_URL`` = the slow proxy in front of the
|
||||
mock: tests 1–2 (the phase machine at the endpoint + the phase-aware
|
||||
button label);
|
||||
* the FAST app — ``BOR_LLM_BASE_URL`` = the mock directly (it stays on
|
||||
its port for the fast tests): test 3 (the pending markers + the
|
||||
gap-fill recovery — the canned summaries are byte-stable).
|
||||
|
||||
KB fixture — a host temp dir (``tmp_path_factory``; the app runs on
|
||||
the same host) registered as a ``kind=local`` source (the
|
||||
``test_local_directory_sources.py`` / phase-94 API-registration
|
||||
pattern; no git anywhere): ``syncsum/`` with TWO ≥ 2-doc folders —
|
||||
``alpha/`` (2 docs) and ``bravo/`` (2 docs), no root-level files. The
|
||||
generator's candidate set is therefore exactly 3: the source root
|
||||
(``folder_path ""``) + the two folders (the recursive-subtree ≥ 2 rule).
|
||||
|
||||
Autouse cleanup (the phase-96 pattern): before each test wait for no
|
||||
running sync on EITHER app, then truncate the shared registry + KB
|
||||
tables (documents/chunks/query_log/steering_notes/kb_overview/
|
||||
git_sources/folder_summaries) — each test seeds its own source and
|
||||
runs its own sync; after each test the same truncate.
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule):
|
||||
1. ``test_sync_status_reports_the_summary_phases`` (endpoint — the
|
||||
slow leg, the phase-64 tight-poll recorder): some running tick has
|
||||
``phase "import"`` (with the live ``current_file`` + counts the
|
||||
import always carried), some have ``phase "overview"``, some have
|
||||
``phase "summaries"`` with ``summaries_total == 3``, non-null
|
||||
``current_summary`` (starting with the source name — one tick the
|
||||
BARE source name, the source-root call) and ``summaries_done``
|
||||
climbing 1 → 2 → 3; every summaries tick keeps
|
||||
``files_done == files_total`` (the import is finished — the
|
||||
reported pause is now labeled); the terminal has ``phase``/
|
||||
``current_summary`` null but keeps ``summaries_done ==
|
||||
summaries_total == 3`` (the keep-final-counts convention).
|
||||
2. ``test_sync_button_names_the_summary_phase`` (UI — the slow leg):
|
||||
a fresh admin page on ``/sources.html`` clicks **Sync sources**;
|
||||
the page's OWN 2 s poll renders ``#sync-label`` matching
|
||||
``Summarizing folders… [folder] (n/3)`` (the asserted text is the
|
||||
RENDERED DOM label, with the button ``title`` riding the same
|
||||
untruncated value — A4); on settle the button reads the terminal
|
||||
``Synced HH:MM`` (the phase-32 contract) and ``#sync-result``
|
||||
carries the counts line.
|
||||
3. ``test_missing_folder_summaries_read_as_pending_and_self_heal``
|
||||
(tree — the fast mock leg): after a successful sync (every
|
||||
candidate stores its canned line) NO marker is anywhere; DELETE
|
||||
one folder's row AND the source-root row directly (the phase-96
|
||||
row-deletion pattern) + re-fetch (the phase-77 re-show refresh) →
|
||||
the two affected rows show ``Summary pending`` (+ the D4 title,
|
||||
the D4 class, the always-present Edit) while the intact folder
|
||||
shows its stored line and NO marker; the source level AND the
|
||||
affected folder's level block show the D4 pending note; saving a
|
||||
manual description from the row's Edit clears the marker IN PLACE
|
||||
(no reload); a second UNCHANGED sync's gap-fill regenerates the
|
||||
OTHER deleted row (the source root) — its marker is gone, its
|
||||
cell/level carry the deterministic mock line, and the manual row
|
||||
is untouched (``only_missing`` + ``manually_edited``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import FolderSummary
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
MOCK_PORT,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||
# port in a combined run — this module's apps bind their own ports
|
||||
# instead (a same-port second uvicorn dies on bind and would drive the
|
||||
# wrong server). Env-overridable.
|
||||
SLOW_APP_PORT = int(os.environ.get("E2E_APP_PORT_SYNCSUM_SLOW", "8142"))
|
||||
FAST_APP_PORT = int(os.environ.get("E2E_APP_PORT_SYNCSUM_FAST", "8143"))
|
||||
SLOW_APP_URL = f"http://127.0.0.1:{SLOW_APP_PORT}"
|
||||
FAST_APP_URL = f"http://127.0.0.1:{FAST_APP_PORT}"
|
||||
|
||||
#: The slow-LLM proxy's port (the conftest's mock LLM stays on
|
||||
#: MOCK_PORT — the FAST app points at it directly for test 3).
|
||||
SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT_SYNCSUM", "8904"))
|
||||
SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}"
|
||||
|
||||
#: Per-LLM-request delay on the proxy — sized so ONE LLM call outlives
|
||||
#: the recorder's ~100 ms cadence by ~15× (the phase-64 rationale):
|
||||
#: the overview span (one call ≈ 1.5 s) and the summaries span (three
|
||||
#: calls ≈ 4.5 s) are densely sampled by the recorder AND outlive the
|
||||
#: UI's 2 s poll period (any window ≥ the period catches a tick — the
|
||||
#: rendered-label assertion is deterministic). A changed sync is
|
||||
#: ≈ 10 calls ≈ 15–18 s (see the module docstring).
|
||||
SLOW_DELAY_S = "1.5"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixture constants (deterministic, token-controlled)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: The local source — the temp directory's basename (``kind=local`` →
|
||||
#: the directory's basename is the source name, phase 38).
|
||||
SOURCE = "syncsum"
|
||||
|
||||
#: The two ≥ 2-doc folders (path order — the tree rows' order).
|
||||
FOLDER_A = "alpha"
|
||||
FOLDER_B = "bravo"
|
||||
FOLDERS = (FOLDER_A, FOLDER_B)
|
||||
|
||||
N_FILES = 4 # two folders × 2 docs
|
||||
N_CANDIDATES = 3 # the source root + the two folders (the ≥ 2-doc rule)
|
||||
|
||||
#: The mock's byte-stable ``FOLDER_SUMMARY_MODE`` lines for this fixture
|
||||
#: (the phase-94 template — the label is the ``FOLDER_HEADER_PREFIX``
|
||||
#: tail: ``<source>`` for the root, ``<source>/<folder>`` for a folder).
|
||||
SUM_ROOT = f"Fixture folder summary for {SOURCE}."
|
||||
SUM_ALPHA = f"Fixture folder summary for {SOURCE}/{FOLDER_A}."
|
||||
SUM_BRavo = f"Fixture folder summary for {SOURCE}/{FOLDER_B}."
|
||||
|
||||
#: The D4 "Summary pending" marker copy (phase 98 task 04 — verbatim
|
||||
#: from the D4 decision): the row-cell text, the row-cell ``title``,
|
||||
#: and the level-block pending note.
|
||||
PENDING_COPY = "Summary pending"
|
||||
PENDING_TITLE = "No stored description yet — the next sync will generate one."
|
||||
PENDING_NOTE = (
|
||||
"No description stored yet — the next sync will generate one. "
|
||||
"(You can write one yourself.)"
|
||||
)
|
||||
|
||||
#: The hand-written description test 3 saves on the affected folder —
|
||||
#: a distinctive sentence no part of the fixture or the canned
|
||||
#: template contains (the round-trip assertions can never pass against
|
||||
#: the old text, the phase-97 pattern).
|
||||
MANUAL_BRavo = (
|
||||
"Owner note: bravo holds the bravo-a and bravo-b fixture files — "
|
||||
"written by hand after the row was deleted. (RESE-SYNCSUM-01)"
|
||||
)
|
||||
|
||||
#: "Synced HH:MM" — the local-time last-result label (sources.js's
|
||||
#: fmtSyncTime), any hour/minute (the test_sync_button.py pattern).
|
||||
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}$")
|
||||
|
||||
#: Generous settle budgets: a changed sync against the slowed LLM is
|
||||
#: ≈ 15–18 s; the UI's 2 s poll settles at most one tick after the
|
||||
#: terminal state lands.
|
||||
SETTLE_TIMEOUT_MS = 90_000
|
||||
|
||||
|
||||
def _md(title: str, body: str) -> str:
|
||||
return f"# {title}\n\n{body}\n"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def syncsum_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The one-source temp tree (see the module docstring): the app
|
||||
servers run on the same host, so the paths are visible to them.
|
||||
The directory NAME is the source name (``kind=local``, phase 38)."""
|
||||
root = tmp_path_factory.mktemp("bor_synccom")
|
||||
src = root / SOURCE
|
||||
for folder in FOLDERS:
|
||||
(src / folder).mkdir(parents=True)
|
||||
for letter in ("a", "b"):
|
||||
(src / folder / f"{folder}-{letter}.md").write_text(
|
||||
_md(
|
||||
f"Synccom {folder.title()} {letter.upper()}",
|
||||
f"Synccom {folder} fixture note {letter.upper()}: covers "
|
||||
f"topic {letter.upper()} of the {SOURCE} source tree.",
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert len(list(src.rglob("*.md"))) == N_FILES
|
||||
return src
|
||||
|
||||
|
||||
def _app_env(base_url: str, tmp_path_factory: pytest.TempPathFactory) -> dict[str, str]:
|
||||
"""The per-module app env (the conftest pattern, cf.
|
||||
``test_oneshot_llm_retry.py`` / ``test_kb_tree.py``): the
|
||||
``base_url`` picks the LLM (slow proxy vs. the mock directly),
|
||||
``BOR_GIT_SOURCES`` forced empty (the sync source is this suite's
|
||||
own DB-registered local row), and the leak-guarded code defaults."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url
|
||||
)
|
||||
# Mock-calibrated threshold (conftest pattern): this suite never
|
||||
# asks the chat model anything — the gate is never on a path.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
# Phase 67: instant retry waits + the code-default budget (the
|
||||
# conftest leak-guard pattern).
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
||||
# reads it from cwd) — override it with an EMPTY value (the env var
|
||||
# beats the .env file): the registry must hold EXACTLY the local
|
||||
# directory this suite registers.
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def slow_llm(mock_llm: int) -> Iterator[int]:
|
||||
"""The delay-injecting reverse proxy in front of the mock LLM
|
||||
(``tests/e2e/slow_llm.py``) — the timing fixture for tests 1–2:
|
||||
the phase-machine ticks and the rendered phase label need the run
|
||||
to outlive both the ~100 ms recorder cadence and the UI's 2 s
|
||||
poll (see ``SLOW_DELAY_S``)."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S
|
||||
env["E2E_MOCK_PORT"] = str(MOCK_PORT)
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app",
|
||||
"--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{SLOW_URL}/v1/models")
|
||||
yield SLOW_PORT
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(
|
||||
mock_llm: int,
|
||||
slow_llm: int,
|
||||
syncsum_dir: Path,
|
||||
tmp_path_factory: pytest.TempPathFactory,
|
||||
) -> Iterator[str]:
|
||||
"""The SLOW app under test (tests 1–2): ``BOR_LLM_BASE_URL`` is the
|
||||
slow proxy in front of the mock (the timing fixture — see the
|
||||
module docstring). ``syncsum_dir`` is a dependency only for the
|
||||
fixture ordering (the temp tree exists before the first test)."""
|
||||
env = _app_env(f"{SLOW_URL}/v1", tmp_path_factory)
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(SLOW_APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{SLOW_APP_URL}/api/health")
|
||||
yield SLOW_APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def fast_app_server(
|
||||
mock_llm: int,
|
||||
syncsum_dir: Path,
|
||||
tmp_path_factory: pytest.TempPathFactory,
|
||||
) -> Iterator[str]:
|
||||
"""The FAST app under test (test 3): ``BOR_LLM_BASE_URL`` is the
|
||||
mock DIRECTLY (it stays on its port for the fast tests — the
|
||||
canned ``FOLDER_SUMMARY_MODE`` lines are byte-stable there, and
|
||||
the pending-marker / gap-fill assertions key on that exact text)."""
|
||||
env = _app_env(f"http://127.0.0.1:{mock_llm}/v1", tmp_path_factory)
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(FAST_APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{FAST_APP_URL}/api/health")
|
||||
yield FAST_APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def fast_app_url(fast_app_server: str) -> str:
|
||||
return fast_app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
|
||||
share one Postgres, so a leftover git_sources row or document would
|
||||
pollute the top-level rows, the candidate count, and the counts
|
||||
the syncs and the tree assertions pin exactly."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit() # without the commit the TRUNCATE rolls back (the house pattern)
|
||||
|
||||
|
||||
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
|
||||
"""No background sync may leak across tests (the run state lives in
|
||||
the app's memory, and a still-running run would keep importing into
|
||||
the NEXT test's truncated KB): wait for the app's sync status to
|
||||
be non-running BEFORE the truncate. Own admin session (the
|
||||
endpoint is admin-only) — usually a no-op: the tests settle only
|
||||
after their run's terminal state."""
|
||||
with httpx.Client(base_url=base_url, timeout=5.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
if r.json()["state"] != "running":
|
||||
return
|
||||
time.sleep(0.2)
|
||||
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(app_url: str, fast_app_url: str, db_ready: None) -> Iterator[None]:
|
||||
"""Per-test KB isolation (the phase-96 cleanup pattern): the
|
||||
shared E2E Postgres is truncated before AND after every test —
|
||||
each test seeds its own source and runs its own sync(s), so the
|
||||
KB rows + folder_summaries rows of the temp source are exactly
|
||||
this test's own. The pre-wait keeps a leaked run from the
|
||||
previous test (module apps are shared) from racing the truncate."""
|
||||
for base in (app_url, fast_app_url):
|
||||
_wait_sync_settled(base)
|
||||
_truncate_all()
|
||||
yield
|
||||
_truncate_all()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _register_source(base_url: str, path: str) -> None:
|
||||
"""Register the temp directory through the authenticated API (the
|
||||
``test_local_directory_sources.py`` / phase-94 pattern — a plain
|
||||
directory is an API/DB-only operation)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": path}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
|
||||
|
||||
def _run_sync(base_url: str, timeout_s: float = 120.0) -> dict[str, Any]:
|
||||
"""Login + ``POST /api/sync`` + poll the status endpoint until the
|
||||
run reaches a terminal state (the ``test_local_directory_sources``
|
||||
pattern, over plain httpx)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
def _folder_rows() -> dict[str, tuple[str, bool]]:
|
||||
"""The source's stored folder summaries
|
||||
``{folder_path: (summary, manually_edited)}`` (``""`` = the source
|
||||
root) — the test process's direct DB access (the E2E's other
|
||||
established lens)."""
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(
|
||||
select(
|
||||
FolderSummary.folder_path,
|
||||
FolderSummary.summary,
|
||||
FolderSummary.manually_edited,
|
||||
).where(FolderSummary.source == SOURCE)
|
||||
).all()
|
||||
return {
|
||||
folder: (summary, manually)
|
||||
for folder, summary, manually in rows
|
||||
}
|
||||
|
||||
|
||||
def _delete_folder_rows(pairs: list[tuple[str, str]]) -> None:
|
||||
"""DELETE stored ``folder_summaries`` rows directly (the phase-96
|
||||
row-deletion pattern — simulating a historical fail-soft miss):
|
||||
the rows are gone from the app's back without any KB change."""
|
||||
with SessionLocal() as db:
|
||||
for source, folder in pairs:
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries "
|
||||
"WHERE source = :s AND folder_path = :f"
|
||||
),
|
||||
{"s": source, "f": folder},
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
class _TickRecorder:
|
||||
"""The deterministic layer, concurrent with the test's own client.
|
||||
|
||||
A daemon thread that tight-polls (~100 ms cadence)
|
||||
``GET /api/sync/status`` with its OWN admin session (``httpx`` —
|
||||
the test's client drives itself in the meantime; the Playwright
|
||||
sync API is not thread-safe, so the thread never touches it),
|
||||
recording every tick from the first poll: the idle prelude, the
|
||||
running ticks, and the terminal body. The thread only READS the
|
||||
same endpoint the UI's 2 s poll reads — it starts no jobs and
|
||||
cannot skew the run. (The phase-64 recorder pattern.)
|
||||
|
||||
The stale-terminal guard (the module apps are shared — a previous
|
||||
test's terminal lives in the app's memory): a terminal is
|
||||
accepted only after a RUNNING tick was observed (this suite's runs
|
||||
outlive the 100 ms cadence by ~15× — ``require_running`` is
|
||||
always on here)."""
|
||||
|
||||
def __init__(self, app_url: str, path: str) -> None:
|
||||
self._url = f"{app_url}{path}"
|
||||
self._login_url = f"{app_url}/api/login"
|
||||
self._t0 = time.time()
|
||||
self._ticks: list[dict[str, Any]] = []
|
||||
self._terminal: dict[str, Any] | None = None
|
||||
self._stop = threading.Event()
|
||||
self._thread: threading.Thread | None = None
|
||||
|
||||
def _terminal_is_this_run(self, body: dict[str, Any], saw_running: bool) -> bool:
|
||||
if saw_running:
|
||||
return True
|
||||
started = body.get("started_at")
|
||||
if not started:
|
||||
return False
|
||||
try:
|
||||
started_at = datetime.fromisoformat(str(started)).timestamp()
|
||||
except ValueError:
|
||||
return False
|
||||
return started_at >= self._t0 - 2.0 # tolerance for the pre-submit gap
|
||||
|
||||
def start(self) -> None:
|
||||
def run() -> None:
|
||||
with httpx.Client(timeout=5.0) as client:
|
||||
# Its own admin session — the recorder's reads must not
|
||||
# depend on (or disturb) the test's client's cookie.
|
||||
client.post(self._login_url, json={"password": ADMIN_PASSWORD})
|
||||
saw_running = False
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
r = client.get(self._url)
|
||||
if r.status_code == 200:
|
||||
body = r.json()
|
||||
self._ticks.append(body)
|
||||
if body["state"] == "running":
|
||||
saw_running = True
|
||||
elif (
|
||||
body["state"] in ("success", "failed")
|
||||
and self._terminal_is_this_run(body, saw_running)
|
||||
):
|
||||
self._terminal = body
|
||||
return
|
||||
except Exception: # noqa: BLE001 — blip: retry next tick
|
||||
pass
|
||||
self._stop.wait(0.1)
|
||||
|
||||
self._thread = threading.Thread(target=run, daemon=True)
|
||||
self._thread.start()
|
||||
|
||||
def stop(self, timeout_s: float = 120.0) -> dict[str, Any]:
|
||||
"""Join until a terminal tick is recorded (or fail the test)."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while time.monotonic() < deadline:
|
||||
if self._terminal is not None:
|
||||
break
|
||||
time.sleep(0.05)
|
||||
self._stop.set()
|
||||
if self._thread is not None:
|
||||
self._thread.join(timeout=5)
|
||||
assert self._terminal is not None, (
|
||||
"the recorder saw no terminal state "
|
||||
f"(last ticks: {self._ticks[-3:] if self._ticks else 'none'})"
|
||||
)
|
||||
return self._terminal
|
||||
|
||||
@property
|
||||
def running_ticks(self) -> list[dict[str, Any]]:
|
||||
return [t for t in self._ticks if t["state"] == "running"]
|
||||
|
||||
|
||||
def _wait_top_level(page: Page) -> None:
|
||||
"""The admin boot / re-fetch has rendered the top level: the single
|
||||
source row is in the folders table (the tree's fetch settled) and
|
||||
the breadcrumb is hidden (the top level is home)."""
|
||||
expect(page.locator("#folders-tbody tr")).to_have_count(1, timeout=30_000)
|
||||
expect(page.locator("#kb-crumb")).to_be_hidden()
|
||||
|
||||
|
||||
def _drill(page: Page, *names: str) -> None:
|
||||
"""Drill one level at a time (client-side — no fetch, no URL
|
||||
change): each name is the EXACT text of the source/folder link at
|
||||
the current level (the row builders' link text: the source name,
|
||||
or the folder's last path segment — the phase-97 pattern)."""
|
||||
for name in names:
|
||||
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Endpoint — the phase machine: import keeps the file label, the
|
||||
# overview is named, the folder span reports the folder + a
|
||||
# done/total that climbs to the candidate count
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_sync_status_reports_the_summary_phases(
|
||||
app_url: str, syncsum_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""A KB-changing sync of the temp source (candidates: the source
|
||||
root + the two folders = 3) is tight-polled by the ~100 ms
|
||||
recorder from the 202 to the terminal (the phase-64 pattern): the
|
||||
recorded ticks show the D1 phase machine end to end — the
|
||||
model-check prelude keeps ``phase`` null, the import keeps its
|
||||
byte-identical live-file fields with ``phase "import"``, the
|
||||
overview span is named, and the folder span carries
|
||||
``current_summary`` + ``summaries_done``/``summaries_total``
|
||||
(climbing 1 → 2 → 3) with ``files_done == files_total`` (the
|
||||
import is finished — the pause the user reported) — while the
|
||||
terminal clears ``phase``/``current_summary`` but KEEPS the final
|
||||
summary counts (the phase-64 keep-final-counts convention)."""
|
||||
_register_source(app_url, str(syncsum_dir))
|
||||
|
||||
recorder = _TickRecorder(app_url, "/api/sync/status")
|
||||
recorder.start()
|
||||
with httpx.Client(base_url=app_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
terminal = recorder.stop(timeout_s=120)
|
||||
|
||||
# --- the terminal (D1: clear the phase + folder, keep the final
|
||||
# summary counts; the phase-64 keep-final-counts convention extends
|
||||
# to the file counts) ---
|
||||
assert terminal["state"] == "success", terminal
|
||||
detail = terminal["detail"]
|
||||
assert detail["added"] == N_FILES, detail
|
||||
assert terminal["current_file"] is None
|
||||
assert terminal["files_done"] == N_FILES
|
||||
assert terminal["files_total"] == N_FILES
|
||||
assert terminal["phase"] is None, terminal
|
||||
assert terminal["current_summary"] is None, terminal
|
||||
assert terminal["summaries_done"] == N_CANDIDATES, terminal
|
||||
assert terminal["summaries_total"] == N_CANDIDATES, terminal
|
||||
|
||||
running = recorder.running_ticks
|
||||
assert running, "the recorder caught no running tick"
|
||||
|
||||
# --- the prelude (model check): ``phase`` null — the bare
|
||||
# "Syncing…" label stays (the phase-64 pins hold). Two probes ×
|
||||
# 1.5 s under the slow proxy: the recorder cannot miss it. ---
|
||||
prelude = [t for t in running if t["phase"] is None]
|
||||
assert prelude, f"no phase-less prelude tick: {running[:6]}"
|
||||
assert all(t["current_file"] is None for t in prelude), prelude
|
||||
|
||||
# --- the import phase: the byte-identical file label + counts
|
||||
# (phase 64), now carrying ``phase "import"`` (D1) ---
|
||||
imp = [t for t in running if t["phase"] == "import"]
|
||||
assert imp, f"no import-phase tick: {[t['phase'] for t in running]}"
|
||||
with_file = [t for t in imp if t["current_file"]]
|
||||
assert with_file, f"no import tick carried a current_file: {imp[:6]}"
|
||||
assert all(
|
||||
t["current_file"].startswith(f"{SOURCE}/") for t in with_file
|
||||
), with_file
|
||||
assert all(t["files_total"] == N_FILES for t in with_file), with_file
|
||||
dones = [t["files_done"] for t in with_file]
|
||||
assert dones == sorted(dones), f"files_done not monotonic: {dones}"
|
||||
assert dones[0] >= 1 and max(dones) == N_FILES, dones
|
||||
|
||||
# --- the overview phase: named (D1) ---
|
||||
ov = [t for t in running if t["phase"] == "overview"]
|
||||
assert ov, f"no overview-phase tick: {[t['phase'] for t in running]}"
|
||||
|
||||
# --- the summaries phase (D1/D5): the folder being summarized +
|
||||
# the done/total through the long span ---
|
||||
sm = [t for t in running if t["phase"] == "summaries"]
|
||||
assert sm, f"no summaries-phase tick: {[t['phase'] for t in running]}"
|
||||
# EVERY summaries tick: the import is finished — the file count
|
||||
# sits still (the reported pause is now labeled).
|
||||
for t in sm:
|
||||
assert t["files_done"] == t["files_total"] == N_FILES, t
|
||||
# The steady ticks (after the generator's loop starts): total is
|
||||
# the candidate count and the folder part is non-null, starting
|
||||
# with the source name (D1's shape).
|
||||
steady = [t for t in sm if t["summaries_total"] == N_CANDIDATES]
|
||||
assert steady, f"no steady summaries tick (total {N_CANDIDATES}): {sm[:4]}"
|
||||
for t in steady:
|
||||
assert t["current_summary"], t
|
||||
assert t["current_summary"].startswith(SOURCE), t
|
||||
# The per-done positions: the sorted candidate order is
|
||||
# (root, alpha, bravo) — the bare source name is the ROOT call,
|
||||
# the folders ride ``source/folder``. Under the 15× sizing each
|
||||
# position is held ≈ 1.5 s, so the recorded progression is exactly
|
||||
# 1 → 2 → 3 (monotonic, every value present, never revisited).
|
||||
by_done: dict[int, list[str]] = {}
|
||||
for t in steady:
|
||||
by_done.setdefault(t["summaries_done"], []).append(t["current_summary"])
|
||||
assert sorted(by_done) == [1, 2, 3], f"summaries_done progression: {sorted(by_done)}"
|
||||
assert set(by_done[1]) == {SOURCE}, by_done[1] # the bare source name — the root
|
||||
assert set(by_done[2]) == {f"{SOURCE}/{FOLDER_A}"}, by_done[2]
|
||||
assert set(by_done[3]) == {f"{SOURCE}/{FOLDER_B}"}, by_done[3]
|
||||
|
||||
# --- the phase machine's ORDER: null → import → overview →
|
||||
# summaries (summaries is the last phase — the run goes straight
|
||||
# to the bump/terminal after it) ---
|
||||
phased = [t for t in running if t["phase"] is not None]
|
||||
assert phased[0]["phase"] == "import", phased[0]
|
||||
first_sm = next(
|
||||
i for i, t in enumerate(phased) if t["phase"] == "summaries"
|
||||
)
|
||||
assert any(t["phase"] == "overview" for t in phased[:first_sm])
|
||||
assert all(t["phase"] == "summaries" for t in phased[first_sm:])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. UI — the sync button names the phase: the page's own 2 s poll
|
||||
# renders the summaries label (folder + counts), then the phase-32
|
||||
# terminal settle
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_sync_button_names_the_summary_phase(
|
||||
page: Page, app_url: str, syncsum_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""A fresh admin page on ``/sources.html`` clicks **Sync sources**;
|
||||
the page's OWN 2 s poll renders ``#sync-label`` as
|
||||
``Summarizing folders… [source/folder] (n/3)`` (asserted on the
|
||||
RENDERED text — the label is built by the page JS — with the
|
||||
button ``title`` carrying the same untruncated value, A4); on
|
||||
settle the button shows the terminal ``Synced HH:MM`` (the
|
||||
phase-32 contract) and ``#sync-result`` carries the counts line.
|
||||
The prelude/import labels stay the phase-64 shape (the endpoint
|
||||
test pins those phases; the UI pins the new summaries label)."""
|
||||
page.set_default_timeout(30_000)
|
||||
_register_source(app_url, str(syncsum_dir))
|
||||
login(page, app_url) # lands on /sources.html (the button's home)
|
||||
|
||||
btn = page.locator("#sync-btn")
|
||||
expect(btn).to_be_visible(timeout=30_000)
|
||||
# The button is settled and retry-ready at boot: idle ("Sync
|
||||
# sources") — or the RE-ATTACHED terminal from the previous
|
||||
# test's sync (the module app is shared, its in-memory sync state
|
||||
# survives the test boundary, and the phase-32 boot re-attach
|
||||
# renders the last result). Waiting on the settled label also
|
||||
# proves the boot re-attach COMPLETED (no later re-render can
|
||||
# clobber the running state) — the phase-64 pattern.
|
||||
expect(page.locator("#sync-label")).to_have_text(
|
||||
re.compile(r"^(Sync sources|Synced \d{1,2}:\d{2})$")
|
||||
)
|
||||
expect(btn).to_be_enabled()
|
||||
|
||||
btn.click()
|
||||
|
||||
# The click's immediate state (the 202 moment — the bare prefix,
|
||||
# the phase-64 contract): disabled, aria-busy, spinning icon, no
|
||||
# error…
|
||||
expect(btn).to_be_disabled()
|
||||
expect(btn).to_have_attribute("aria-busy", "true")
|
||||
expect(btn.locator(".sync-icon")).to_have_class(re.compile(r"\bis-spinning\b"))
|
||||
expect(page.locator("#sync-label")).to_have_text("Syncing…")
|
||||
expect(page.locator("#sync-error-banner")).to_be_hidden()
|
||||
|
||||
# UI layer (D2): the page's own 2 s poll names the summaries phase
|
||||
# — the RENDERED label matches the D2 shape (the folder part is
|
||||
# optional — the phase's first poll may precede the first hook)
|
||||
# with the (n/3) counts, and the button ``title`` carries the same
|
||||
# untruncated value (A4). ``wait_for_function`` checks BOTH in ONE
|
||||
# in-browser evaluation: a poll tick updates label + title together
|
||||
# in a single JS task, so the atomic check can never straddle one
|
||||
# (the JS regex is the D2 shape, unit-pinned in
|
||||
# tests/unit/test_frontend_sync_upload.py).
|
||||
page.wait_for_function(
|
||||
"""() => {
|
||||
const label = document.querySelector('#sync-label');
|
||||
const btn = document.querySelector('#sync-btn');
|
||||
if (!label || !btn) return false;
|
||||
const text = label.textContent;
|
||||
const re = /^Summarizing folders…(?: \\S+)? \\(\\d+\\/3\\)$/;
|
||||
return re.test(text) && btn.getAttribute('title') === text;
|
||||
}""",
|
||||
timeout=SETTLE_TIMEOUT_MS,
|
||||
)
|
||||
|
||||
# The success settle (the phase-32 contract, preserved): "Synced
|
||||
# HH:MM" + the counts result line, the button re-enabled, no
|
||||
# error — the run's whole span (prelude + import + overview +
|
||||
# summaries) was labeled along the way (test 1 pins the rest).
|
||||
expect(page.locator("#sync-label")).to_have_text(
|
||||
SYNCED_LABEL, timeout=60_000
|
||||
)
|
||||
expect(btn).to_be_enabled()
|
||||
expect(btn).not_to_have_attribute("aria-busy")
|
||||
expect(page.locator("#sync-result")).to_have_text(f"{N_FILES} added")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. Tree — the pending markers: missing rows read "waiting", a manual
|
||||
# save clears the marker in place, and the next unchanged sync's
|
||||
# gap-fill heals the other deleted row
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_missing_folder_summaries_read_as_pending_and_self_heal(
|
||||
page: Page, fast_app_url: str, syncsum_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""After a successful (fast mock) sync — every candidate stores
|
||||
its deterministic canned line — NO ``Summary pending`` is
|
||||
anywhere. DELETE one folder's row AND the source-root row directly
|
||||
(the phase-96 pattern) + re-fetch (the phase-77 re-show refresh) →
|
||||
the two affected rows show ``Summary pending`` (+ the D4 title,
|
||||
the D4 class, the always-present Edit) while the intact folder
|
||||
shows its stored line and NO marker; the source level AND the
|
||||
affected folder's level block show the D4 pending note. Saving a
|
||||
manual description from the row's Edit clears the marker IN PLACE
|
||||
(the cell shows the saved text — no reload). A second UNCHANGED
|
||||
sync's gap-fill then regenerates the OTHER deleted row (the
|
||||
source root): its marker is gone, its cell/level carry the
|
||||
deterministic mock line — and the manual row is untouched
|
||||
(``only_missing`` + ``manually_edited``)."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, fast_app_url) # lands on /sources.html (the RAG view)
|
||||
|
||||
# The changed sync under the fast mock: every candidate (root +
|
||||
# the two folders) stores its canned line; the terminal keeps the
|
||||
# full-regeneration summary counts.
|
||||
_register_source(fast_app_url, str(syncsum_dir))
|
||||
body = _run_sync(fast_app_url)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == N_FILES, detail
|
||||
assert detail["updated"] == 0 and detail["pruned"] == 0, detail
|
||||
assert detail["overview"] is True, detail
|
||||
assert body["summaries_done"] == N_CANDIDATES, body
|
||||
assert body["summaries_total"] == N_CANDIDATES, body
|
||||
# The full regeneration pinned in the DB: one AI row per candidate.
|
||||
assert _folder_rows() == {
|
||||
"": (SUM_ROOT, False),
|
||||
FOLDER_A: (SUM_ALPHA, False),
|
||||
FOLDER_B: (SUM_BRavo, False),
|
||||
}
|
||||
|
||||
# The re-fetch (the phase-77 re-show refresh — the nav re-click):
|
||||
# the synced catalog renders, and NO marker is anywhere — every
|
||||
# description-bearing row has a stored line.
|
||||
page.click("#nav-sources")
|
||||
_wait_top_level(page)
|
||||
src_row = page.locator("#folders-tbody tr")
|
||||
expect(src_row.locator("a.folder-link")).to_have_text(SOURCE)
|
||||
expect(src_row.locator("td:nth-child(2)")).to_have_text(str(N_FILES))
|
||||
expect(src_row.locator("td:nth-child(3) span")).to_have_text(SUM_ROOT)
|
||||
expect(page.locator(".kb-summary-pending")).to_have_count(0)
|
||||
|
||||
# The gap: delete the bravo row AND the source-root row directly
|
||||
# (a historical fail-soft miss — the rows are gone from the app's
|
||||
# back, no KB change anywhere).
|
||||
_delete_folder_rows([(SOURCE, FOLDER_B), (SOURCE, "")])
|
||||
assert set(_folder_rows()) == {FOLDER_A}
|
||||
|
||||
# The re-fetch: the two affected rows carry the D4 marker (the
|
||||
# pending set = exactly the missing rows — the D3 concept), the
|
||||
# intact folder shows its stored line and NO marker.
|
||||
page.click("#nav-sources")
|
||||
_wait_top_level(page)
|
||||
src_row = page.locator("#folders-tbody tr")
|
||||
s_span = src_row.locator("td:nth-child(3) span")
|
||||
expect(s_span).to_have_text(PENDING_COPY)
|
||||
expect(s_span).to_have_class("kb-summary-pending")
|
||||
expect(s_span).to_have_attribute("title", PENDING_TITLE)
|
||||
# The Edit button is KEPT with the marker (a manual save creates
|
||||
# the row and clears the marker in place).
|
||||
expect(src_row.locator(".kb-summary-edit")).to_be_visible()
|
||||
|
||||
_drill(page, SOURCE)
|
||||
# The SOURCE level itself is pending too (its root row is gone) —
|
||||
# the level block shows the D4 pending note (title = the source
|
||||
# name — the source root is folder "").
|
||||
expect(page.locator("#kb-level")).to_be_visible()
|
||||
expect(page.locator("#kb-level-title")).to_have_text(SOURCE)
|
||||
expect(page.locator("#kb-level-summary")).to_have_text(PENDING_NOTE)
|
||||
rows = page.locator("#folders-tbody tr")
|
||||
expect(rows).to_have_count(2)
|
||||
alpha_row = rows.nth(0)
|
||||
bravo_row = rows.nth(1)
|
||||
expect(alpha_row.locator("a.folder-link")).to_have_text(FOLDER_A)
|
||||
# The intact folder: the stored line, never the marker.
|
||||
expect(alpha_row.locator("td:nth-child(3) span")).to_have_text(SUM_ALPHA)
|
||||
expect(alpha_row.locator("td:nth-child(3) span")).not_to_have_class(
|
||||
"kb-summary-pending"
|
||||
)
|
||||
# The affected folder: the marker (copy + class + title).
|
||||
b_span = bravo_row.locator("td:nth-child(3) span")
|
||||
expect(b_span).to_have_text(PENDING_COPY)
|
||||
expect(b_span).to_have_class("kb-summary-pending")
|
||||
expect(b_span).to_have_attribute("title", PENDING_TITLE)
|
||||
|
||||
# Drill into the affected folder: ITS level block shows the D4
|
||||
# pending note (title = the full source-relative path).
|
||||
bravo_row.locator("a.folder-link").click()
|
||||
expect(page.locator("#kb-level-title")).to_have_text(f"{SOURCE}/{FOLDER_B}")
|
||||
expect(page.locator("#kb-level-summary")).to_have_text(PENDING_NOTE)
|
||||
|
||||
# The row's Edit → Save a manual description → the marker is gone
|
||||
# IN PLACE (the cell shows the saved text — no reload, the D4
|
||||
# in-place clear).
|
||||
page.click(f'#kb-crumb a.kb-crumb-link:text-is("{SOURCE}")') # up to the source level
|
||||
bravo_row = page.locator("#folders-tbody tr").nth(1)
|
||||
expect(bravo_row.locator("a.folder-link")).to_have_text(FOLDER_B)
|
||||
bravo_row.locator(".kb-summary-edit").click()
|
||||
page.fill(".kb-summary-editor", MANUAL_BRavo)
|
||||
page.click(".kb-summary-save")
|
||||
expect(bravo_row.locator(".kb-summary-status")).to_have_text(
|
||||
"Description updated."
|
||||
)
|
||||
b_span = bravo_row.locator("td:nth-child(3) span")
|
||||
expect(b_span).to_have_text(MANUAL_BRavo)
|
||||
expect(b_span).not_to_have_class("kb-summary-pending")
|
||||
|
||||
# The second (UNCHANGED-KB) sync: the gap probe finds exactly the
|
||||
# source root missing (alpha stored, bravo manual) → the targeted
|
||||
# fill regenerates the root row only.
|
||||
body = _run_sync(fast_app_url)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == 0, detail
|
||||
assert detail["updated"] == 0 and detail["pruned"] == 0, detail
|
||||
assert detail["overview"] is False, detail # unchanged → no overview burn
|
||||
# The terminal keeps the gap-fill's final counts (ONE missing row).
|
||||
assert body["summaries_done"] == 1, body
|
||||
assert body["summaries_total"] == 1, body
|
||||
# The DB pins the targeted fill: the root is BACK with its canned
|
||||
# text (AI-written again), bravo keeps the MANUAL row (untouched —
|
||||
# a regeneration would have rewritten the owner's words), alpha
|
||||
# untouched.
|
||||
rows_db = _folder_rows()
|
||||
assert rows_db[""] == (SUM_ROOT, False), rows_db
|
||||
assert rows_db[FOLDER_B] == (MANUAL_BRavo, True), rows_db
|
||||
assert rows_db[FOLDER_A] == (SUM_ALPHA, False), rows_db
|
||||
|
||||
# The re-fetch: the marker is GONE — the current level (the source
|
||||
# root) shows the regenerated canned line in its level block, the
|
||||
# folder rows show the stored texts, and no marker anywhere.
|
||||
page.click("#nav-sources")
|
||||
expect(page.locator("#kb-level")).to_be_visible()
|
||||
expect(page.locator("#kb-level-title")).to_have_text(SOURCE)
|
||||
expect(page.locator("#kb-level-summary")).to_have_text(SUM_ROOT)
|
||||
rows = page.locator("#folders-tbody tr")
|
||||
expect(rows.nth(0).locator("td:nth-child(3) span")).to_have_text(SUM_ALPHA)
|
||||
expect(rows.nth(1).locator("td:nth-child(3) span")).to_have_text(MANUAL_BRavo)
|
||||
expect(page.locator(".kb-summary-pending")).to_have_count(0)
|
||||
# And at the top level: the source row's cell carries the
|
||||
# regenerated root line, no marker.
|
||||
page.locator("#kb-crumb a.kb-crumb-link").nth(0).click()
|
||||
src_row = page.locator("#folders-tbody tr")
|
||||
s_span = src_row.locator("td:nth-child(3) span")
|
||||
expect(s_span).to_have_text(SUM_ROOT)
|
||||
expect(s_span).not_to_have_class("kb-summary-pending")
|
||||
expect(page.locator(".kb-summary-pending")).to_have_count(0)
|
||||
@@ -14,7 +14,7 @@ from datetime import UTC, datetime, timedelta
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import func, select, text
|
||||
from sqlalchemy import delete, func, select, text
|
||||
|
||||
import app.api.docs as docs_api
|
||||
from app.config import Settings
|
||||
@@ -22,6 +22,7 @@ from app.core import tokens as token_service
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import Chunk, Document, FolderSummary, GitSource
|
||||
from app.rag import git_sources as rag_git_sources
|
||||
from app.rag.folder_summaries import missing_folder_summaries
|
||||
|
||||
_TREE_TABLES = "chunks, documents, folder_summaries, git_sources"
|
||||
|
||||
@@ -65,6 +66,25 @@ def _tree_file_nodes(sources) -> list[dict]:
|
||||
return files
|
||||
|
||||
|
||||
def _tree_pending_keys(sources) -> set[tuple[str, str]]:
|
||||
"""Every ``(source, folder_path)`` flagged ``summary_pending`` in a
|
||||
tree response — the SOURCE root rides ``folder_path = ""`` (the
|
||||
``folder_summaries`` convention); walked recursively over the whole
|
||||
tree (the D3 set the cross-check compares against
|
||||
:func:`missing_folder_summaries`)."""
|
||||
keys: set[tuple[str, str]] = set()
|
||||
|
||||
def _walk(source_name: str, node: dict) -> None:
|
||||
if node.get("summary_pending"):
|
||||
keys.add((source_name, node["path"] if node.get("kind") == "folder" else ""))
|
||||
for child in node.get("children", ()):
|
||||
_walk(source_name, child)
|
||||
|
||||
for source in sources:
|
||||
_walk(source["name"], source)
|
||||
return keys
|
||||
|
||||
|
||||
def test_docs_empty_shape(admin_client, db) -> None:
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
@@ -177,9 +197,12 @@ def test_docs_tree_populated_shape_order_counts_summaries(admin_client, db) -> N
|
||||
assert [s["name"] for s in sources] == ["Homelab", "Deployments"]
|
||||
|
||||
homelab, deployments = sources
|
||||
assert set(homelab) == {"name", "documents", "summary", "children"}
|
||||
# Wire-additive (phase 98, task 03): the pre-pending keys are all
|
||||
# still there, joined by ``summary_pending``.
|
||||
assert set(homelab) == {"name", "documents", "summary", "summary_pending", "children"}
|
||||
assert homelab["documents"] == 4 # the whole recursive count
|
||||
assert homelab["summary"] == "Homelab docs." # the (source, "") row
|
||||
assert homelab["summary_pending"] is False # the stored root row covers it
|
||||
assert deployments["summary"] is None # no stored root row
|
||||
assert deployments["documents"] == 2
|
||||
|
||||
@@ -276,13 +299,147 @@ def test_docs_tree_indexed_only_source_after_registered(admin_client, db) -> Non
|
||||
assert [s["name"] for s in sources] == ["Alpha", "Midx", "Zeta"]
|
||||
alpha, midx, zeta = sources
|
||||
assert (alpha["documents"], alpha["children"], alpha["summary"]) == (0, [], None)
|
||||
assert alpha["summary_pending"] is False # 0 documents — never pending
|
||||
assert midx["documents"] == 1
|
||||
assert midx["summary_pending"] is False # 1 document — below the minimum
|
||||
assert zeta["documents"] == 1
|
||||
assert zeta["summary_pending"] is False # 1 document — below the minimum
|
||||
assert [c["path"] for c in midx["children"]] == ["m1.md"]
|
||||
|
||||
_truncate_tree_tables(db)
|
||||
|
||||
|
||||
def test_docs_tree_summary_pending_on_source_and_folder_nodes(admin_client, db) -> None:
|
||||
"""The endpoint returns ``summary_pending`` on SOURCE + FOLDER
|
||||
nodes (phase 98, task 03 — D3): true iff the recursive count is
|
||||
≥ 2 AND no stored row; false WITH a stored row (any — the endpoint
|
||||
cannot tell AI from manual); false for a < 2-document folder
|
||||
(never pending) — including one NESTED. File nodes carry no flag."""
|
||||
_truncate_tree_tables(db)
|
||||
base = datetime.now(UTC)
|
||||
db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git", added_at=base))
|
||||
# k8s → 3 documents (talos + cluster + charts), NO stored row → pending
|
||||
_seed_doc(db, "Homelab", "k8s/talos.md", "Talos", 1, base)
|
||||
_seed_doc(db, "Homelab", "k8s/cluster.md", "Cluster", 1, base)
|
||||
# k8s/helm → 1 document — below the 2-doc minimum, never pending
|
||||
_seed_doc(db, "Homelab", "k8s/helm/charts.md", "Charts", 1, base)
|
||||
# wiki → 2 documents, WITH a stored row → not pending
|
||||
_seed_doc(db, "Homelab", "wiki/one.md", "One", 1, base)
|
||||
_seed_doc(db, "Homelab", "wiki/two.md", "Two", 1, base)
|
||||
db.add_all(
|
||||
[
|
||||
# The 5-doc source root IS covered → the source node is not pending
|
||||
FolderSummary(source="Homelab", folder_path="", summary="Homelab docs."),
|
||||
FolderSummary(source="Homelab", folder_path="wiki", summary="Wiki pages."),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
|
||||
r = admin_client.get("/api/docs/tree")
|
||||
assert r.status_code == 200
|
||||
(homelab,) = r.json()["sources"]
|
||||
assert set(homelab) == {"name", "documents", "summary", "summary_pending", "children"}
|
||||
assert homelab["summary"] == "Homelab docs."
|
||||
assert homelab["summary_pending"] is False
|
||||
# Direct subfolders in path order: k8s < wiki.
|
||||
k8s, wiki = [c for c in homelab["children"] if c["kind"] == "folder"]
|
||||
assert k8s["path"] == "k8s"
|
||||
assert k8s["summary"] is None
|
||||
assert k8s["summary_pending"] is True # 3 docs, no stored row
|
||||
helm = k8s["children"][0]
|
||||
assert (helm["kind"], helm["path"]) == ("folder", "k8s/helm")
|
||||
assert helm["summary_pending"] is False # 1 doc — never pending
|
||||
assert wiki["summary"] == "Wiki pages."
|
||||
assert wiki["summary_pending"] is False # 2 docs, but a stored row covers it
|
||||
# File nodes carry no pending flag at all (the file table has no
|
||||
# description column — D3's file exclusion).
|
||||
for node in (k8s, wiki, helm):
|
||||
for child in node["children"]:
|
||||
if child["kind"] == "file":
|
||||
assert "summary_pending" not in child
|
||||
|
||||
_truncate_tree_tables(db)
|
||||
|
||||
|
||||
def test_docs_tree_pending_set_equals_missing_folder_summaries(admin_client, db) -> None:
|
||||
"""The D3 cross-check (ONE concept end to end): with a PARTIAL
|
||||
summary table (some rows deleted — the phase-96 gap-fill pattern),
|
||||
the set of ``(source, folder_path)`` flagged pending in the fetched
|
||||
tree (source root = ``""``) equals
|
||||
:func:`missing_folder_summaries` — the marker can never drift from
|
||||
the gap-fill."""
|
||||
_truncate_tree_tables(db)
|
||||
base = datetime.now(UTC)
|
||||
db.add(GitSource(url="https://github.com/reese/Alpha.git", kind="git", added_at=base))
|
||||
db.add(
|
||||
GitSource(
|
||||
url="https://github.com/reese/Beta.git",
|
||||
kind="git",
|
||||
added_at=base + timedelta(hours=1),
|
||||
)
|
||||
)
|
||||
db.add(
|
||||
GitSource(
|
||||
url="https://github.com/reese/Gamma.git",
|
||||
kind="git",
|
||||
added_at=base + timedelta(hours=2),
|
||||
)
|
||||
)
|
||||
# Alpha: a/b holds 2 docs, c holds 1 (NEVER a candidate), the root
|
||||
# holds 4 — candidates (Alpha, ""), (Alpha, "a"), (Alpha, "a/b").
|
||||
_seed_doc(db, "Alpha", "a/b/c1.md", "C1", 1, base)
|
||||
_seed_doc(db, "Alpha", "a/b/c2.md", "C2", 1, base)
|
||||
_seed_doc(db, "Alpha", "c/solo.md", "Solo", 1, base)
|
||||
_seed_doc(db, "Alpha", "top.md", "Top", 1, base)
|
||||
# Beta: x holds 2 docs, the root holds 2 — candidates
|
||||
# (Beta, ""), (Beta, "x").
|
||||
_seed_doc(db, "Beta", "x/one.md", "One", 1, base)
|
||||
_seed_doc(db, "Beta", "x/two.md", "Two", 1, base)
|
||||
# Gamma: registered, 0 documents — no candidates at all.
|
||||
# Seed every candidate row, then DELETE two of them (the phase-96
|
||||
# direct-row-deletion pattern — a fail-soft miss / cleared row).
|
||||
db.add_all(
|
||||
FolderSummary(source=s, folder_path=f, summary=t)
|
||||
for (s, f), t in {
|
||||
("Alpha", ""): "Alpha root.",
|
||||
("Alpha", "a"): "Alpha a.",
|
||||
("Alpha", "a/b"): "Alpha a b.",
|
||||
("Beta", ""): "Beta root.",
|
||||
("Beta", "x"): "Beta x.",
|
||||
}.items()
|
||||
)
|
||||
db.commit()
|
||||
db.execute(
|
||||
delete(FolderSummary).where(
|
||||
FolderSummary.source == "Alpha", FolderSummary.folder_path == "a"
|
||||
)
|
||||
)
|
||||
db.execute(
|
||||
delete(FolderSummary).where(
|
||||
FolderSummary.source == "Beta", FolderSummary.folder_path == ""
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
r = admin_client.get("/api/docs/tree")
|
||||
assert r.status_code == 200
|
||||
pending = _tree_pending_keys(r.json()["sources"])
|
||||
# THE cross-check: the marker set IS the gap-fill's candidate set.
|
||||
assert pending == set(missing_folder_summaries(db))
|
||||
# And the explicit expectation (the test is readable without the
|
||||
# helper): exactly the two deleted keys, root riding "".
|
||||
assert pending == {("Alpha", "a"), ("Beta", "")}
|
||||
# Never flagged: the < 2-doc folder (Alpha/c), the 0-document
|
||||
# registered source (Gamma), and every node that still holds a row.
|
||||
assert ("Alpha", "c") not in pending
|
||||
assert not any(name == "Gamma" for name, _ in pending)
|
||||
assert ("Alpha", "a/b") not in pending
|
||||
assert ("Beta", "x") not in pending
|
||||
assert ("Alpha", "") not in pending
|
||||
|
||||
_truncate_tree_tables(db)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------
|
||||
# PATCH /api/folders/summary (phase 97, task 03) — the admin
|
||||
# folder-description editor: update / create / source-root / clear /
|
||||
|
||||
@@ -69,6 +69,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Callable, Iterator
|
||||
from datetime import UTC, datetime
|
||||
@@ -315,16 +316,29 @@ class FakeFolderSummaries:
|
||||
generator with ``only_missing=True`` — the fake records the flag
|
||||
the same way it records ``skip`` (the real gap probe,
|
||||
``missing_folder_summaries``, runs against the real tables).
|
||||
Phase 98 (task 01): records the ``on_progress`` hook the runner
|
||||
wires into the generation branches (a live closure while the
|
||||
wiring holds, None if it regresses); a canned *progress* list of
|
||||
``(done, total, source, folder_path)`` steps is fired through the
|
||||
hook when given — the same way the import's file counter is driven
|
||||
through its hook, so the status's summary counters are testable
|
||||
deterministically.
|
||||
"""
|
||||
|
||||
ZERO = {"generated": 0, "failed": 0, "pruned": 0}
|
||||
|
||||
def __init__(self, stats: dict[str, int] | None = None) -> None:
|
||||
def __init__(
|
||||
self,
|
||||
stats: dict[str, int] | None = None,
|
||||
progress: list[tuple[int, int, str, str]] | None = None,
|
||||
) -> None:
|
||||
self.stats = stats if stats is not None else dict(self.ZERO)
|
||||
self.progress_steps = list(progress or [])
|
||||
self.llms: list[LLMClient] = []
|
||||
self.sessions: list[Session] = []
|
||||
self.skip_flags: list[bool] = []
|
||||
self.only_missing_flags: list[bool] = []
|
||||
self.progress_hooks: list[Callable[[int, int, str, str], None] | None] = []
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -333,13 +347,18 @@ class FakeFolderSummaries:
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
self.skip_flags.append(skip)
|
||||
self.only_missing_flags.append(only_missing)
|
||||
self.progress_hooks.append(on_progress)
|
||||
if skip:
|
||||
return dict(self.ZERO)
|
||||
self.llms.append(llm)
|
||||
self.sessions.append(db)
|
||||
for done, total, source, folder_path in self.progress_steps:
|
||||
if on_progress is not None:
|
||||
on_progress(done, total, source, folder_path)
|
||||
return dict(self.stats)
|
||||
|
||||
|
||||
@@ -394,7 +413,13 @@ def test_admin_sync_success_reports_full_detail(
|
||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||
fake_overview = FakeOverview(ok=True)
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
|
||||
# Phase 98 (task 01): the canned summary steps fire through the
|
||||
# runner's live hook closure — the terminal's summary counters
|
||||
# (kept per the keep-final-counts convention) are pinned below.
|
||||
fake_folders = FakeFolderSummaries(
|
||||
progress=[(1, 2, "repo", ""), (2, 2, "repo", "a")]
|
||||
)
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folders)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.get("/api/sync/status").json() == {
|
||||
@@ -407,6 +432,11 @@ def test_admin_sync_success_reports_full_detail(
|
||||
"current_file": None,
|
||||
"files_done": 0,
|
||||
"files_total": 0,
|
||||
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
|
||||
"phase": None,
|
||||
"current_summary": None,
|
||||
"summaries_done": 0,
|
||||
"summaries_total": 0,
|
||||
}
|
||||
|
||||
r = sync_client.post("/api/sync")
|
||||
@@ -445,6 +475,15 @@ def test_admin_sync_success_reports_full_detail(
|
||||
# Phase 41: the probe ran first and got the very client the import
|
||||
# and the overview reuse.
|
||||
assert probe_seen == [fake_import.llms[0]]
|
||||
# Phase 98 (task 01): the summary hook was wired into the
|
||||
# changed-KB branch and fired the canned steps — the terminal
|
||||
# clears phase + current_summary and KEEPS the hook's final
|
||||
# summary counts (the phase-64 keep-final-counts convention).
|
||||
assert len(fake_folders.progress_hooks) == 1
|
||||
assert fake_folders.progress_hooks[0] is not None
|
||||
assert body["phase"] is None
|
||||
assert body["current_summary"] is None
|
||||
assert body["summaries_done"] == 2 and body["summaries_total"] == 2
|
||||
|
||||
|
||||
def test_unchanged_kb_skips_overview_refresh(
|
||||
@@ -518,10 +557,279 @@ def test_double_trigger_while_running_returns_409(
|
||||
assert body["started_at"] is not None
|
||||
assert body["finished_at"] is None
|
||||
assert body["error"] is None
|
||||
# Phase 98 (task 01): the in-flight run (parked in the fake
|
||||
# import's delay) reports the import phase — the summary span has
|
||||
# not started, so its counters are 0/0.
|
||||
assert body["phase"] == "import"
|
||||
assert body["current_summary"] is None
|
||||
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
|
||||
|
||||
# The (single) run completes; the import ran exactly once.
|
||||
_poll(sync_client, "success")
|
||||
body = _poll(sync_client, "success")
|
||||
assert len(fake_import.sources) == 1
|
||||
# Phase 98 (task 01): the terminal cleared the phase keys (the
|
||||
# fake folder step fired no hook — nothing to keep).
|
||||
assert body["phase"] is None and body["current_summary"] is None
|
||||
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
|
||||
|
||||
|
||||
# --- phase 98 (task 01): the status's phase machine --------------------
|
||||
# The background task runs on the app's event loop, so every gate below
|
||||
# parks AWAY from the loop (``asyncio.to_thread(event.wait)``) — a
|
||||
# blocking wait on the loop thread would deadlock the very status
|
||||
# endpoint the test is polling.
|
||||
|
||||
|
||||
class _GatedImport:
|
||||
"""An import that fires the runner's progress hook once, then
|
||||
parks on a threading gate — the test reads the status mid-import
|
||||
(the phase ``"import"`` + the file hook)."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
summary: ImportSummary,
|
||||
started: threading.Event,
|
||||
release: threading.Event,
|
||||
) -> None:
|
||||
self.summary = summary
|
||||
self.started = started
|
||||
self.release = release
|
||||
self.hook_calls: list[tuple[str, str, int, int]] = []
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
sources: list[Path],
|
||||
llm: LLMClient,
|
||||
*,
|
||||
prune: bool = False,
|
||||
limit: int | None = None,
|
||||
session: Session | None = None,
|
||||
progress: Callable[[str, str, int, int], None] | None = None,
|
||||
ignore_by_root: dict[str, list[str]] | None = None,
|
||||
include_hidden_by_root: dict[str, bool] | None = None,
|
||||
) -> ImportSummary:
|
||||
if progress is not None:
|
||||
progress("repo", "notes/deep.md", 1, 3)
|
||||
self.hook_calls.append(("repo", "notes/deep.md", 1, 3))
|
||||
self.started.set()
|
||||
await asyncio.to_thread(self.release.wait)
|
||||
return self.summary
|
||||
|
||||
|
||||
class _GatedOverview:
|
||||
"""A KB-overview step that parks on a threading gate once it
|
||||
starts — the test reads the status mid-overview (phase
|
||||
``"overview"``)."""
|
||||
|
||||
def __init__(self, started: threading.Event, release: threading.Event) -> None:
|
||||
self.started = started
|
||||
self.release = release
|
||||
|
||||
async def __call__(self, llm: LLMClient, session: Session | None = None) -> bool:
|
||||
self.started.set()
|
||||
await asyncio.to_thread(self.release.wait)
|
||||
return True
|
||||
|
||||
|
||||
class _GatedFolderSummaries:
|
||||
"""A folder-summary step that fires the runner's progress hook
|
||||
(canned steps), parks mid-span — the (long) phase the user
|
||||
reported — then fires the final steps and returns canned stats.
|
||||
The test reads the status mid-span (the phase ``"summaries"`` +
|
||||
the hook's folder + counters)."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
started: threading.Event,
|
||||
release: threading.Event,
|
||||
steps_before: list[tuple[int, int, str, str]],
|
||||
steps_after: list[tuple[int, int, str, str]],
|
||||
) -> None:
|
||||
self.started = started
|
||||
self.release = release
|
||||
self.steps_before = steps_before
|
||||
self.steps_after = steps_after
|
||||
self.hook: Callable[[int, int, str, str], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
db: Session,
|
||||
llm: LLMClient,
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
assert not skip, "these runs take a generation branch, never --limit"
|
||||
self.hook = on_progress
|
||||
for done, total, source, folder_path in self.steps_before:
|
||||
if on_progress is not None:
|
||||
on_progress(done, total, source, folder_path)
|
||||
self.started.set()
|
||||
await asyncio.to_thread(self.release.wait)
|
||||
for done, total, source, folder_path in self.steps_after:
|
||||
if on_progress is not None:
|
||||
on_progress(done, total, source, folder_path)
|
||||
return {"generated": 3, "failed": 0, "pruned": 0, "kept_manual": 0}
|
||||
|
||||
|
||||
def test_status_reports_the_phase_machine_across_the_run(
|
||||
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 98 (task 01), the full state machine: idle reports the
|
||||
four keys null/0/0/0; the model-check + clone/pull prelude
|
||||
reports ``phase: null`` (D1 — the bare label's pin); the import
|
||||
reports ``"import"`` with the file hook (the phase-64 shape,
|
||||
unchanged); the KB-overview step reports ``"overview"``; the
|
||||
folder-summary span reports ``"summaries"`` with the hook's
|
||||
folder + done/total (the import's final file position kept — the
|
||||
pause the user reported); the success terminal clears ``phase`` +
|
||||
``current_summary`` and KEEPS the hook's final summary counts
|
||||
(the phase-64 keep-final-counts convention)."""
|
||||
repo_url = f"file://{tmp_path / 'repo.git'}"
|
||||
_seed(db, repo_url)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
_, fake_clone = _fake_clone()
|
||||
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||
_stub_probe(monkeypatch)
|
||||
gated_import = _GatedImport(
|
||||
ImportSummary(files=3, added=1, updated=1, unchanged=1),
|
||||
threading.Event(),
|
||||
threading.Event(),
|
||||
)
|
||||
monkeypatch.setattr(sync_api, "import_sources", gated_import)
|
||||
gated_overview = _GatedOverview(threading.Event(), threading.Event())
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", gated_overview)
|
||||
gated_folders = _GatedFolderSummaries(
|
||||
threading.Event(),
|
||||
threading.Event(),
|
||||
steps_before=[(1, 3, "repo", "")],
|
||||
steps_after=[(3, 3, "repo", "notes")],
|
||||
)
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
|
||||
|
||||
_login(sync_client)
|
||||
# Idle: the four phase keys ride along as null/0/0/0.
|
||||
body = sync_client.get("/api/sync/status").json()
|
||||
assert body["state"] == "idle"
|
||||
assert body["phase"] is None
|
||||
assert body["current_summary"] is None
|
||||
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
|
||||
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
try:
|
||||
assert gated_import.started.wait(5.0), "the run never reached the import"
|
||||
s = sync_client.get("/api/sync/status").json()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "import"
|
||||
assert s["current_file"] == "repo/notes/deep.md" # the file hook as today
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
assert s["current_summary"] is None
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
gated_import.release.set()
|
||||
assert gated_overview.started.wait(5.0), "the run never reached the overview"
|
||||
s = sync_client.get("/api/sync/status").json()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "overview"
|
||||
assert s["current_summary"] is None
|
||||
gated_overview.release.set()
|
||||
assert gated_folders.started.wait(5.0), "the run never reached the summaries"
|
||||
s = sync_client.get("/api/sync/status").json()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "summaries"
|
||||
# The hook's first step: the source-root row (the bare source
|
||||
# name — folder_path "" never gets a slash).
|
||||
assert s["current_summary"] == "repo"
|
||||
assert s["summaries_done"] == 1 and s["summaries_total"] == 3
|
||||
# The import's final position is kept through the summary span
|
||||
# (the "number pauses" span — the user's report).
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
gated_folders.release.set()
|
||||
finally:
|
||||
gated_import.release.set()
|
||||
gated_overview.release.set()
|
||||
gated_folders.release.set()
|
||||
|
||||
body = _poll(sync_client, "success")
|
||||
# Terminal: phase + current_summary cleared ... (the final step
|
||||
# fired post-park, so the kept counts are its position).
|
||||
assert body["phase"] is None
|
||||
assert body["current_summary"] is None
|
||||
assert body["summaries_done"] == 3 and body["summaries_total"] == 3
|
||||
# Wiring: the summary hook was the runner's live closure (the
|
||||
# counters above came through it); the file hook fired once.
|
||||
assert gated_folders.hook is not None
|
||||
assert gated_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
|
||||
|
||||
|
||||
def test_failed_terminal_after_the_summary_hook_clears_phase_keeps_counts(
|
||||
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 98 (task 01), the except terminal path: a run that dies
|
||||
AFTER the summary hook has fired (here: the folder step raises
|
||||
post-hook) clears ``phase`` + ``current_summary`` in the failed
|
||||
terminal AND keeps the hook's final summary counts next to the
|
||||
error — the same keep-final-counts convention as success (D1:
|
||||
BOTH terminal paths)."""
|
||||
repo_url = f"file://{tmp_path / 'repo.git'}"
|
||||
_seed(db, repo_url)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
_, fake_clone = _fake_clone()
|
||||
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||
_stub_probe(monkeypatch)
|
||||
fake_import = FakeImportSources(
|
||||
ImportSummary(files=3, added=1, updated=1, unchanged=1)
|
||||
)
|
||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||
|
||||
class _HookThenFail:
|
||||
"""Fires two progress steps through the runner's hook, then
|
||||
raises — the failure lands AFTER the hook moved the
|
||||
counters (the keep-counts path the success test pins for the
|
||||
happy terminal)."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.hook: Callable[[int, int, str, str], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
db: Session,
|
||||
llm: LLMClient,
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
self.hook = on_progress
|
||||
assert on_progress is not None, "the runner must wire the hook"
|
||||
on_progress(1, 3, "repo", "")
|
||||
on_progress(2, 3, "repo", "notes")
|
||||
raise RuntimeError("simulated post-hook failure (test sentinel)")
|
||||
|
||||
failing = _HookThenFail()
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", failing)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "failed")
|
||||
|
||||
assert "simulated post-hook failure" in body["error"]
|
||||
assert body["phase"] is None # cleared in the failed terminal
|
||||
assert body["current_summary"] is None
|
||||
# The hook's FINAL summary counts survive the failure (D1).
|
||||
assert body["summaries_done"] == 2 and body["summaries_total"] == 3
|
||||
assert failing.hook is not None # the wiring held
|
||||
|
||||
|
||||
# --- admin: failures -------------------------------------------------------
|
||||
@@ -560,6 +868,15 @@ def test_git_failure_marks_failed_and_skips_import(
|
||||
assert "fatal: repository not found" in body["error"]
|
||||
assert body["detail"] == {}
|
||||
assert body["finished_at"] is not None
|
||||
# Phase 98 (task 01): the failed terminal (died in the clone/pull
|
||||
# prelude — ``phase`` was still null) clears the phase keys; the
|
||||
# run never reached the import or summary spans, so all counters
|
||||
# are 0/0.
|
||||
assert body["phase"] is None
|
||||
assert body["current_summary"] is None
|
||||
assert body["current_file"] is None
|
||||
assert body["files_done"] == 0 and body["files_total"] == 0
|
||||
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
|
||||
assert fake_import.sources == [] # no partial import
|
||||
assert fake_overview.llms == []
|
||||
# Phase 53: a FAILED sync never bumps — the version is untouched.
|
||||
@@ -847,6 +1164,12 @@ def test_import_error_is_reported_with_credentials_masked(
|
||||
assert "*****@aipi.reeseapps.com" in body["error"] # credentials masked
|
||||
assert "user:secret" not in body["error"]
|
||||
assert "connection refused" in body["error"] # the reason survives
|
||||
# Phase 98 (task 01): the failed terminal (died mid-import) clears
|
||||
# phase + current_summary; the run never reached the summary span,
|
||||
# so those counters are 0/0.
|
||||
assert body["phase"] is None
|
||||
assert body["current_summary"] is None
|
||||
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
|
||||
|
||||
|
||||
# --- phase 41: model probe (fail fast before any clone) --------------------
|
||||
|
||||
@@ -729,6 +729,13 @@ def test_api_changed_sync_generates_folder_rows(
|
||||
assert body["detail"]["added"] == 2
|
||||
assert body["detail"]["overview"] is True
|
||||
assert body["detail"]["sources_version"] == 1
|
||||
# Phase 98 (task 01): the progress hook fired on the CHANGED-KB
|
||||
# regeneration branch — the terminal clears phase + current_summary
|
||||
# and keeps the hook's final counts (2 candidates: root + a/ — the
|
||||
# only writer of those counters is the hook itself).
|
||||
assert body["phase"] is None
|
||||
assert body["current_summary"] is None
|
||||
assert body["summaries_done"] == 2 and body["summaries_total"] == 2
|
||||
# One LLM client for probe + import + overview + folder summaries;
|
||||
# exactly two FOLDER_SUMMARY_MODE calls (root + the 2-doc a/ folder).
|
||||
assert len(clients) == 1
|
||||
@@ -770,6 +777,15 @@ def test_api_unchanged_resync_burns_zero_folder_calls(
|
||||
body = _poll(sync_client, "success")
|
||||
assert body["detail"]["overview"] is False
|
||||
assert body["detail"]["sources_version"] == 1 # unchanged → no bump
|
||||
# Phase 98 (task 01): the no-gap skip branch never CALLS the
|
||||
# generator — the hook never fires, so the terminal's summary
|
||||
# counters stay 0/0 (the run stayed in the import phase through to
|
||||
# the terminal, which then cleared it). The counter keep at 0/0 is
|
||||
# the deterministic pin of "never fired": a fired hook's counts
|
||||
# would survive to the terminal (the keep-final-counts rule).
|
||||
assert body["phase"] is None
|
||||
assert body["current_summary"] is None
|
||||
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
|
||||
assert len(clients) == 2
|
||||
# Zero GENERATION calls — no folder summary, no KB overview. The
|
||||
# only chat the re-sync's client makes is the phase-41 probe's ping.
|
||||
@@ -849,6 +865,12 @@ def test_api_unchanged_resync_with_gap_fills_only_the_missing_row(
|
||||
"chunks", "summaries", "summary_errors", "overview",
|
||||
"sources_version",
|
||||
}
|
||||
# Phase 98 (task 01): the progress hook fired on the UNCHANGED-KB
|
||||
# GAP-FILL branch — total is the missing count (the one deleted
|
||||
# row), kept at the terminal next to the cleared phase keys.
|
||||
assert body["phase"] is None
|
||||
assert body["current_summary"] is None
|
||||
assert body["summaries_done"] == 1 and body["summaries_total"] == 1
|
||||
assert len(clients) == 2
|
||||
# Targeted fill — exactly ONE folder call, the missing folder only
|
||||
# (plus the phase-41 probe's ping on the same client).
|
||||
@@ -910,6 +932,11 @@ def test_api_folder_lite_failure_keeps_rows_stays_green_and_bumps(
|
||||
assert body["error"] is None
|
||||
assert body["detail"]["updated"] == 1
|
||||
assert body["detail"]["sources_version"] == 2 # the bump was not blocked
|
||||
# Phase 98 (task 01): the failed (fail-soft) folder still advanced
|
||||
# the hook's counter — the hook fires BEFORE the attempt, so the
|
||||
# terminal keeps 2/2, not 1/2.
|
||||
assert body["phase"] is None and body["current_summary"] is None
|
||||
assert body["summaries_done"] == 2 and body["summaries_total"] == 2
|
||||
assert current_sources_version(db) == 2
|
||||
rows_after = _rows(db)
|
||||
assert rows_after["LocalDocs", "a"] == rows_before["LocalDocs", "a"]
|
||||
@@ -951,6 +978,11 @@ def test_api_empty_table_first_sync_regenerates(
|
||||
|
||||
assert body["detail"]["overview"] is False # unchanged → no overview
|
||||
assert body["detail"]["sources_version"] == 1 # unchanged → no bump
|
||||
# Phase 98 (task 01): the gap-fill branch over the emptied table —
|
||||
# every candidate missing (2), the hook's final counts kept at the
|
||||
# terminal next to the cleared phase keys.
|
||||
assert body["phase"] is None and body["current_summary"] is None
|
||||
assert body["summaries_done"] == 2 and body["summaries_total"] == 2
|
||||
assert len(clients) == 2
|
||||
assert len(_folder_calls(clients[1])) == 2 # the folders regenerated
|
||||
assert set(_rows(db)) == {("LocalDocs", ""), ("LocalDocs", "a")}
|
||||
|
||||
@@ -48,7 +48,9 @@ class _FakeLLM:
|
||||
|
||||
Records each ``(system, user)`` request and the ``model`` kwarg;
|
||||
returns the canned reply, or raises — either a fixed exception or a
|
||||
per-folder failure keyed on the user message's ``Folder: …`` header
|
||||
per-folder failure keyed on the user message's FIRST LINE (the
|
||||
``Folder: …`` header) — EXACT match, so a nested folder's header
|
||||
(``Folder: S/a/b``) can never shadow its parent's (``Folder: S/a``)
|
||||
(the per-folder fail-soft tests).
|
||||
"""
|
||||
|
||||
@@ -74,8 +76,9 @@ class _FakeLLM:
|
||||
system = messages[0]["content"]
|
||||
user = messages[-1]["content"]
|
||||
self.requests.append((system, user))
|
||||
header = user.splitlines()[0] if user else ""
|
||||
for folder in self._fail_folders:
|
||||
if FOLDER_HEADER_PREFIX + folder in user:
|
||||
if header == FOLDER_HEADER_PREFIX + folder:
|
||||
raise LLMError(f"simulated lite-model failure for {folder}")
|
||||
if self._fail is not None:
|
||||
raise self._fail
|
||||
@@ -916,3 +919,144 @@ def test_gap_probe_subsumes_the_table_empty_gate(db: Session, clean_tables) -> N
|
||||
db.execute(text("DELETE FROM folder_summaries"))
|
||||
db.commit()
|
||||
assert missing_folder_summaries(db) == [("FSU", ""), ("FSU", "a")] # emptied again
|
||||
|
||||
|
||||
# ---------- on_progress hook (phase 98, task 01) ----------
|
||||
|
||||
|
||||
def _record_progress() -> tuple[list[tuple[int, int, str, str]], Any]:
|
||||
"""A fresh event list + the ``on_progress`` recorder that appends
|
||||
every ``(done, total, source, folder_path)`` event it receives."""
|
||||
events: list[tuple[int, int, str, str]] = []
|
||||
|
||||
def record(done: int, total: int, source: str, folder_path: str) -> None:
|
||||
events.append((done, total, source, folder_path))
|
||||
|
||||
return events, record
|
||||
|
||||
|
||||
def test_on_progress_fires_once_per_candidate_in_sorted_key_order(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""Phase 98 (task 01): the hook fires once per candidate, in the
|
||||
same sorted ``(source, folder_path)`` order the folders are
|
||||
attempted — done climbs 1..total, total = the candidate count
|
||||
(the single-doc FSU-solo root is not a candidate — no event)."""
|
||||
_seed_catalogue(db)
|
||||
llm = _FakeLLM()
|
||||
events, record = _record_progress()
|
||||
stats = asyncio.run(generate_folder_summaries(db, llm, on_progress=record))
|
||||
assert events == [
|
||||
(1, 3, "FSU", ""),
|
||||
(2, 3, "FSU", "a"),
|
||||
(3, 3, "FSU", "a/b"),
|
||||
]
|
||||
assert llm.calls == 3 # one event per attempt, in the same order
|
||||
assert stats["generated"] == 3
|
||||
|
||||
|
||||
def test_on_progress_manual_skip_still_advances(db: Session, clean_tables) -> None:
|
||||
"""A manual-skip key is an INSTANT skip — no ``lite`` call burns,
|
||||
but the counter still advances for it (D5: the UI's position moves
|
||||
on either outcome)."""
|
||||
_seed_catalogue(db)
|
||||
db.add(
|
||||
FolderSummary(
|
||||
source="FSU", folder_path="a", summary="owner text",
|
||||
manually_edited=True,
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
llm = _FakeLLM()
|
||||
events, record = _record_progress()
|
||||
stats = asyncio.run(generate_folder_summaries(db, llm, on_progress=record))
|
||||
assert events == [
|
||||
(1, 3, "FSU", ""),
|
||||
(2, 3, "FSU", "a"), # the instant manual skip still advances
|
||||
(3, 3, "FSU", "a/b"),
|
||||
]
|
||||
assert llm.calls == 2, "the skip itself burns no call"
|
||||
assert stats["kept_manual"] == 1 and stats["generated"] == 2
|
||||
|
||||
|
||||
def test_on_progress_failed_key_still_advances(db: Session, clean_tables) -> None:
|
||||
"""A failed (``LLMError``) key still advances — the hook fires
|
||||
BEFORE the attempt, so a fail-soft miss is visible to the UI as a
|
||||
completed step (the run never flips to failed, neither does the
|
||||
counter stall)."""
|
||||
_seed_catalogue(db)
|
||||
llm = _FakeLLM(fail_folders=("FSU/a",))
|
||||
events, record = _record_progress()
|
||||
stats = asyncio.run(generate_folder_summaries(db, llm, on_progress=record))
|
||||
assert events == [
|
||||
(1, 3, "FSU", ""),
|
||||
(2, 3, "FSU", "a"), # the failed attempt still advances
|
||||
(3, 3, "FSU", "a/b"),
|
||||
]
|
||||
assert stats["failed"] == 1 and stats["generated"] == 2
|
||||
|
||||
|
||||
def test_on_progress_only_missing_reports_the_missing_count(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""Under ``only_missing=True`` ``total`` is the MISSING count (the
|
||||
loop-start count after the missing filter, not the full candidate
|
||||
count) and the events name exactly the missing keys, sorted."""
|
||||
_seed_catalogue(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries"
|
||||
" WHERE (source, folder_path) IN (('FSU', ''), ('FSU', 'a/b'))"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
llm = _FakeLLM()
|
||||
events, record = _record_progress()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, only_missing=True, on_progress=record)
|
||||
)
|
||||
assert events == [
|
||||
(1, 2, "FSU", ""),
|
||||
(2, 2, "FSU", "a/b"),
|
||||
]
|
||||
assert llm.calls == 2
|
||||
assert stats["generated"] == 2
|
||||
|
||||
|
||||
def test_on_progress_skip_true_fires_zero_calls(db: Session, clean_tables) -> None:
|
||||
"""``skip=True`` (the ``--limit`` debug run) returns before the
|
||||
loop — the hook never fires."""
|
||||
_seed_catalogue(db)
|
||||
llm = _FakeLLM()
|
||||
events, record = _record_progress()
|
||||
stats = asyncio.run(generate_folder_summaries(db, llm, skip=True, on_progress=record))
|
||||
assert events == []
|
||||
assert llm.calls == 0
|
||||
assert stats == {"generated": 0, "failed": 0, "pruned": 0, "kept_manual": 0}
|
||||
|
||||
|
||||
def test_on_progress_none_is_a_zero_cost_noop(db: Session, clean_tables) -> None:
|
||||
"""``on_progress=None`` (the ``scripts/import_docs.py`` CLI path)
|
||||
behaves byte-identically to today: the fake LLM's call log, the
|
||||
stats dict, and the stored rows are identical across a hooked and
|
||||
a hook-less run."""
|
||||
_seed_catalogue(db)
|
||||
hooked = _FakeLLM()
|
||||
events, record = _record_progress()
|
||||
stats_hooked = asyncio.run(
|
||||
generate_folder_summaries(db, hooked, on_progress=record)
|
||||
)
|
||||
requests_hooked = hooked.requests.copy()
|
||||
assert len(events) == 3, "the hooked run fired (the contrast is real)"
|
||||
db.rollback() # the generator only flushes — drop the uncommitted rows
|
||||
|
||||
plain = _FakeLLM()
|
||||
stats_plain = asyncio.run(generate_folder_summaries(db, plain, on_progress=None))
|
||||
|
||||
assert plain.requests == requests_hooked, "the fake LLM's call log is unchanged"
|
||||
assert stats_plain == stats_hooked
|
||||
assert _rows(db) == {
|
||||
("FSU", ""): REPLY, ("FSU", "a"): REPLY, ("FSU", "a/b"): REPLY
|
||||
}
|
||||
|
||||
@@ -34,6 +34,18 @@ settled result line is
|
||||
off the status's {"message": "uploaded"} detail; the nameless variant
|
||||
after a reload / on the 409 re-attach), and the success announce
|
||||
names the next step.
|
||||
|
||||
Phase 98 (task 02) pins the SYNC job's phase-aware label (D2):
|
||||
fmtSyncPhaseLabel — "overview" → exactly "Writing KB overview…",
|
||||
"summaries" → "Summarizing folders… <current_summary> (n/m)" (the
|
||||
folder part omitted while current_summary is null → the bare
|
||||
"Summarizing folders… (n/m)"), anything else → the byte-identical
|
||||
phase-64 fmtSyncLabel("sync", …) fall-through — plus the status-object
|
||||
threading (the tick's branch 1 + the initSyncButton running re-attach
|
||||
pass the WHOLE status; the upload job and the 202/409 click stay on
|
||||
the bare label — the upload status has no phase) and the A4 title-rule
|
||||
adjustment (the full untruncated label rides the button title for
|
||||
EVERY running label — the file-only rule is gone).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -182,30 +194,86 @@ def test_fmt_sync_label_signature_and_prefixes() -> None:
|
||||
# ---------- enterSyncRunningState: full path to title + announcer ----------
|
||||
|
||||
|
||||
def test_enter_running_state_writes_full_path_to_title_and_announcer() -> None:
|
||||
def test_fmt_sync_phase_label_is_the_exact_d2_contract() -> None:
|
||||
"""fmtSyncPhaseLabel(status) — the SYNC job's phase-aware label
|
||||
(phase 98 task 02, D2): "overview" → exactly "Writing KB
|
||||
overview…" (no folder, no counts); "summaries" →
|
||||
"Summarizing folders… <current_summary> (n/m)" — the folder part
|
||||
(one leading space + current_summary) is omitted while
|
||||
current_summary is null (the phase's first poll → the bare
|
||||
"Summarizing folders… (n/m)"); anything else (the null prelude,
|
||||
"import") → the phase-64 fmtSyncLabel fall-through, byte-identical
|
||||
(same builder, same fields, same order). The UPLOAD job never
|
||||
enters here — its status has no phase (D2)."""
|
||||
body = _fn(_js(), "fmtSyncPhaseLabel")
|
||||
assert "function fmtSyncPhaseLabel(status)" in body
|
||||
# "overview": the exact copy, no folder, no counts.
|
||||
i_ov = body.find('status.phase === "overview"')
|
||||
i_sum = body.find('status.phase === "summaries"')
|
||||
assert -1 < i_ov < i_sum
|
||||
ov = body[i_ov:i_sum]
|
||||
assert 'return "Writing KB overview…";' in ov
|
||||
assert "current_summary" not in ov and "summaries_done" not in ov, (
|
||||
"the overview label carries no folder and no counts"
|
||||
)
|
||||
# "summaries": the exact template — optional folder, counts always.
|
||||
i_fall = body.find("return fmtSyncLabel")
|
||||
assert i_sum < i_fall, "the summaries branch precedes the fall-through"
|
||||
sum_ = body[i_sum:i_fall]
|
||||
assert 'status.current_summary ? ` ${status.current_summary}` : ""' in sum_, (
|
||||
"the folder part — one leading space + the folder, or nothing"
|
||||
)
|
||||
assert (
|
||||
"return `Summarizing folders…${folder} "
|
||||
"(${status.summaries_done}/${status.summaries_total})`;"
|
||||
) in sum_, "the exact D2 copy (with and without the folder)"
|
||||
# anything else (null prelude, "import"): byte-identical to today's
|
||||
# sync label — same builder, same fields, same order.
|
||||
assert (
|
||||
'return fmtSyncLabel("sync", status.current_file, '
|
||||
"status.files_done, status.files_total);"
|
||||
) in body[i_fall:]
|
||||
|
||||
|
||||
def test_enter_running_state_is_phase_aware_and_announces_full_label() -> None:
|
||||
"""The running-state entry keeps the §7.4 never-stale mechanics
|
||||
(disabled, aria-busy, spinning icon, the stale .is-error removed)
|
||||
and — phase 64 — writes the FULL untruncated current file to the
|
||||
button title (removed when null: no file yet) and the full
|
||||
untruncated fmtSyncLabel text to BOTH the label span and
|
||||
#sync-result (the aria-live announcer reads the full live path;
|
||||
CSS only ellipsizes the button's span)."""
|
||||
and — phase 98 (task 02, D2) — is phase-aware for the SYNC job:
|
||||
a sync-kind call carrying the status object takes the
|
||||
fmtSyncPhaseLabel form, while the upload job (no status — its
|
||||
endpoint has no phase) and the 202/409 click keep the bare
|
||||
fmtSyncLabel label, byte-identical. The FULL untruncated label —
|
||||
whatever its form — rides the button title (set for EVERY running
|
||||
label: the phase-64 "title only when there is a current file"
|
||||
rule is adjusted away) and #sync-result (the aria-live announcer);
|
||||
CSS ellipsizes only the label span."""
|
||||
body = _fn(_js(), "enterSyncRunningState")
|
||||
assert (
|
||||
"function enterSyncRunningState(kind, currentFile, done, total, status)"
|
||||
in body
|
||||
)
|
||||
assert "syncBtn.disabled = true" in body
|
||||
assert 'syncBtn.setAttribute("aria-busy", "true")' in body
|
||||
assert "syncIcon.classList.add(\"is-spinning\")" in body
|
||||
assert "syncBtn.classList.remove(\"is-error\")" in body
|
||||
assert "syncBtn.title = currentFile" in body, "the full path on hover"
|
||||
assert 'syncBtn.removeAttribute("title")' in body, "removed when no file yet"
|
||||
i_set = body.find("if (currentFile) syncBtn.title = currentFile")
|
||||
i_remove = body.find('syncBtn.removeAttribute("title")')
|
||||
assert -1 < i_set < i_remove, "title is set (not removed) only when a file exists"
|
||||
assert "fmtSyncLabel(kind, currentFile, done, total)" in body
|
||||
i_label = body.find("fmtSyncLabel(kind, currentFile, done, total)")
|
||||
# the phase-aware split: sync + status → the D2 builder; the
|
||||
# fall-through (upload / the click, no status) is the byte-
|
||||
# identical phase-64 builder.
|
||||
i_phase = body.find("fmtSyncPhaseLabel(status)")
|
||||
i_fall = body.find("fmtSyncLabel(kind, currentFile, done, total)")
|
||||
assert 'kind === "sync" && status' in body
|
||||
assert -1 < i_phase < i_fall, "the phase label precedes the fall-through"
|
||||
# A4: the untruncated label rides the title for EVERY running label
|
||||
# (the remove-when-no-file rule is gone) and #sync-result.
|
||||
assert "syncBtn.title = label" in body, "the full label on hover, always"
|
||||
assert 'syncBtn.removeAttribute("title")' not in body, (
|
||||
"the title is set for every running label (D2: not just file labels)"
|
||||
)
|
||||
i_title = body.find("syncBtn.title = label")
|
||||
i_span = body.find("syncLabel.textContent = label")
|
||||
i_result = body.find("syncResult.textContent = label")
|
||||
assert -1 < i_label < i_span < i_result, (
|
||||
"one label: built once, written to the span AND the announcer"
|
||||
assert -1 < i_title < i_span < i_result, (
|
||||
"one label: built once, written to title, span AND the announcer"
|
||||
)
|
||||
|
||||
|
||||
@@ -327,14 +395,16 @@ def test_click_branch_enters_sync_running_without_a_file() -> None:
|
||||
|
||||
|
||||
def test_reattach_adopts_a_running_upload_only() -> None:
|
||||
"""initSyncButton: the sync branches are unchanged (running
|
||||
re-enters with the live file; the terminals render the last
|
||||
result). With the sync IDLE it fetches the upload status: a RUNNING
|
||||
upload run re-attaches (running state, upload kind — phase 90: no
|
||||
live file, the label stays bare "Importing…" — the synthetic
|
||||
running frame, the poll starts); a terminal upload is a no-op —
|
||||
the fall-through is the plain idle settle (the boot-time loadTree()
|
||||
already shows the current catalog)."""
|
||||
"""initSyncButton: the sync terminals render the last result; the
|
||||
RUNNING sync re-enters with the live file AND the whole status
|
||||
(phase 98 task 02: the label is phase-aware — a mid-summaries
|
||||
reload re-enters with the summaries label). With the sync IDLE it
|
||||
fetches the upload status: a RUNNING upload run re-attaches
|
||||
(running state, upload kind — phase 90: no live file, the label
|
||||
stays bare "Importing…" — the synthetic running frame, the poll
|
||||
starts); a terminal upload is a no-op — the fall-through is the
|
||||
plain idle settle (the boot-time loadTree() already shows the
|
||||
current catalog)."""
|
||||
body = _fn(_js(), "initSyncButton")
|
||||
assert "await fetchIsAdmin()" in body, "admin-only (no extra fetch)"
|
||||
assert 'fetch("/api/git-sources/upload/status")' in body
|
||||
@@ -356,6 +426,61 @@ def test_reattach_adopts_a_running_upload_only() -> None:
|
||||
assert body.rstrip().removesuffix("}").rstrip().endswith("applySyncIdle(status);")
|
||||
|
||||
|
||||
def test_tick_sync_running_passes_the_status_for_the_phase_label() -> None:
|
||||
"""Branch 1 of the two-job tick (phase 98 task 02): the sync
|
||||
running state passes the WHOLE syncStatus object as the fifth
|
||||
argument — the phase-aware label (overview / summaries) reads its
|
||||
phase fields. The upload running branch stays on the BARE four-arg
|
||||
label (its status has no phase — D2): no fifth argument, no status
|
||||
object anywhere in the branch."""
|
||||
tick = _tick(_js())
|
||||
b_sync_run = tick.find('syncStatus.state === "running"')
|
||||
b_up_run = tick.find('uploadStatus && uploadStatus.state === "running"')
|
||||
b_sync_ok = tick.find('syncStatus.state === "success"')
|
||||
assert -1 < b_sync_run < b_up_run < b_sync_ok
|
||||
branch1 = tick[b_sync_run:b_up_run]
|
||||
i_call = branch1.find("enterSyncRunningState(")
|
||||
call = branch1[i_call:branch1.find(");", i_call)]
|
||||
assert re.search(
|
||||
r'"sync",\s*syncStatus\.current_file,\s*syncStatus\.files_done,\s*'
|
||||
r"syncStatus\.files_total,\s*syncStatus\s*$",
|
||||
call,
|
||||
), "branch 1 passes the whole status (fifth arg) for the phase-aware label"
|
||||
branch2 = tick[b_up_run:b_sync_ok]
|
||||
i_call2 = branch2.find("enterSyncRunningState(")
|
||||
call2 = branch2[i_call2:branch2.find(");", i_call2)]
|
||||
assert re.fullmatch(
|
||||
r'enterSyncRunningState\(\s*"upload",\s*uploadStatus\.current_file,\s*'
|
||||
r"uploadStatus\.files_done,\s*uploadStatus\.files_total\s*$",
|
||||
call2,
|
||||
), "the upload job keeps the bare four-arg label — no status, no phase (D2)"
|
||||
assert "phase" not in call2 and "summaries" not in call2
|
||||
|
||||
|
||||
def test_reattach_sync_running_passes_the_status_for_the_phase_label() -> None:
|
||||
"""initSyncButton (phase 98 task 02): the RUNNING sync re-attach
|
||||
passes the WHOLE status object — a reload mid-summaries re-enters
|
||||
the running state with the summaries label (the never-stale
|
||||
contract); the RUNNING upload re-attach stays on the bare four-arg
|
||||
label (its status has no phase — D2)."""
|
||||
body = _fn(_js(), "initSyncButton")
|
||||
i_sync = body.find('"sync", status.current_file')
|
||||
assert i_sync != -1
|
||||
call = body[i_sync:body.find(");", i_sync)]
|
||||
assert re.search(
|
||||
r'"sync",\s*status\.current_file,\s*status\.files_done,\s*'
|
||||
r"status\.files_total,\s*status\s*$",
|
||||
call,
|
||||
), "the RUNNING sync re-attach passes the status object (fifth arg)"
|
||||
i_up = body.find('"upload", upload.current_file')
|
||||
assert i_up != -1
|
||||
call2 = body[i_up:body.find(");", i_up)]
|
||||
assert re.fullmatch(
|
||||
r'"upload",\s*upload\.current_file,\s*upload\.files_done,\s*upload\.files_total\s*$',
|
||||
call2,
|
||||
), "the RUNNING upload re-attach keeps the bare four-arg label (D2)"
|
||||
|
||||
|
||||
# ---------- the section header + the page comment ----------
|
||||
|
||||
|
||||
|
||||
@@ -283,3 +283,135 @@ def test_cross_check_nested_level_matches_group_folder_listing() -> None:
|
||||
("one/a.md", "A"),
|
||||
("one/b.md", "B"),
|
||||
]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------
|
||||
# summary_pending (phase 98, task 03) — the D3 rule (ONE concept):
|
||||
# a SOURCE or FOLDER node is pending iff its recursive document count
|
||||
# ≥ MIN_DOCS_PER_FOLDER (2) AND it has NO stored folder_summaries row
|
||||
# (AI or manual — any row). That is exactly
|
||||
# ``app.rag.folder_summaries.missing_folder_summaries``'s candidate
|
||||
# set — the marker never drifts from the gap-fill (the integration
|
||||
# cross-check in ``tests/integration/test_docs_api.py`` pins it
|
||||
# end to end). FILE nodes carry no flag.
|
||||
# --------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_folder_two_docs_no_stored_row_is_pending() -> None:
|
||||
"""A ≥ 2-doc folder with NO stored row → ``summary_pending`` true
|
||||
(the marker's "waiting to generate" semantics); file nodes carry
|
||||
no flag at all (the file table has no description column)."""
|
||||
rows = [
|
||||
("S", "one/a.md", "A", 1, T0),
|
||||
("S", "one/b.md", "B", 1, T0),
|
||||
]
|
||||
(source,) = build_kb_tree(["S"], rows, {})
|
||||
(one,) = _folder_nodes(source)
|
||||
assert one.documents == 2
|
||||
assert one.summary is None
|
||||
assert one.summary_pending is True
|
||||
file = _file_nodes(one)[0]
|
||||
assert "summary_pending" not in file.__class__.model_fields
|
||||
assert "summary_pending" not in file.model_dump()
|
||||
|
||||
|
||||
def test_folder_with_stored_row_is_not_pending() -> None:
|
||||
"""The same ≥ 2-doc folder WITH a stored row — ANY row, the builder
|
||||
cannot tell AI from manual — is NOT pending (a description exists).
|
||||
A row on the folder does not cover the source root: with no
|
||||
``(source, "")`` row the SOURCE node stays pending."""
|
||||
rows = [
|
||||
("S", "one/a.md", "A", 1, T0),
|
||||
("S", "one/b.md", "B", 1, T0),
|
||||
]
|
||||
(source,) = build_kb_tree(["S"], rows, {("S", "one"): "Manual."})
|
||||
(one,) = _folder_nodes(source)
|
||||
assert one.summary == "Manual."
|
||||
assert one.summary_pending is False
|
||||
# The source root (2 docs) has no (source, "") row of its own.
|
||||
assert source.summary is None
|
||||
assert source.summary_pending is True
|
||||
|
||||
|
||||
def test_single_document_folder_never_pending() -> None:
|
||||
"""A < 2-document folder is NEVER pending (it never gets a summary
|
||||
— its one file line IS its description), even with no stored row
|
||||
— while its ≥ 2-doc source root (no root row) still is."""
|
||||
rows = [
|
||||
("S", "solo/only.md", "Only", 1, T0), # 1-doc folder
|
||||
("S", "top.md", "Top", 1, T0), # source total = 2
|
||||
]
|
||||
(source,) = build_kb_tree(["S"], rows, {})
|
||||
(solo,) = _folder_nodes(source)
|
||||
assert solo.documents == 1
|
||||
assert solo.summary_pending is False
|
||||
assert source.summary_pending is True
|
||||
|
||||
|
||||
def test_name_collision_pending_follows_recursive_count() -> None:
|
||||
"""The phase-94 count-rule edge: documents ``one/a`` AND ``one/a/b``
|
||||
→ folder ``one/a`` exists and its recursive count is 2 (the document
|
||||
whose path EQUALS the folder name counts — the ``path == folder``
|
||||
arm) → pending true with no stored row, even though the folder has
|
||||
only ONE direct file — pending follows the RECURSIVE count, not
|
||||
the number of direct children. A stored row on the NESTED folder
|
||||
alone clears only that marker (the rule is per node)."""
|
||||
rows = [
|
||||
("S", "one/a", "File A", 1, T0), # a file wearing the folder's name
|
||||
("S", "one/a/b.md", "B", 1, T0), # makes ``one/a`` a folder
|
||||
]
|
||||
(source,) = build_kb_tree(["S"], rows, {})
|
||||
one = _folder_nodes(source)[0]
|
||||
assert one.path == "one"
|
||||
assert one.documents == 2
|
||||
assert one.summary_pending is True
|
||||
a = _folder_nodes(one)[0]
|
||||
assert a.path == "one/a"
|
||||
assert a.documents == 2 # "one/a" itself + "one/a/b.md"
|
||||
assert len(_file_nodes(a)) == 1 # ONE direct file — the count is not that
|
||||
assert a.summary_pending is True
|
||||
# Stored row on the nested folder only: it clears that node, and
|
||||
# only that node.
|
||||
(nested,) = build_kb_tree(["S"], rows, {("S", "one/a"): "Nested."})
|
||||
one2 = _folder_nodes(nested)[0]
|
||||
a2 = _folder_nodes(one2)[0]
|
||||
assert a2.summary == "Nested."
|
||||
assert a2.summary_pending is False
|
||||
assert one2.summary_pending is True
|
||||
|
||||
|
||||
def test_source_root_pending_and_zero_document_source_never() -> None:
|
||||
"""The source root: a source with ≥ 2 docs and NO ``(source, "")``
|
||||
row → the SOURCE node is pending; the stored root row clears it.
|
||||
A registered 0-document source is NEVER pending (0 < the minimum —
|
||||
there is nothing to summarize), with or without a manual row."""
|
||||
rows = [
|
||||
("Full", "x/1.md", "1", 1, T0),
|
||||
("Full", "y.md", "Y", 1, T0),
|
||||
]
|
||||
full, empty = build_kb_tree(["Full", "Empty"], rows, {})
|
||||
assert full.documents == 2
|
||||
assert full.summary_pending is True
|
||||
assert empty.documents == 0
|
||||
assert empty.summary_pending is False
|
||||
full2, empty2 = build_kb_tree(
|
||||
["Full", "Empty"], rows, {("Full", ""): "Root desc."}
|
||||
)
|
||||
assert full2.summary == "Root desc."
|
||||
assert full2.summary_pending is False
|
||||
assert empty2.summary_pending is False
|
||||
|
||||
|
||||
def test_two_sources_pending_independently() -> None:
|
||||
"""Pending is computed per source: with one source holding a root
|
||||
row and the other not, only the rowless source's node is pending —
|
||||
the markers never leak across sources."""
|
||||
rows = [
|
||||
("A", "a1.md", "A1", 1, T0),
|
||||
("A", "a2.md", "A2", 1, T0),
|
||||
("B", "b1.md", "B1", 1, T0),
|
||||
("B", "b2.md", "B2", 1, T0),
|
||||
]
|
||||
a, b = build_kb_tree(["A", "B"], rows, {("A", ""): "A root."})
|
||||
assert (a.summary, a.summary_pending) == ("A root.", False)
|
||||
assert (b.summary, b.summary_pending) == (None, True)
|
||||
|
||||
+221
-21
@@ -68,6 +68,39 @@ affordance, mirrored) — the source pins this module gains for it:
|
||||
``reset()`` handle called on every re-render (PLAN §7.4);
|
||||
* the five ``.kb-summary-*`` classes in styles.css (house palette, no
|
||||
new hue in the phase-97 block).
|
||||
|
||||
Phase 98 (task 04) adds the "SUMMARY PENDING" markers — the pins this
|
||||
module gains for them:
|
||||
|
||||
* the row Description cell's THREE text states (``makeDescCell``):
|
||||
stored → the stored text (NEVER the marker); no stored +
|
||||
``summary_pending`` → the ``kb-summary-pending`` class on the
|
||||
existing text span + the exact ``Summary pending`` copy + the exact
|
||||
D4 title (textContent/title only — the house rule); neither → the
|
||||
empty cell (the ls rule, unchanged) — with the Edit button
|
||||
UNCONDITIONAL in all three (a manual save creates the row);
|
||||
* the in-place clear: the editor's success path sets
|
||||
``node.summary_pending = false`` immediately after
|
||||
``node.summary = data.summary`` (no re-fetch — the marker clears in
|
||||
the surface where the edit happened);
|
||||
* the level block's OR condition (a stored description OR
|
||||
``summary_pending``) + the EXACT pending note — neither stored nor
|
||||
pending stays hidden (the ls rule, unchanged) — and the level's
|
||||
pending text takes the muted ``kb-summary-pending`` class on render
|
||||
(the block is REUSED across levels — a stored level clears it);
|
||||
* the editor's close re-derives the display state from the node (task
|
||||
05 defect fix): the pending marker's muted class + D4 tooltip
|
||||
CANNOT survive a save (the flag is cleared first — the in-place
|
||||
clear, with the stale "next sync" tooltip gone), and a cancel
|
||||
restores the surface's pending display (each surface passes its
|
||||
pending copy via ``pendingText`` — the row's ``Summary pending``
|
||||
marker, the level's D4 note — and its tooltip via ``pendingTitle``,
|
||||
the row only);
|
||||
|
||||
* ``.kb-summary-pending`` in styles.css (the ink-soft muted AA pair —
|
||||
text + color, never color alone; no font/white-space/italic
|
||||
overrides, so the row height is unchanged; no new hue in the
|
||||
phase-97 block).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -358,27 +391,45 @@ def test_top_level_lists_sources_and_hides_the_file_table() -> None:
|
||||
)
|
||||
|
||||
|
||||
def test_level_block_shows_the_stored_description_only() -> None:
|
||||
def test_level_block_shows_the_stored_description_or_the_pending_note() -> None:
|
||||
"""Inside a source/folder: the level block shows the CURRENT
|
||||
level's stored description — the title is the full source-relative
|
||||
path (e.g. `alpha/two`; the source root shows the source name) —
|
||||
and is HIDDEN when none is stored (the ls rule: count only, no
|
||||
placeholder)."""
|
||||
or, since Phase 98 (task 04, D4), the PENDING note when the level
|
||||
is summary_pending (no stored description yet — the next sync's
|
||||
gap-fill will generate it, or the owner writes one via the block's
|
||||
Edit button). HIDDEN only when NEITHER is stored nor pending (the
|
||||
ls rule: count only, no placeholder — unchanged)."""
|
||||
js = _js()
|
||||
start = js.find("function renderLevel(")
|
||||
body = js[start : js.find("function renderEmpty(")]
|
||||
assert "if (node.summary) {" in body, "the ls rule: no description → no block"
|
||||
summary_if = body.find("if (node.summary) {")
|
||||
shown = body[summary_if : body.find("} else {", summary_if)]
|
||||
cond = "if (node.summary || node.summary_pending) {"
|
||||
assert cond in body, "the block shows for a stored OR a pending level"
|
||||
shown = body[body.find(cond) : body.find("} else {", body.find(cond))]
|
||||
assert "levelTitleEl.textContent = current.folder" in shown
|
||||
assert "current.source + \"/\" + current.folder" in shown, (
|
||||
"the folder title is the full source-relative path"
|
||||
)
|
||||
assert ": current.source;" in shown, "the source root title is the source name"
|
||||
assert "levelSummaryEl.textContent = node.summary;" in shown
|
||||
assert "levelSummaryEl.textContent = node.summary ||" in shown, (
|
||||
"the stored text first, the pending note as the fall-through"
|
||||
)
|
||||
assert (
|
||||
"No description stored yet — the next sync will generate one. (You can write one yourself.)"
|
||||
in shown
|
||||
), "the EXACT D4 pending note"
|
||||
assert (
|
||||
'levelSummaryEl.className = node.summary ? "" : "kb-summary-pending";'
|
||||
in shown
|
||||
), (
|
||||
"the pending text is the muted marker style — and a stored level CLEARS a "
|
||||
"previous pending level's class (the block is reused, task-05 defect fix)"
|
||||
)
|
||||
assert "levelEl.hidden = false;" in shown
|
||||
hidden = body[body.find("} else {", body.find("if (node.summary) {")) :]
|
||||
assert "levelEl.hidden = true;" in hidden[:200], "no stored row → the block is hidden"
|
||||
hidden = body[body.find("} else {", body.find(cond)) :]
|
||||
assert "levelEl.hidden = true;" in hidden[:200], (
|
||||
"neither stored nor pending → the block is hidden (the ls rule)"
|
||||
)
|
||||
|
||||
|
||||
def test_level_lists_subfolders_and_direct_files_only() -> None:
|
||||
@@ -595,8 +646,9 @@ def test_tree_cells_never_use_innerhtml_with_derived_data() -> None:
|
||||
for sink in (
|
||||
"link.textContent = s.name",
|
||||
"link.textContent = f.path.split(\"/\").pop()",
|
||||
'text.textContent = (node && node.summary) || ""',
|
||||
"levelSummaryEl.textContent = node.summary;",
|
||||
"text.textContent = node.summary",
|
||||
'text.textContent = "Summary pending"',
|
||||
"levelSummaryEl.textContent = node.summary ||",
|
||||
"a.textContent = label",
|
||||
"span.textContent = label",
|
||||
"countTd.textContent = String(s.documents)",
|
||||
@@ -686,30 +738,135 @@ def test_level_block_ships_the_static_edit_button() -> None:
|
||||
|
||||
def test_row_description_cell_builds_text_and_always_present_edit() -> None:
|
||||
"""makeDescCell (the shared Description-cell builder, called from
|
||||
BOTH row builders): the stored description as a text node
|
||||
(textContent — never innerHTML) + the Edit button added
|
||||
BOTH row builders): the description text as a text node
|
||||
(textContent — never innerHTML) + the Edit button built
|
||||
UNCONDITIONALLY — a description can be CREATED where none is
|
||||
stored (a < 2-document folder, the generator's fail-soft miss), so
|
||||
there is NO gate on the stored value. The button is a real
|
||||
type=button with a human aria-label, and the shared editor is
|
||||
wired with a CONSTANT target { node, source, folder }."""
|
||||
the button sits OUTSIDE the three-state text branch (Phase 98:
|
||||
the text gates on the node state, the button never does). The
|
||||
button is a real type=button with a human aria-label, and the
|
||||
shared editor is wired with a CONSTANT target { node, source,
|
||||
folder }."""
|
||||
body = _fn(_js(), "makeDescCell")
|
||||
assert 'text.textContent = (node && node.summary) || ""' in body, (
|
||||
"the stored description is a text node (or an empty cell)"
|
||||
# The Edit button is built unconditionally — the whole button
|
||||
# block (from its creation to the append) carries no `if` gate on
|
||||
# the stored description or the pending flag.
|
||||
btn_block = body[
|
||||
body.find("const btn = document.createElement") : body.find("td.append(text, btn)")
|
||||
]
|
||||
assert "if (" not in btn_block, (
|
||||
"always present: no gate on the stored description"
|
||||
)
|
||||
assert 'btn.type = "button"' in body
|
||||
assert 'btn.className = "kb-summary-edit"' in body
|
||||
assert 'btn.textContent = "Edit"' in body
|
||||
assert 'btn.setAttribute("aria-label", `Edit description: ${label}`)' in body
|
||||
assert "td.append(text, btn)" in body
|
||||
assert "if (node" not in body, (
|
||||
"always present: no gate on the stored description"
|
||||
)
|
||||
assert "getTarget: () => ({ node, source, folder })" in body, (
|
||||
"a row's target is a constant (its own node)"
|
||||
)
|
||||
|
||||
|
||||
def test_desc_cell_pending_marker_branch() -> None:
|
||||
"""Phase 98 (task 04, D4): makeDescCell's THREE text states, in
|
||||
order — (1) a stored summary → the stored text, NEVER the
|
||||
marker; (2) no stored summary AND node.summary_pending → the
|
||||
marker: the kb-summary-pending class on the EXISTING text span +
|
||||
the exact "Summary pending" copy + the exact D4 title
|
||||
(textContent/title only — the house rule, no innerHTML); (3)
|
||||
neither stored nor pending → the empty cell (the ls rule,
|
||||
unchanged)."""
|
||||
body = _fn(_js(), "makeDescCell")
|
||||
stored_i = body.find("if (node && node.summary) {")
|
||||
pending_i = body.find("else if (node && node.summary_pending) {")
|
||||
empty_i = body.find("} else {")
|
||||
assert -1 < stored_i < pending_i < empty_i, ("stored → pending → empty, in order")
|
||||
stored = body[stored_i:pending_i]
|
||||
assert "text.textContent = node.summary;" in stored, "stored → the stored text"
|
||||
assert "kb-summary-pending" not in stored, "a stored summary is NEVER the marker"
|
||||
assert "Summary pending" not in stored
|
||||
pending = body[pending_i:empty_i]
|
||||
assert 'text.className = "kb-summary-pending"' in pending
|
||||
assert 'text.textContent = "Summary pending"' in pending, "the D4 marker copy"
|
||||
assert (
|
||||
'text.title = "No stored description yet — the next sync will generate one."'
|
||||
in pending
|
||||
), "the D4 title"
|
||||
empty = body[empty_i : empty_i + 160]
|
||||
assert 'text.textContent = ""' in empty, "neither stored nor pending → the empty cell"
|
||||
# The marker is textContent/title only — the house rule (no
|
||||
# innerHTML anywhere in the cell builder).
|
||||
code = re.sub(r"//.*?$|/\*.*?\*/", "", body, flags=re.S | re.M)
|
||||
assert "innerHTML" not in code
|
||||
|
||||
|
||||
def test_save_success_clears_the_pending_flag_in_place() -> None:
|
||||
"""Phase 98 (task 04, D4): a successful save syncs the in-memory
|
||||
node IN PLACE — node.summary = data.summary AND, immediately
|
||||
after, node.summary_pending = false (a created/updated description
|
||||
is no longer pending: the marker clears in the surface where the
|
||||
edit happened, NO re-fetch). The flag clear sits between the
|
||||
summary sync and the announcement, and nothing after it re-fetches
|
||||
the tree (the re-fetch stays the safety net, not the in-place
|
||||
clear)."""
|
||||
body = _fn(_js(), "wireDescriptionEdit")
|
||||
save = body[body.find("async function saveDescription()") :]
|
||||
sync_i = save.find("node.summary = data.summary")
|
||||
clear_i = save.find("node.summary_pending = false")
|
||||
announce_i = save.find(
|
||||
'closeEditor(data.summary === null ? "Description cleared." : "Description updated.")'
|
||||
)
|
||||
assert -1 < sync_i < clear_i < announce_i, (
|
||||
"summary sync → pending-flag clear → announce (in place)"
|
||||
)
|
||||
tail = save[clear_i:]
|
||||
assert "loadTree()" not in tail and 'fetch("' not in tail, (
|
||||
"the in-place clear is the whole of it — no re-fetch"
|
||||
)
|
||||
|
||||
|
||||
def test_close_editor_rederives_the_pending_display() -> None:
|
||||
"""Phase 98 (task 05 defect fix, D4): closeEditor re-renders the
|
||||
display state from the node with the SAME three-state rule the
|
||||
surfaces use — the pending marker's muted class + D4 tooltip
|
||||
CANNOT survive a save (the success path cleared
|
||||
``node.summary_pending`` first, so a stale "next sync" tooltip
|
||||
under a just-created description is impossible), and a cancel
|
||||
RESTORES the surface's pending display. Each surface passes its
|
||||
pending copy via ``pendingText`` (the row's ``Summary pending``
|
||||
marker, the level's D4 note) and its tooltip via ``pendingTitle``
|
||||
(the row only — D4's title is the row cell's, the level's <p>
|
||||
carries none). textContent/class/title only — the house rule."""
|
||||
body = _fn(_js(), "wireDescriptionEdit")
|
||||
close = body[body.find("function closeEditor(") : body.find("function openEditor()")]
|
||||
flag_i = close.find('const pending = node !== null && stored === "" && node.summary_pending;')
|
||||
value_i = close.find("const value = pending && pendingText ? pendingText : stored;")
|
||||
class_i = close.find('textEl.className = pending ? "kb-summary-pending" : "";')
|
||||
title_i = close.find("if (pendingTitle) textEl.title = pendingTitle;")
|
||||
clear_title_i = close.find('textEl.removeAttribute("title")')
|
||||
render_i = close.find("textEl.textContent = value")
|
||||
assert -1 < flag_i < value_i < class_i < title_i < clear_title_i < render_i, (
|
||||
"flag → value → class → title (set/clear) → text, in order"
|
||||
)
|
||||
# The row passes its marker copy + the EXACT D4 tooltip…
|
||||
cell = _fn(_js(), "makeDescCell")
|
||||
assert 'pendingText: "Summary pending"' in cell
|
||||
assert (
|
||||
'pendingTitle: "No stored description yet — the next sync will generate one."'
|
||||
in cell
|
||||
), "the D4 tooltip is the row's"
|
||||
# …and the level passes the D4 note WITHOUT a tooltip.
|
||||
js = _js()
|
||||
wire = js[js.find("levelEditor = wireDescriptionEdit({") : js.find("/* ---------- view boot")]
|
||||
assert 'pendingText:' in wire and "pendingTitle" not in wire, (
|
||||
"the level's <p> carries the note, never a tooltip (D4)"
|
||||
)
|
||||
assert (
|
||||
"No description stored yet — the next sync will generate one. (You can write one yourself.)"
|
||||
in wire
|
||||
), "the level's pending copy is the D4 note"
|
||||
|
||||
|
||||
def test_editor_swap_builds_textarea_save_cancel_and_live_region() -> None:
|
||||
"""Edit swaps the description UI for the inline editor: a
|
||||
<textarea class="kb-summary-editor"> prefilled via .value (NEVER
|
||||
@@ -1012,3 +1169,46 @@ def test_styles_carry_the_folder_editor_classes() -> None:
|
||||
# suppression for these controls).
|
||||
assert ":focus-visible {" in css
|
||||
assert "outline: 3px solid var(--brand)" in css
|
||||
|
||||
|
||||
# ---------- the "Summary pending" markers (phase 98, task 04) ----------
|
||||
|
||||
|
||||
def test_styles_carry_the_pending_marker_class() -> None:
|
||||
"""Phase 98 (task 04, D4): .kb-summary-pending mutes the row-cell
|
||||
marker via the ink-soft token (5.1:1 on --surface — AA; text +
|
||||
color, never color alone — B5). The class sits on the EXISTING
|
||||
description text span, so the row cell's font-size/line-height
|
||||
apply — no font/white-space/italic overrides (the marker must not
|
||||
change row height), and the phase-97 block carries no new hue
|
||||
(the phase-92 monochrome invariant)."""
|
||||
css = _text(STYLES_CSS)
|
||||
m = re.search(r"(?<![\w-])\.kb-summary-pending\s*\{([^}]*)\}", css)
|
||||
assert m, "styles.css must define .kb-summary-pending"
|
||||
rule = m.group(1)
|
||||
assert "color: var(--ink-soft)" in rule, "the muted AA pair (5.1:1 on --surface)"
|
||||
for prop in ("font-style", "font-size", "line-height", "white-space"):
|
||||
assert prop not in rule, f"the marker keeps the row's {prop} (row height unchanged)"
|
||||
block = _css_block(css, "KB drill-down tree (phase 97", "Git sources page (phase 35)")
|
||||
assert ".kb-summary-pending" in block, "the marker class lives in the phase-97 region"
|
||||
assert not re.search(r"#[0-9a-fA-F]{3,8}\b", block), "no new hue (phase-92 invariant)"
|
||||
assert not re.search(r"rgba?\(", block), "no new hue (phase-92 invariant)"
|
||||
|
||||
|
||||
def test_module_docstring_documents_the_pending_markers() -> None:
|
||||
"""The house per-phase module-note convention: the phase-98
|
||||
task-04 section records the marker's surfaces — the row cell's
|
||||
D4 copy + title, the level block's pending note — and the
|
||||
in-place clear on the editor's success path."""
|
||||
js = _js()
|
||||
header = js[: js.find("import { fetchIsAdmin }")]
|
||||
assert "Phase 98 (task 04)" in header
|
||||
assert '"Summary pending"' in header, "the D4 marker copy"
|
||||
assert "No stored description yet — the next sync will generate one." in header, (
|
||||
"the D4 row-cell title"
|
||||
)
|
||||
assert (
|
||||
"No description stored yet — the next sync will generate one. (You can write one yourself.)"
|
||||
in header
|
||||
), "the level block's pending note"
|
||||
assert "node.summary_pending = false" in header, "the in-place clear"
|
||||
|
||||
@@ -21,6 +21,19 @@ null/0/0); mid-run the status reports the file the (mock) import is
|
||||
processing, through the runner's own hook closure (no file yet during
|
||||
the clone/pull phase — A4); the terminal states clear
|
||||
``current_file`` while keeping the run's final counts.
|
||||
|
||||
Phase 98 (task 01): the status's phase machine joins the pins — the
|
||||
four new keys (``phase`` / ``current_summary`` / ``summaries_done`` /
|
||||
``summaries_total``) ride along as null/0/0/0 idle, the prelude keeps
|
||||
``phase: null``, the import reports ``phase: "import"``, and the
|
||||
terminal states clear ``phase`` + ``current_summary`` while keeping
|
||||
the run's final summary counts.
|
||||
|
||||
Phase 98 (task 02): the running-state title rule is adjusted — the
|
||||
FULL untruncated LABEL (the file label, "Writing KB overview…", or
|
||||
"Summarizing folders… <folder> (n/m)") rides the button title for
|
||||
EVERY running label; the phase-64 "title only when there is a current
|
||||
file" (remove-when-null) rule is gone.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -256,18 +269,29 @@ def test_sources_js_running_state_is_never_stale() -> None:
|
||||
fresh run starts clean: the previous failure's title / aria-label /
|
||||
.is-error come off NOW, not when the run settles.
|
||||
|
||||
Phase 64 (task 04): the button title carries the FULL untruncated
|
||||
current file (removed when null — no file yet) and #sync-result
|
||||
(the aria-live announcer) carries the same untruncated label."""
|
||||
Phase 64 (task 04): #sync-result (the aria-live announcer) carries
|
||||
the FULL untruncated label. Phase 98 (task 02): the title rule is
|
||||
adjusted — the FULL untruncated LABEL (the file label, the overview
|
||||
label, or the summaries label — whatever the running form is) rides
|
||||
the button title for EVERY running label; the "title only when
|
||||
there is a current file" (remove-when-null) rule is gone."""
|
||||
js = _text(SOURCES_JS)
|
||||
body = _body(js, "enterSyncRunningState")
|
||||
assert "syncBtn.disabled = true" in body
|
||||
assert 'syncBtn.setAttribute("aria-busy", "true")' in body
|
||||
assert "syncBtn.title = currentFile" in body, "the full path on hover"
|
||||
assert 'syncBtn.removeAttribute("title")' in body, "removed when no file yet"
|
||||
title_if = body.find("if (currentFile) syncBtn.title = currentFile")
|
||||
title_else = body.find("syncBtn.removeAttribute(\"title\")")
|
||||
assert -1 < title_if < title_else, "title is set, not removed, only when a file exists"
|
||||
assert "syncBtn.title = label" in body, (
|
||||
"the full untruncated label on hover, for every running label"
|
||||
)
|
||||
assert 'syncBtn.removeAttribute("title")' not in body, (
|
||||
"the title is set for EVERY running label (phase 98: the file-only "
|
||||
"rule is adjusted away)"
|
||||
)
|
||||
title_i = body.find("syncBtn.title = label")
|
||||
span_i = body.find("syncLabel.textContent = label")
|
||||
result_i = body.find("syncResult.textContent = label")
|
||||
assert -1 < title_i < span_i < result_i, (
|
||||
"one label: built once, written to title, span AND the announcer"
|
||||
)
|
||||
assert 'syncBtn.setAttribute("aria-label", "Sync sources")' in body
|
||||
assert "syncBtn.classList.remove(\"is-error\")" in body
|
||||
assert "syncIcon.classList.add(\"is-spinning\")" in body
|
||||
@@ -793,7 +817,12 @@ def _patch_sync_seams(
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
# Phase 98 (task 01): the runner wires the progress hook into
|
||||
# the generation branches — the fake accepts it (the state-
|
||||
# machine tests' canned run never fires it: zero summary
|
||||
# counters, like a no-gap skip).
|
||||
return {"generated": 0, "failed": 0, "pruned": 0}
|
||||
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folder_summaries)
|
||||
@@ -835,8 +864,9 @@ def test_idle_status_pins_full_shape_including_progress_keys(
|
||||
fresh_sync_status: None,
|
||||
) -> None:
|
||||
"""Idle: the three phase-64 progress keys ride along as null/0/0,
|
||||
and EVERY pre-existing key is unchanged — the full response dict is
|
||||
pinned, so the current UI and every existing consumer keep working."""
|
||||
the four phase-98 phase keys as null/0/0/0, and EVERY pre-existing
|
||||
key is unchanged — the full response dict is pinned, so the
|
||||
current UI and every existing consumer keep working."""
|
||||
assert sync_api.sync_status() == {
|
||||
"state": "idle",
|
||||
"started_at": None,
|
||||
@@ -846,6 +876,11 @@ def test_idle_status_pins_full_shape_including_progress_keys(
|
||||
"current_file": None,
|
||||
"files_done": 0,
|
||||
"files_total": 0,
|
||||
# Phase 98 (task 01): the phase-machine keys — null/0/0/0 idle.
|
||||
"phase": None,
|
||||
"current_summary": None,
|
||||
"summaries_done": 0,
|
||||
"summaries_total": 0,
|
||||
}
|
||||
|
||||
|
||||
@@ -869,19 +904,29 @@ def test_mid_run_status_reports_current_file(
|
||||
thread, errors = _start_run()
|
||||
try:
|
||||
assert clone.started.wait(5.0), "the run never reached the clone phase"
|
||||
# Clone/pull phase: running — but no file yet (A4: bare "Syncing…").
|
||||
# Clone/pull prelude: running — but no file yet (A4: bare
|
||||
# "Syncing…") and no phase yet either (D1: the prelude stays
|
||||
# null — the bare label's pin).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] is None
|
||||
assert s["current_file"] is None
|
||||
assert s["current_summary"] is None
|
||||
assert s["files_done"] == 0 and s["files_total"] == 0
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
clone.release.set()
|
||||
assert fake_import.started.wait(5.0), "the run never reached the import"
|
||||
# Import phase: the hook's file is live on the status.
|
||||
# Import phase: the hook's file is live on the status, and the
|
||||
# phase machine names the phase (the summary span has not
|
||||
# started — its counters are still 0/0).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "import"
|
||||
assert s["current_file"] == "repo/notes/deep.md"
|
||||
assert s["files_done"] == 1
|
||||
assert s["files_total"] == 3
|
||||
assert s["current_summary"] is None
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
fake_import.release.set()
|
||||
finally:
|
||||
clone.release.set()
|
||||
@@ -894,11 +939,17 @@ def test_mid_run_status_reports_current_file(
|
||||
# assigned to the status above).
|
||||
assert fake_import.prune_flags == [True]
|
||||
assert fake_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
|
||||
# Success terminal: current_file null, final counts retained.
|
||||
# Success terminal: current_file null, final file counts retained;
|
||||
# the phase machine clears phase + current_summary — the fake
|
||||
# folder step never fired the summary hook, so the summary counts
|
||||
# stay 0/0 (the keep-final-counts convention keeps what fired).
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "success"
|
||||
assert s["phase"] is None
|
||||
assert s["current_summary"] is None
|
||||
assert s["current_file"] is None
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
assert s["error"] is None
|
||||
assert s["started_at"] is not None and s["finished_at"] is not None
|
||||
|
||||
@@ -942,7 +993,10 @@ def test_failed_terminal_clears_current_file_keeps_counts(
|
||||
s = sync_api.sync_status()
|
||||
assert s["state"] == "failed"
|
||||
assert s["current_file"] is None # cleared in the terminal state
|
||||
assert s["phase"] is None # phase 98: cleared in the terminal state
|
||||
assert s["current_summary"] is None
|
||||
assert s["files_done"] == 1 and s["files_total"] == 3 # final counts kept
|
||||
assert s["summaries_done"] == 0 and s["summaries_total"] == 0
|
||||
assert s["detail"] == {}
|
||||
error = s["error"] or ""
|
||||
assert "*****@aipi.example.com" in error # credentials masked
|
||||
|
||||
Reference in New Issue
Block a user