phase: 187_sync_peak_memory_line
Build and Push Containers / build-and-push-app (push) Successful in 1m47s
Build and Push Containers / build-and-push-db (push) Successful in 13s

**Phase 187 — final verification pass: all criteria met, repo 100% green.**

Verified (all four tasks already in `complete/`)
- `app/core/memory.py`: `rss_mb` / `rss_peak_mb` / `cgroup_peak_mb` / `fmt_mb`, pure stdlib, never-raise, **100%** coverage, 24 branch tests.
- Lifecycle lines append-only: `sync: started … rss_mb=`, all three terminals `… rss_mb= rss_peak_mb=` (failed = sanitized one-liner beside the surviving `logger.exception`), `import: summary … rss_mb=`; prefixes byte-identical.
- `GET /api/sync/status`: additive `rss_mb`/`rss_peak_mb` (`int|null`), keyed off the ONE `_sample_rss` (run start, phase transitions, forced terminals, 5 s-throttled hook), `peak >= current` invariant, anonymous 403.
- `sources.js` `fmtSyncPeakClause`: ` · peak <N> MB` trailing the counts line / "Sync cancelled", nothing when null, no new control/live region.

Gates (exact commands)
- `uv run pytest --cov=app --cov-report=term-missing` → **4570 passed**, TOTAL **99%**, `app/api/sync.py` **100%**, `app/core/memory.py` **100%**
- `uv run ruff check .` → clean · `uv run pyright` → **0 errors**
- `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` → **5 passed in isolation** (ran twice)
- All 27 touched E2E families re-run **in isolation**, all green (progress_box 8, cancel 3, status_freshness 4, summary_visibility 3, sync_button 3, session_reset 4, git_freeze 2, git_log_stream_dates 5, PDF/image suites, …)

Completion criteria: every checkbox ✅ except the commit + phase-dir move, which belong to the harness.

Documentation
- README reviewed in full (pages, commands, config tables cross-checked against `app/config.py` — 85 env rows, all defaults match; phase-187 ops section, greps, process-vs-cgroup caveat): consistent, no changes needed.
- `AGENTS.md`: **fixed one pre-existing inconsistency** — rule 6 claimed the no-CDN test covers "the index page"; it actually parametrizes 10 pages (all but `/doc-edit.html`).

Notable
- Two failures early in the pass were self-inflicted by running the E2E module concurrently with the full suite on the shared Postgres (TRUNCATE deadlock + one order-dependent `test_overview` flake); both vanish under the documented serial isolation — no code defect, no change made.
- **PLAN §9 still needs the owner's reconciliation** (phase 187 adds the sync memory fields + two status keys; PLAN is never agent-edited — AGENTS rule 10 carries the spec). `TODO.md` holds an unrelated owner note, left untouched.

Next pending phase: `188_chunks_hnsw_index`.
This commit is contained in:
2026-10-16 15:22:51 -04:00
parent 644f7de696
commit c054b47df5
72 changed files with 3826 additions and 96 deletions
+61
View File
@@ -0,0 +1,61 @@
"""Shared sync-result-line helpers (phase 187, task 03).
The Sources page's sync result line — the ``fmtSyncResult`` counts line
and the neutral ``"Sync cancelled"`` line the ``#sync-progress`` box
speaks — now trails the run's PEAK-MEMORY clause (`` · peak 1936 MB``),
sourced from ``GET /api/sync/status``'s additive ``rss_peak_mb`` /
``rss_mb`` pair (the 2026-10-15 incident: "how much RAM did that sync
use?" used to mean reading ``systemd`` unit-teardown lines off three
boots; now it is on the page that started the run and in the log).
The clause is per-run DATA, so the suites that pin the OWNER-VISIBLE
counts pin the counts plus an OPTIONAL clause:
* :func:`counts_line` — a Playwright-ready pattern: the exact expected
line, with `` · peak <N> MB`` allowed at the end (and the whole line
anchored, so a mutated counts line still fails);
* :func:`strip_peak_clause` — for the suites that compare two runs' or
two apps' lines byte-for-byte: the memory number belongs to the
process, not to the KB, so it is not part of a byte-identity contract.
A NULL pair renders NOTHING (``fmtSyncPeakClause``), which is why the
clause is optional rather than required: an idle app, a status that
predates phase 187, and the upload job all settle without it.
"""
from __future__ import annotations
import re
#: The clause sources.js appends (``fmtSyncPeakClause``), as a regex —
#: optional, because a null peak/current pair appends the empty string.
PEAK_CLAUSE = r"(?: · peak \d+ MB)?"
#: The same clause, for a substitution (``strip_peak_clause``): anchored
#: at end of line, so nothing else is ever stripped.
_PEAK_CLAUSE_TAIL = re.compile(r" · peak \d+ MB$")
def counts_line(expected: str | re.Pattern[str]) -> re.Pattern[str]:
"""Match the EXACT ``expected`` sync result line, clause included or
not — the pin a suite writes when it cares about the counts, not
about the number of megabytes the run happened to reach.
``expected`` is either the literal line (escaped, so the ``·`` and
the digits mean exactly what they say) or a suite's own shape
pattern (its anchors are dropped and re-applied around the optional
clause, so ``^3 added$`` becomes ``^3 added(?: · peak \\d+ MB)?$``).
"""
if isinstance(expected, re.Pattern):
body = expected.pattern
if body.startswith("^"):
body = body[1:]
if body.endswith("$"):
body = body[:-1]
return re.compile(rf"^{body}{PEAK_CLAUSE}$")
return re.compile(rf"^{re.escape(expected)}{PEAK_CLAUSE}$")
def strip_peak_clause(text: str) -> str:
"""``text`` without its trailing peak clause (if any) — the
byte-identity comparison's view of the line."""
return _PEAK_CLAUSE_TAIL.sub("", text)
+2 -1
View File
@@ -90,6 +90,7 @@ from app.models import Document
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
from e2e.mock_llm import IMAGE_DESCRIPTION_ANSWER
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
GIT_SOURCES_URL = "/git-sources.html"
@@ -381,7 +382,7 @@ def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled()
expect(page.locator("#sync-progress-text")).to_have_text(expected_result)
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(expected_result))
def _seed_image_doc(
+2 -1
View File
@@ -102,6 +102,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -447,7 +448,7 @@ def _expect_counts_line(page: Page, expected: str) -> None:
"""The counts line — the owner-visible sync report on the single sync
text surface (phase 150; sources.js fmtSyncResult; the zero "updated"
part is omitted)."""
expect(page.locator("#sync-progress-text")).to_have_text(expected, timeout=15_000)
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(expected), timeout=15_000)
def _goto_sources(page: Page, base_url: str) -> None:
+4 -3
View File
@@ -104,6 +104,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -483,7 +484,7 @@ def test_sync_survives_a_force_push(
assert body["state"] == "success", body
# The fixture doc (one A9-format file) is the only change — the
# counts line rides the box (phase 150).
expect(page.locator("#sync-progress-text")).to_have_text("1 added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("1 added"))
# The REAL clone was imported: the fixture path is in the Sources
# table (the drill-down tree, the test_sync_button pattern).
_expect_fixture_row(page)
@@ -511,7 +512,7 @@ def test_sync_survives_a_force_push(
# …and the digest delta re-indexed the file as updated (the counts
# are the secondary signal — the content assertion below is
# primary). The counts line rides the box (phase 150).
expect(page.locator("#sync-progress-text")).to_have_text("0 added · 1 updated")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("0 added · 1 updated"))
# The KB holds the NEW head: the document modal serves the V2
# sentinel — the reset's content flowed through the import.
_expect_fixture_row(page)
@@ -533,7 +534,7 @@ def test_sync_survives_a_force_push(
# "1 unchanged" means the content hash did not move — drift back
# to V1 would have re-reported as "1 updated" (the no-drift pin).
# The counts line rides the box (phase 150).
expect(page.locator("#sync-progress-text")).to_have_text("0 added · 1 unchanged")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("0 added · 1 unchanged"))
# ---------------------------------------------------------------------------
+2 -1
View File
@@ -137,6 +137,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
from scripts.git_sync import file_commit_dates
from scripts.import_docs import repo_name
@@ -488,7 +489,7 @@ def _run_sync(page: Page, expected_line: str | re.Pattern[str]) -> dict[str, Any
expect(btn).to_be_enabled() # never stale at the terminal state
expect(page.locator("#sync-error-banner")).to_be_hidden()
expect(page.locator("#sync-progress-text")).to_have_text(
expected_line, timeout=UI_SETTLE_TIMEOUT_MS
counts_line(expected_line), timeout=UI_SETTLE_TIMEOUT_MS
)
return dict(body["detail"] or {})
+5 -4
View File
@@ -103,6 +103,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -515,7 +516,7 @@ def test_add_with_branch_syncs_that_branch(
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=15_000)
# Both branch docs are fresh after the per-test truncate — the
# counts line rides the box (phase 150).
expect(page.locator("#sync-progress-text")).to_have_text("2 added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("2 added"))
# --- the branch's content is what got indexed ----------------------
expect(catalog_has_doc(page, FIXTURE_NAME, "release-only.md")).to_have_count(1)
@@ -539,7 +540,7 @@ def test_add_with_branch_syncs_that_branch(
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=15_000)
# Nothing re-embeds (sha256 delta) — the no-op run announces the
# unchanged count (release's two files) on the box (phase 150).
expect(page.locator("#sync-progress-text")).to_have_text("0 added · 2 unchanged")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("0 added · 2 unchanged"))
# The branch's doc survived the prune re-import.
expect(catalog_has_doc(page, FIXTURE_NAME, "release-only.md")).to_have_count(1)
@@ -577,7 +578,7 @@ def test_default_branch_source_unaffected(
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=15_000)
# The default branch's two docs are fresh — the counts line rides
# the box (phase 150).
expect(page.locator("#sync-progress-text")).to_have_text("2 added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("2 added"))
# The default branch (main) is what landed — the pre-phase
# byte-identical guarantee, observable through the UI…
@@ -684,7 +685,7 @@ def test_row_editor_branch_change_fails_loud(
# The KB is still empty (the failed sync aborted before the
# import) — the restored branch's two docs are fresh additions;
# the counts line rides the box (phase 150).
expect(page.locator("#sync-progress-text")).to_have_text("2 added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("2 added"))
# And the branch's doc is what is indexed again.
expect(catalog_has_doc(page, FIXTURE_NAME, "release-only.md")).to_have_count(1)
+2 -1
View File
@@ -99,6 +99,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -330,7 +331,7 @@ def _sync_in_app(page: Page, app_url: str) -> None:
expect(btn).to_be_enabled() # never stale — re-enabled at the terminal state
# Both fixture docs are fresh after the per-test truncate — the
# counts line rides the box (phase 150).
expect(page.locator("#sync-progress-text")).to_have_text("2 added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("2 added"))
def _source_node(tree_json: dict[str, Any]) -> dict[str, Any]:
+2 -1
View File
@@ -59,6 +59,7 @@ from app.models import Chunk, Document
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
from e2e.mock_llm import IMAGE_DESCRIPTION_ANSWER
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
GIT_SOURCES_URL = "/git-sources.html"
@@ -342,7 +343,7 @@ def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled()
expect(page.locator("#sync-progress-text")).to_have_text(expected_result)
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(expected_result))
def _captured_systems(mock_llm: int) -> list[str]:
+2 -1
View File
@@ -51,6 +51,7 @@ from app.rag.agent import IMAGE_DOC_MARKER
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
from e2e.mock_llm import IMAGE_DESCRIPTION_ANSWER
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
GIT_SOURCES_URL = "/git-sources.html"
@@ -280,7 +281,7 @@ def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled()
expect(page.locator("#sync-progress-text")).to_have_text(expected_result)
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(expected_result))
def _drill_to_source(page: Page, name: str) -> None:
+2 -1
View File
@@ -70,6 +70,7 @@ from app.models import Document
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
from e2e.mock_llm import IMAGE_DESCRIPTION_ANSWER, IMAGE_DESCRIPTION_CONTEXT_ANSWER
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -366,7 +367,7 @@ def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled()
expect(page.locator("#sync-progress-text")).to_have_text(expected_result)
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(expected_result))
def _ask(page: Page, app_url: str, question: str) -> None:
+2 -1
View File
@@ -88,6 +88,7 @@ from app.rag.pdf import is_blank_image
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
from e2e.mock_llm import IMAGE_DESCRIPTION_CONTEXT_ANSWER
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
SOURCE_ON = "e2e-pdf151" # the story app's archive stem (archive_source_name)
@@ -427,7 +428,7 @@ def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled()
expect(page.locator("#sync-progress-text")).to_have_text(expected_result)
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(expected_result))
def _drill_to_source(page: Page, name: str) -> None:
+2 -1
View File
@@ -59,6 +59,7 @@ from app.models import Document
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
from e2e.mock_llm import IMAGE_DESCRIPTION_CONTEXT_ANSWER
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
SOURCE_NAME = "e2e-pdf" # the archive stem (archive_source_name)
@@ -324,7 +325,7 @@ def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled()
expect(page.locator("#sync-progress-text")).to_have_text(expected_result)
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(expected_result))
def _drill_to_source(page: Page, name: str) -> None:
+2 -1
View File
@@ -147,6 +147,7 @@ from app.rag.pdf import (
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
from e2e.mock_llm import PICTORIAL_PAGE_ANSWER
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
SOURCE_NAME = "e2e-pictorial" # the archive stem (archive_source_name)
@@ -479,7 +480,7 @@ def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled()
expect(page.locator("#sync-progress-text")).to_have_text(expected_result)
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(expected_result))
def _drill_to_source(page: Page, name: str) -> None:
+2 -1
View File
@@ -101,6 +101,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -403,7 +404,7 @@ def _sync_from_sources_page(page: Page, app_url: str) -> dict[str, Any]:
def _expect_counts_line(page: Page, expected: str) -> None:
"""The Sources page's counts line reads exactly ``expected`` —
"9 added" / "0 added · 3 unchanged · 6 pruned"."""
expect(page.locator("#sync-progress-text")).to_have_text(expected, timeout=15_000)
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(expected), timeout=15_000)
def _add_local_source(page: Page, app_url: str, path: Path) -> str:
+3 -2
View File
@@ -52,6 +52,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -273,7 +274,7 @@ def test_admin_sync_lifecycle(page: Page, app_url: str, db_ready: None) -> None:
# The fixture doc (one A9-format file) is the only change — the
# settle's counts line rides the box (phase 150: the box is the
# sole sync text surface).
expect(page.locator("#sync-progress-text")).to_have_text("1 added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("1 added"))
# The REAL clone was imported: the fixture path is in the Sources
# table (the sentinel lives inside it) — via the drill-down tree
@@ -297,7 +298,7 @@ def test_admin_sync_lifecycle(page: Page, app_url: str, db_ready: None) -> None:
# Nothing re-embedded (sha256 delta) — the no-op run announces the
# unchanged count instead of an empty live region (the box keeps
# its result line — phase 150).
expect(page.locator("#sync-progress-text")).to_have_text("0 added · 1 unchanged")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("0 added · 1 unchanged"))
# The doc survived the prune re-import (its file is still in the
# repo) — via the drill-down tree (phase 97).
_expect_fixture_row(page)
+5 -3
View File
@@ -93,6 +93,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import PEAK_CLAUSE, counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -135,8 +136,9 @@ SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}")
#: The settled SUCCESS result line — "N added" or "N added · M
#: unchanged" (sources.js's fmtSyncResult — updated/pruned are zero on
#: the re-sync: the repo is unchanged and nothing was pruned).
RESULT_LINE = re.compile(r"^(\d+) added(?: · (\d+) unchanged)?$")
#: the re-sync: the repo is unchanged and nothing was pruned) with the
#: phase-187 peak clause optional at the end (null pair → no clause).
RESULT_LINE = re.compile(rf"^(\d+) added(?: · (\d+) unchanged)?{PEAK_CLAUSE}$")
#: Settle budgets: the cancel terminal lands ~1 s after the click and
#: the UI's 2 s poll settles at most one tick after the terminal
@@ -547,7 +549,7 @@ def test_admin_cancel_lifecycle(page: Page, app_url: str, db_ready: None) -> Non
# Phase 150: the neutral result line rides the box (the sole sync
# text surface — the retired announcer span is gone).
expect(page.locator("#sync-progress-text")).to_have_text(
"Sync cancelled", timeout=SYNC_SETTLE_TIMEOUT_MS
counts_line("Sync cancelled"), timeout=SYNC_SETTLE_TIMEOUT_MS
)
# A1: the cancel is never rendered as a failure — no error banner,
# no error modal in the DOM.
+2 -1
View File
@@ -69,6 +69,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -267,7 +268,7 @@ def _expect_settle_line(page: Page, expected: str) -> None:
Phase 150: the box IS the result line's surface (the retired
announcer span is gone)."""
expect(page.locator("#sync-progress-text")).to_have_text(
expected, timeout=SYNC_TIMEOUT_MS
counts_line(expected), timeout=SYNC_TIMEOUT_MS
)
+2 -1
View File
@@ -145,6 +145,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -948,5 +949,5 @@ def test_label_shows_the_chunk_part(
expect(page.locator(".sync-label-chunk")).to_be_hidden()
expect(page.locator(".sync-label-chunk")).to_have_text("")
# The counts line rides the box (phase 150 — the sole sync surface).
expect(page.locator("#sync-progress-text")).to_have_text(f"{N_FILES} added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(f"{N_FILES} added"))
expect(page.locator("#sync-error-banner")).to_be_hidden()
+6 -4
View File
@@ -129,6 +129,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import PEAK_CLAUSE, counts_line
from scripts.import_docs import repo_name
REPO = Path(__file__).resolve().parents[2]
@@ -187,9 +188,10 @@ RUN_SETTLE_TIMEOUT_MS = 90_000
UI_SETTLE_TIMEOUT_MS = 15_000
#: "Synced HH:MM" (``sources.js`` ``fmtSyncTime``) and the settled
#: SUCCESS result line (``fmtSyncResult``) — updated/pruned are 0 here.
#: SUCCESS result line (``fmtSyncResult``) — updated/pruned are 0 here,
#: and the phase-187 peak clause is optional (null pair → no clause).
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}")
RESULT_LINE = re.compile(r"^(\d+) added(?: · (\d+) unchanged)?$")
RESULT_LINE = re.compile(rf"^(\d+) added(?: · (\d+) unchanged)?{PEAK_CLAUSE}$")
#: The console noise the anonymous probes legitimately produce (this
#: module's own tests are admin — the collector stays as the uncaught-JS
@@ -616,7 +618,7 @@ def test_status_polls_responsive_during_long_clone(
# And it stayed in its live state: no error banner, no modal, the
# neutral running copy, the Cancel control still there.
expect(page.locator("#sync-progress-text")).to_have_text("Syncing…")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("Syncing…"))
expect(btn).to_be_disabled()
expect(cancel).to_be_visible()
expect(page.locator("#sync-error-banner")).to_be_hidden()
@@ -784,7 +786,7 @@ def test_cancel_during_clone_reaches_terminal(
# The neutral UI settle: "Sync cancelled", retry-ready, no error.
expect(page.locator("#sync-progress-text")).to_have_text(
"Sync cancelled", timeout=UI_SETTLE_TIMEOUT_MS
counts_line("Sync cancelled"), timeout=UI_SETTLE_TIMEOUT_MS
)
expect(page.locator("#sync-label")).to_have_text(
"Sync sources", timeout=UI_SETTLE_TIMEOUT_MS
+7 -4
View File
@@ -141,6 +141,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -710,13 +711,15 @@ def test_label_advances_through_the_image_span(
if snap["box"] != last:
seen.append(snap["box"])
last = snap["box"]
if snap["box"] == SETTLE_TEXT:
if counts_line(SETTLE_TEXT).fullmatch(snap["box"]):
break
time.sleep(0.2)
# --- the settle (the UI's 2 s poll renders the terminal at most one
# tick after it lands; the endpoint is the source of truth) ---
assert last == SETTLE_TEXT, f"the run never settled to the counts line: {seen[-5:]}"
assert counts_line(SETTLE_TEXT).fullmatch(str(last)), (
f"the run never settled to the counts line: {seen[-5:]}"
)
terminal = _wait_terminal(app_url)
# The counts line's ground truth: the PDF + its 4 embedded-image
# docs — one file, 5 added.
@@ -806,7 +809,7 @@ def test_label_advances_through_the_image_span(
# surface): the image part is not in it, not in the title, not in
# the aria (the settle restores the ship aria value).
box = page.locator("#sync-progress-text").inner_text()
assert box == SETTLE_TEXT, box
assert counts_line(SETTLE_TEXT).fullmatch(box), box
assert "· image" not in box and "· image" not in (btn.get_attribute("title") or "")
assert btn.get_attribute("aria-label") == "Sync sources", btn.get_attribute("aria-label")
expect(page.locator("#sync-error-banner")).to_be_hidden()
@@ -869,7 +872,7 @@ def test_mid_run_reload_re_attaches_with_image_position(
# terminal clears the position, test 1 pins the endpoint half),
# the terminal label, the parts cleared + empty.
expect(page.locator("#sync-progress-text")).to_have_text(
SETTLE_TEXT, timeout=SETTLE_TIMEOUT_MS
counts_line(SETTLE_TEXT), timeout=SETTLE_TIMEOUT_MS
)
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=30_000)
btn = page.locator("#sync-btn")
+2 -1
View File
@@ -157,6 +157,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -688,7 +689,7 @@ def _settle(page: Page, app_url: str) -> None:
# retired announcer spoke, byte-identical: the fresh 4-file
# fixture's "4 added"), hiding only when its text is empty.
expect(page.locator("#sync-progress")).to_be_visible()
expect(page.locator("#sync-progress-text")).to_have_text(f"{N_FILES} added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(f"{N_FILES} added"))
expect(page.locator("#sync-error-banner")).to_be_hidden()
+3 -2
View File
@@ -69,6 +69,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -454,7 +455,7 @@ def test_healthy_sync_still_succeeds(
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled() # never stale — re-enabled at the terminal state
# The counts line rides the box (phase 150 — the sole sync surface).
expect(page.locator("#sync-progress-text")).to_have_text("1 added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("1 added"))
# The REAL clone was imported: the fixture path is in the Sources
# table (the sentinel lives inside it) — via the drill-down tree
@@ -475,7 +476,7 @@ def test_healthy_sync_still_succeeds(
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled()
expect(page.locator("#sync-progress-text")).to_have_text("0 added · 1 unchanged")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line("0 added · 1 unchanged"))
# The doc survived the prune re-import — via the drill-down tree
# (phase 97).
_expect_fixture_row(page)
+844
View File
@@ -0,0 +1,844 @@
"""Phase 187 (task 04) E2E (Playwright, mock-only): the sync's PEAK MEMORY
is visible on the page that started the run — and it is the truth.
Source: owner incident 2026-10-15. Reconstructing "how much RAM did that
sync use?" meant reading ``systemd`` unit-teardown lines from three boots
(``brain-of-code-app.service: … 3.3G memory peak``), and boot −1 — the run
that killed the box — had NO teardown record at all, because the VM never
shut down cleanly. The app had no opinion about its own memory: no log
line, no status field, nothing on the Sources page. Phase 187 made it a
number (``app/core/memory.py``, the lifecycle lines, the status keys) and
this module is the browser proof of the last mile.
What the phase put on the wire, in the operator's words:
* ``GET /api/sync/status`` carries the ADDITIVE pair ``rss_mb`` (the
process's CURRENT resident set, MB) and ``rss_peak_mb`` (its HIGH-WATER
mark) — ``int | null``, where ``null`` means "no reading" and NEVER 0;
sampled at the run start, at every ``phase`` transition, throttled to
one read per 5 s through the import's per-file hook, and FORCED in every
terminal, which KEEPS the final pair next to the final counts;
* the Sources page appends `` · peak <N> MB`` to the result line it
already spoke (the ``fmtSyncResult`` counts line, "Sync cancelled") —
trailing text on a line that already exists, never a new control, never
a new live region, and NOTHING rendered when the pair is null.
**Which number this is (read it before comparing it to a dashboard).**
Both fields are PROCESS readings — ``VmRSS`` from ``/proc/self/status`` and
``getrusage(RUSAGE_SELF).ru_maxrss`` — NOT the host's cgroup figure. A
cgroup counts PAGE CACHE too: the incident box's app cgroup showed ~2.16 GB
of ``file`` against ~211 MB of ``anon`` while idle, and ``systemd`` printed
``3.3G memory peak`` for the unit while the uvicorn process was a fraction
of that. Neither number is wrong; LOCKED A1 pins the app to the process
one, and `README.md` (*Operational notes: the sync's memory (phase 187)*)
owns the full caveat and the operator grep
(``grep ' :: ' server.log | grep ' rss_peak_mb='``). This module never
asserts a MAGNITUDE — only that the number is present, typed, terminal,
honest (``peak >= current``, with the one exception the sampler owns: a
kernel high-water refreshed lazily is lifted to the current reading, never
invented from it), and equal on both surfaces.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the deterministic
mock plus the phase-64 ``tests/e2e/slow_llm.py`` delay proxy is what makes
the RUNNING observation a fact instead of a race: the run must outlive the
UI's 2 s status poll (and the tests' ~100 ms one) so the "no clause while
running" pin and the cancel window are both deterministic. Two files →
≈ 10 LLM calls (the two model probes, the per-file embed batches + the
summaries, the KB overview, the source-root folder summary) ≈ 5 s at
``SLOW_DELAY_S = 0.5``.
Isolation (the house shape): ONE module app on its own port (the conftest
session app owns 8123 in a combined run), the mock LLM on the conftest
port behind the proxy, a dedicated ``BOR_SOURCES_DIR`` /
``BOR_PDF_PAGE_DIR`` so no run writes into the operator's, the shared E2E
Postgres truncated BEFORE **and** AFTER every test (after waiting for the
app to hold no running sync), and the admin in through the real form login
(``tests/e2e/auth_helpers.py``). Each test seeds the same two-file
``kind=local`` source directory and registers it through the authenticated
API, so any single test re-runs alone with a fresh KB and the counts line
says ``2 added``.
Test → observable mapping (Playwright Mapping Rule — the rendered DOM is
the observable; the DOM number and the payload number are read as one
comparison, so "the page lies" cannot slip through):
1. ``test_result_line_shows_the_peak`` — a full run from the Sources page:
the box (``#sync-progress-text``) settles the counts line WITH the
trailing clause (``^2 added · peak \\d+ MB$``), the clause matches
``/peak \\d+ MB/``, the aria-live region IS the box (``role=status`` +
``aria-live=polite``, no ``aria-hidden``) and carries the SAME text,
the clause is TEXT and not an element (the box still holds exactly one
child span — no new control), and the megabytes in the DOM EQUAL the
terminal payload's ``rss_peak_mb``. Where the platform cannot know the
pair, the mirror contract is pinned instead: NO clause at all.
2. ``test_status_payload_reports_rss`` — the endpoint end to end: a tick
during ``running`` (a) reports the pair with the documented types
(``int | null``), (b) carries the pre-phase key set UNCHANGED (the
additive pin: exactly the pre-187 keys plus the memory pair) and (c)
shows NO ``peak`` clause on the page — the clause is terminal-only;
after the terminal ``rss_peak_mb`` is a POSITIVE int with
``peak >= current`` (the invariant the ONE sampler owns: the kernel
refreshes its high-water lazily, so ``_sample_rss`` lifts a lagging peak
to the current reading — a peak under the run's own footprint would be a
false number), and the same pair survives a later poll (the page
reloaded an hour later still answers).
3. ``test_anonymous_cannot_read_memory`` — the A4 exposure guard: a
logged-out ``GET /api/sync/status`` is 403 with the pre-phase body
``{"detail": "admin only"}`` — no memory number on an anonymous
surface, before or after a run — and the page shows the sign-in gate
with no sync button and no peak clause anywhere in the DOM.
4. ``test_cancelled_terminal_keeps_the_peak`` — the ``test_sync_cancel.py``
shape: cancel mid-run and the NEUTRAL "Sync cancelled" state still
renders (no banner, no modal, the button retry-ready) with the clause
present and equal to the cancelled payload's ``rss_peak_mb`` — the
run that stopped early is exactly the run whose memory matters.
5. ``test_peak_clause_fits_at_375`` — PLAN §7 at phone width: after a run
settled at 375 px, the clause is on screen, the box does not clip it
(``scrollWidth <= clientWidth`` for the box and its text), the box
stays inside the viewport and the PAGE has no horizontal overflow.
"""
from __future__ import annotations
import os
import re
import subprocess
import sys
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import httpx
import pytest
from playwright.sync_api import Browser, ConsoleMessage, Page, ViewportSize, expect
from sqlalchemy import text
from app.config import Settings as _Settings
from app.core.memory import rss_mb, rss_peak_mb
from app.db import SessionLocal
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
MOCK_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
#: Dedicated module ports (the house env-overridable pattern): the conftest
#: session app owns 8123 and the mock owns 8901 in a combined run.
APP_PORT = int(os.environ.get("E2E_APP_PORT_PEAKMEM", "8204"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
#: The slow-LLM proxy's port (the conftest's mock stays on MOCK_PORT — the
#: proxy fronts it).
SLOW_PORT = int(os.environ.get("E2E_SLOW_LLM_PORT_PEAKMEM", "8910"))
SLOW_URL = f"http://127.0.0.1:{SLOW_PORT}"
#: Per-LLM-request delay (the phase-64 timing convention): the run spans
#: ≈ 10 calls ≈ 5 s, i.e. ≫ the UI's 2 s status poll — the running window
#: and the cancel window are both observable, not races.
SLOW_DELAY_S = "0.5"
SOURCES_URL = "/sources.html"
#: The fixture directory's basename — the ``source`` a ``kind=local`` row
#: records (``Path(path).name``), stable whatever temp root pytest picked.
#:
#: **Nothing here is allowed to contain the word "peak"** — not the source
#: name, not a path, no marker. Several pins below ask whether the page
#: says ANYTHING about memory (the running label must be clause-free, the
#: anonymous shell must be silent), and those asks read the rendered text;
#: a fixture path like ``peakmem/notes/peak-alpha.md`` rides the running
#: label verbatim and would answer "yes" from the file name alone. So the
#: fixture is ``memline-fixture/notes/mem-*.md`` and the clause stays the
#: only thing on the page that can say the word.
SOURCE_NAME = "memline-fixture"
N_FILES = 2
FIXTURE_FILES = {
"notes/mem-alpha.md": (
"# Memory line alpha\n"
"\n"
"Head marker RESE-MEMLINE-ALPHA-6d13 opens this note.\n"
"\n"
"## Alpha body\n"
"\n"
"A small note that exists so the phase-187 suite has a real run "
"to watch: a real walk, real embeds, a real KB overview, and the "
"run's resident-set clause on the line it settles.\n"
),
"notes/mem-beta.md": (
"# Memory line beta\n"
"\n"
"Head marker RESE-MEMLINE-BETA-9a74 opens this note — the second "
"file, the counts line's denominator.\n"
),
}
#: The clause sources.js appends (``fmtSyncPeakClause``): trailing text on
#: a result line that already exists. REQUIRED here — this suite's whole
#: point is that the number is there (the optional form, for the suites
#: that only pin their own counts, lives in ``e2e.sync_helpers``).
PEAK_CLAUSE = re.compile(r" · peak (\d+) MB$")
#: The clause SHAPE, for the "this surface says nothing about memory" pins
#: (the running label, the anonymous shell). Matched case-insensitively on
#: the word, because the clause is the only thing on these pages allowed to
#: use it — which the fixture naming rule above is what keeps true.
_ANY_PEAK = re.compile(r"peak", re.IGNORECASE)
#: The settled success line: the pre-phase ``fmtSyncResult`` counts line
#: (a fresh KB of two files → "2 added", the zero "updated"/"unchanged"
#: parts omitted) with the phase-187 clause trailing it.
SUCCESS_LINE = re.compile(rf"^{N_FILES} added{PEAK_CLAUSE.pattern}$")
#: The neutral cancel line (phase 134/150 wording, byte-identical) with
#: the clause trailing it.
CANCELLED_LINE = re.compile(rf"^Sync cancelled{PEAK_CLAUSE.pattern}$")
#: "Synced HH:MM" — the local-time last-result label (sources.js's
#: fmtSyncTime), any hour/minute (the test_sync_button.py pattern).
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}$")
#: The status payload's key set BEFORE phase 187 — the additive pin's
#: left-hand side. The payload must stay exactly this set plus the memory
#: pair, forever (the same list the integration suite pins).
PRE_187_STATUS_KEYS = frozenset(
{
"state",
"started_at",
"finished_at",
"detail",
"error",
"current_file",
"files_done",
"files_total",
"current_chunk",
"chunks_total",
"images_done",
"images_total",
"phase",
"current_summary",
"summaries_done",
"summaries_total",
}
)
MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
#: The pre-phase integer count slots (their types are part of the
#: unchanged-shape pin in test 2 — never a float, never null).
_COUNT_KEYS = frozenset(
{
"files_done",
"files_total",
"current_chunk",
"chunks_total",
"images_done",
"images_total",
"summaries_done",
"summaries_total",
}
)
#: The invariant the page can rely on: ``peak >= current``. The raw kernel
#: counters do NOT give that for free (``VmRSS`` and ``ru_maxrss`` are two
#: different sources, both floored to whole MB, and the kernel refreshes the
#: high-water lazily — measured at ``rss_mb=164`` against
#: ``rss_peak_mb=162``), so the ONE sampling point
#: (:func:`app.api.sync._sample_rss`) lifts a lagging peak to the current
#: reading. Slack ZERO: a peak BELOW the run's own current footprint is a
#: false number, and the page must never show one.
MB_SLACK = 0
#: Does THIS platform answer the two helpers? Linux (the deploy target:
#: ``/proc`` + ``resource``) always does. The app under test is a
#: subprocess of this process's own OS, so this answer is the app's answer.
#: Where it is False the value pins step aside and the ABSENCE contract is
#: pinned instead (null renders nothing — never "peak ? MB", never 0).
MEMORY_KNOWABLE = rss_mb() is not None and rss_peak_mb() is not None
#: The house phone viewport at which the clause must still fit (the
#: PLAN §7 / AGENTS rule 5 width: no horizontal overflow at 375 px).
MOBILE_VIEWPORT: ViewportSize = {"width": 375, "height": 720}
#: Settle budgets: a two-file run under the proxy is ≈ 5 s and the UI's 2 s
#: poll settles at most one tick after the terminal — 90 s is an order of
#: magnitude of head-room (the house sync budget, no client timeout).
SETTLE_TIMEOUT_MS = 90_000
#: The browser logs "Failed to load resource: … 403" for the anonymous
#: probes (the whoami gate + the admin-only status fetch) — expected
#: noise, not a defect.
_CONSOLE_NOISE = re.compile(r"Failed to load resource", re.IGNORECASE)
# ---------------------------------------------------------------------------
# The fixture source
# ---------------------------------------------------------------------------
def _seed_source(root: Path) -> Path:
"""(Re-)write the two fixture files under ``root / SOURCE_NAME``.
Idempotent on purpose: every test calls it, so the tree is at its
ORIGINAL bytes whenever a test starts and a lone re-run of any single
test still sees a fresh KB with ``2 added``.
"""
base = root / SOURCE_NAME
for rel, body in FIXTURE_FILES.items():
target = base / rel
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(body, encoding="utf-8")
assert not (base / ".git").exists() # a plain directory — no git anywhere
return base
@pytest.fixture(scope="module")
def source_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The fixture tree the module app walks (the app server runs on the
same host, so the path is visible to it)."""
return _seed_source(tmp_path_factory.mktemp("bor_peak_memory"))
# ---------------------------------------------------------------------------
# The module app + the timing proxy
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def slow_llm(mock_llm: int) -> Iterator[int]:
"""The delay-injecting reverse proxy in front of the mock LLM
(``tests/e2e/slow_llm.py``) — the ONLY thing that makes the RUNNING
observation and the cancel window deterministic (module docstring)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["SLOW_LLM_DELAY_S"] = SLOW_DELAY_S
env["E2E_MOCK_PORT"] = str(MOCK_PORT)
proc = subprocess.Popen(
[
sys.executable, "-m", "uvicorn", "tests.e2e.slow_llm:app",
"--host", "127.0.0.1", "--port", str(SLOW_PORT), "--log-level", "warning",
],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{SLOW_URL}/v1/models")
yield SLOW_PORT
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_server(
mock_llm: int,
slow_llm: int,
source_dir: Path,
tmp_path_factory: pytest.TempPathFactory,
) -> Iterator[str]:
"""The real app under test — per-module env (the conftest pattern):
``BOR_LLM_BASE_URL`` is the SLOW proxy in front of the deterministic
mock, ``BOR_GIT_SOURCES`` forced empty (each test registers its own
local row; an operator's gitignored ``.env`` must never leak a real
source into a run), and a dedicated ``BOR_SOURCES_DIR``.
"""
if USE_REAL_LLM:
pytest.skip(
"mock-only suite — the running window IS the slow proxy in "
"front of the deterministic mock (E2E_REAL_LLM unsupported)"
)
assert mock_llm and source_dir
scratch = tmp_path_factory.mktemp("bor_peak_memory_app")
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = f"{SLOW_URL}/v1"
# Mock-calibrated gate (the conftest pattern): both floors must stay
# <= the threshold or the app refuses to boot (app/config.py).
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
# Instant retry waits + the code-default budget (the conftest leak
# guard — this suite never needs a retry, and never waits 5 s for one).
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
env["BOR_GIT_SOURCES"] = ""
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
env["BOR_PDF_PAGE_DIR"] = str(scratch / "pdf-pages")
proc = subprocess.Popen(
[
sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning",
],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
# ---------------------------------------------------------------------------
# Isolation
# ---------------------------------------------------------------------------
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern, the phase-184 list:
``sources_meta`` and ``folder_summaries`` reset with the KB)."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, kb_overview, "
"git_sources, sources_meta, folder_summaries"
)
)
db.commit() # without the commit the TRUNCATE rolls back
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
"""No run may leak across a truncate: wait for a non-running status with
an own admin session (the endpoint is admin-only). Usually a no-op —
every test settles before it ends."""
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
with httpx.Client(base_url=base_url, timeout=5.0) as client:
client.post("/api/login", json={"password": ADMIN_PASSWORD})
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
if r.json()["state"] != "running":
return
time.sleep(0.2)
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
@pytest.fixture(autouse=True)
def _clean(app_url: str, source_dir: Path, db_ready: None) -> Iterator[None]:
"""Per-test KB isolation: the shared E2E Postgres is truncated before
AND after every test, the app holding no running sync at both
boundaries. The fixture tree is re-seeded first (idempotent) and the
local row is registered through the authenticated API, so every test
owns a fresh two-file source and its own ``2 added``."""
_wait_sync_settled(app_url)
_truncate_all()
_seed_source(source_dir.parent)
with httpx.Client(base_url=app_url, timeout=30.0) as client:
assert client.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
r = client.post(
"/api/git-sources", json={"kind": "local", "path": str(source_dir)}
)
assert r.status_code == 201, r.text
yield
_wait_sync_settled(app_url)
_truncate_all()
@pytest.fixture()
def page(browser: Browser) -> Iterator[Page]:
"""The conftest's page + the console-error collector: an uncaught JS
exception (pageerror) or a console error that is not the browser's
expected 403 "Failed to load resource" noise fails the owning test at
teardown — a clause builder that threw inside ``setSyncProgress`` would
otherwise settle a blank box and read as a passing test.
The conftest's ``page`` is NOT reused here because this suite's whole
subject is the sync result line: a JS error inside the clause builder
must fail the owning test, not leak (the ``test_sync_cancel.py``
pattern)."""
pg = browser.new_page(viewport={"width": 1280, "height": 800})
errors: list[str] = []
def _on_console(msg: ConsoleMessage) -> None:
if msg.type == "error" and not _CONSOLE_NOISE.search(msg.text):
errors.append(msg.text)
pg.on("console", _on_console)
pg.on("pageerror", lambda err: errors.append(str(err)))
try:
yield pg
finally:
pg.close()
assert not errors, f"console/page errors: {errors}"
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _status(page: Page, base_url: str) -> dict[str, Any]:
"""The status payload through the ADMIN browser context (the cookie
rides ``page.request`` — the same surface the page's own poll uses)."""
r = page.request.get(f"{base_url}/api/sync/status")
assert r.status == 200, r.text
body: dict[str, Any] = r.json()
return body
def _wait_sync_state(
page: Page, base_url: str, states: set[str], timeout_s: float = 90.0
) -> dict[str, Any]:
"""Poll the status endpoint at ~100 ms until the run's state is in
*states* (the deterministic layer: the UI's own poll ticks at 2 s)."""
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
body = _status(page, base_url)
if body["state"] in states:
return body
time.sleep(0.1)
raise AssertionError(f"sync did not reach {sorted(states)}: {body}")
def _open_sources(page: Page, base_url: str) -> None:
"""The real form login, landing on the Sources view (the sync button's
home) with the module graph booted."""
login(page, base_url) # lands on /sources.html
btn = page.locator("#sync-btn")
expect(btn).to_be_visible(timeout=30_000)
expect(btn).to_be_enabled()
def _click_sync(page: Page) -> None:
"""Click **Sync sources** and pin the 202 moment: the button disables
(the run is in flight), the box reveals with the bare running label —
which carries NO peak clause (the clause is terminal-only)."""
btn = page.locator("#sync-btn")
btn.click()
expect(btn).to_be_disabled()
expect(page.locator("#sync-progress")).to_have_text("Syncing…")
def _start_from_page(page: Page, base_url: str) -> None:
"""Login on the Sources view, then the click (the two-step form of the
same helper, so a test can change the viewport BETWEEN them)."""
_open_sources(page, base_url)
_click_sync(page)
def _clause_number(text: str) -> int:
"""The MB value the clause carries (loudly absent when there is none)."""
m = PEAK_CLAUSE.search(text)
assert m, f"no peak clause in {text!r}"
return int(m.group(1))
# ---------------------------------------------------------------------------
# 1. The success terminal: the clause on the line that already existed
# ---------------------------------------------------------------------------
def test_result_line_shows_the_peak(page: Page, app_url: str, db_ready: None) -> None:
"""A real sync started from the Sources page ends with the number on
the line the page already spoke — ``2 added · peak <N> MB`` — inside
the sole aria-live region, as TEXT (no new control), and equal to the
terminal payload's ``rss_peak_mb`` (the page and the log can never
disagree, because they read the one forced terminal sample)."""
page.set_default_timeout(30_000)
_start_from_page(page, app_url)
terminal = _wait_sync_state(page, app_url, {"success", "failed", "cancelled"})
assert terminal["state"] == "success", terminal
assert terminal["error"] is None, terminal
# The UI settle (the page's own 2 s poll, one tick after the terminal).
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SETTLE_TIMEOUT_MS)
box = page.locator("#sync-progress")
expect(box).to_be_visible()
text_el = page.locator("#sync-progress-text")
line = text_el.inner_text()
if not MEMORY_KNOWABLE:
# The mirror contract: no reading → NO clause at all (never
# "peak ? MB", never "peak 0 MB"), and the counts line alone.
assert line == f"{N_FILES} added", line
assert not _ANY_PEAK.search(box.inner_text()), box.inner_text()
return
# The clause is there AND the pre-phase wording is intact (the counts
# line keeps its exact bytes; the clause trails it).
assert SUCCESS_LINE.fullmatch(line), line
assert PEAK_CLAUSE.search(line), line
assert line.startswith(f"{N_FILES} added"), line
# The clause is TEXT on a line that already exists — the box still
# holds its ONE child span (no new control, no new live region).
expect(box.locator("> *")).to_have_count(1)
# The box IS the sole aria-live region and it announces the clause
# (phase 150: one live region, one surface — and the clause rides it).
expect(box).to_have_attribute("role", "status")
expect(box).to_have_attribute("aria-live", "polite")
expect(box).not_to_have_attribute("aria-hidden")
assert PEAK_CLAUSE.search(box.inner_text()), box.inner_text()
assert box.inner_text().strip() == line.strip(), (box.inner_text(), line)
# HONESTY: the megabytes on the page ARE the terminal payload's
# ``rss_peak_mb`` — one forced reading, two surfaces.
assert terminal["rss_peak_mb"] is not None, terminal
assert _clause_number(line) == terminal["rss_peak_mb"], (line, terminal)
# And the number survives the page: a reload re-attaches the terminal
# (initSyncButton) and re-renders the SAME clause.
page.reload()
expect(page.locator("#sync-progress-text")).to_have_text(
SUCCESS_LINE, timeout=SETTLE_TIMEOUT_MS
)
assert _clause_number(page.locator("#sync-progress-text").inner_text()) == (
terminal["rss_peak_mb"]
)
# No error surface on a success settle (unchanged contract).
expect(page.locator("#sync-error-banner")).to_be_hidden()
assert page.locator(".sync-modal-backdrop").count() == 0
# ---------------------------------------------------------------------------
# 2. The payload: additive, typed, terminal-kept — and silent while running
# ---------------------------------------------------------------------------
def test_status_payload_reports_rss(page: Page, app_url: str, db_ready: None) -> None:
"""The additive pin end to end, through the browser's own authenticated
request surface: while the run is in flight the payload already carries
the pair (``int | null``) and the pre-phase keys are byte-for-byte
unchanged, while the PAGE shows no clause at all; after the terminal the
peak is a positive integer that stays with the run, and the pair keeps
the invariant the page relies on (``peak >= current`` — the sampling
point lifts the lazily-refreshed high-water, ``MB_SLACK`` is 0).
"""
page.set_default_timeout(30_000)
_start_from_page(page, app_url)
# --- during `running` ------------------------------------------------
running = _wait_sync_state(page, app_url, {"running"})
assert running["state"] == "running", running
# The additive contract on the wire: exactly the pre-phase keys plus
# the memory pair — nothing renamed, nothing dropped.
assert set(running) == PRE_187_STATUS_KEYS | MEMORY_KEYS, set(running)
for key, value in running.items():
if key in MEMORY_KEYS:
# The documented type: int | null (null is "no reading").
assert value is None or isinstance(value, int), (key, value)
elif key == "detail":
assert isinstance(value, dict), (key, value)
elif key in _COUNT_KEYS:
assert isinstance(value, int), (key, value)
else:
assert value is None or isinstance(value, (str, int, bool)), (key, value)
# The types are ints on a platform that knows them (the run start
# FORCES the first sample, so a running tick is never null here).
if MEMORY_KNOWABLE:
assert running["rss_mb"] is not None, running
assert running["rss_peak_mb"] is not None, running
assert running["rss_peak_mb"] >= running["rss_mb"] - MB_SLACK, running
# The RUNNING state shows NO clause: the box carries the live label and
# says nothing about memory (the clause is terminal-only in the UI). The
# word is the tell (the fixture names never carry it), so a clause that
# leaked into a running label fails even when its number is null-ish.
live_text = page.locator("#sync-progress-text").inner_text()
assert live_text.startswith("Syncing…"), live_text
assert not _ANY_PEAK.search(live_text), live_text
assert not _ANY_PEAK.search(page.locator("#sync-progress").inner_text())
# --- after the terminal ---------------------------------------------
terminal = _wait_sync_state(page, app_url, {"success", "failed", "cancelled"})
assert terminal["state"] == "success", terminal
assert set(terminal) == PRE_187_STATUS_KEYS | MEMORY_KEYS, set(terminal)
current, peak = terminal["rss_mb"], terminal["rss_peak_mb"]
if MEMORY_KNOWABLE:
assert peak is not None and isinstance(peak, int), terminal
assert peak > 0, terminal # a real reading, never a placeholder 0
assert current is not None and current > 0, terminal
assert peak >= current - MB_SLACK, (current, peak)
else:
assert peak is None and current is None, terminal
# The pair SURVIVES the run (no reset between the terminal and a later
# poll — the page can be reloaded an hour later and still answers).
again = _status(page, app_url)
assert (again["rss_mb"], again["rss_peak_mb"]) == (current, peak), again
# …and the pre-existing values rode the terminal unchanged too.
assert again["state"] == "success"
assert again["error"] is None
assert isinstance(again["detail"], dict) and again["detail"]["added"] == N_FILES
# ---------------------------------------------------------------------------
# 3. Anonymous: no memory number on any anonymous surface (A4)
# ---------------------------------------------------------------------------
def test_anonymous_cannot_read_memory(
page: Page, browser: Browser, app_url: str, db_ready: None
) -> None:
"""The exposure guard is the status route's own admin gate: a logged-out
visitor gets the pre-phase 403 with the pre-phase body, before or after
a run — and the Sources shell shows the sign-in gate with no sync button
and no memory clause anywhere in the DOM.
The probe runs in a SEPARATE browser context (``browser.new_context()``)
— this test's own page is logged in by the run above, and
``page.context.new_page()`` would SHARE that session's cookies, which
would make the "anonymous" probe an admin one."""
# A run FIRST, so the guard is proven with real numbers in existence.
_start_from_page(page, app_url)
terminal = _wait_sync_state(page, app_url, {"success"})
if MEMORY_KNOWABLE:
assert terminal["rss_peak_mb"] is not None, terminal
ctx = browser.new_context(viewport={"width": 1280, "height": 800})
try:
anon = ctx.new_page()
r = anon.request.get(f"{app_url}/api/sync/status")
assert r.status == 403, r.text
assert r.json() == {"detail": "admin only"}
# The anonymous shell: the gate is the anonymity proof, the sync
# controls are never revealed, and nothing about memory renders.
anon.goto(f"{app_url}/sources.html")
expect(anon.locator("#sources-gate")).to_be_visible(timeout=30_000)
expect(anon.locator("#sync-btn")).to_be_hidden()
expect(anon.locator("#sync-progress")).to_be_hidden()
assert not _ANY_PEAK.search(anon.locator("body").inner_text())
finally:
ctx.close()
# ---------------------------------------------------------------------------
# 4. The cancelled terminal keeps the peak (the test_sync_cancel shape)
# ---------------------------------------------------------------------------
def test_cancelled_terminal_keeps_the_peak(
page: Page, app_url: str, db_ready: None
) -> None:
"""Cancel mid-run: the run lands in the NEUTRAL ``cancelled`` terminal
(no banner, no modal, retry-ready) and the box still carries the
clause — "Sync cancelled · peak <N> MB" — equal to the cancelled
payload's ``rss_peak_mb``. A cancelled run's memory is still data: the
phase-64 keep-final-values convention covers it like the counts."""
page.set_default_timeout(30_000)
_start_from_page(page, app_url)
_wait_sync_state(page, app_url, {"running"})
cancel = page.locator("#sync-cancel-btn")
expect(cancel).to_be_visible()
expect(cancel).to_be_enabled()
cancel.click()
expect(cancel).to_be_disabled() # never stale, one request at a time
body = _wait_sync_state(page, app_url, {"cancelled"}, timeout_s=30.0)
assert body["state"] == "cancelled"
assert body["error"] is None # a cancel is intent, not failure
assert body["detail"] == {}
# The neutral settle, clause and all (the 2 s poll, one tick later).
expect(page.locator("#sync-label")).to_have_text("Sync sources", timeout=30_000)
expect(page.locator("#sync-btn")).to_be_enabled()
expect(cancel).to_be_hidden()
expect(page.locator("#sync-error-banner")).to_be_hidden()
assert page.locator(".sync-modal-backdrop").count() == 0
line = page.locator("#sync-progress-text").inner_text()
if MEMORY_KNOWABLE:
assert body["rss_peak_mb"] is not None, body
assert body["rss_peak_mb"] > 0, body
assert CANCELLED_LINE.fullmatch(line), line
assert _clause_number(line) == body["rss_peak_mb"], (line, body)
else:
assert line == "Sync cancelled", line
# The pair survives the cancel exactly like the counts do.
again = _status(page, app_url)
assert (again["rss_mb"], again["rss_peak_mb"]) == (
body["rss_mb"],
body["rss_peak_mb"],
), again
# ---------------------------------------------------------------------------
# 5. PLAN §7 at 375 px: the clause fits, the page never scrolls sideways
# ---------------------------------------------------------------------------
def test_peak_clause_fits_at_375(page: Page, app_url: str, db_ready: None) -> None:
"""The clause at phone width: the result line wraps INSIDE the bordered
box (``overflow-wrap: anywhere``) — nothing clipped, no horizontal page
overflow, the box inside the viewport (AGENTS rule 5 / PLAN §7). The
clause is a few characters of trailing text, so a layout shift or a
clipped tail at 375 px is exactly what this pins out."""
_open_sources(page, app_url)
page.set_viewport_size(MOBILE_VIEWPORT) # the phone width, then the click
page.wait_for_timeout(150) # the reflow settles before the run paints
_click_sync(page)
terminal = _wait_sync_state(page, app_url, {"success"})
text_el = page.locator("#sync-progress-text")
expect(text_el).to_have_text(
SUCCESS_LINE if MEMORY_KNOWABLE else f"{N_FILES} added",
timeout=SETTLE_TIMEOUT_MS,
)
line = text_el.inner_text()
if MEMORY_KNOWABLE:
assert PEAK_CLAUSE.search(line), line
assert _clause_number(line) == terminal["rss_peak_mb"], (line, terminal)
# The box + its text do not clip, the box stays in the viewport, and
# the PAGE itself never scrolls sideways at 375 px.
page.wait_for_timeout(150) # the reflow settles before the measure
geometry = page.evaluate(
"""() => {
const box = document.querySelector('#sync-progress');
const textEl = document.querySelector('#sync-progress-text');
const b = box.getBoundingClientRect();
return {
boxScrollWidth: box.scrollWidth,
boxClientWidth: box.clientWidth,
textScrollWidth: textEl.scrollWidth,
textClientWidth: textEl.clientWidth,
left: b.left,
right: b.right,
height: b.height,
docScroll: document.documentElement.scrollWidth,
innerWidth: window.innerWidth,
text: textEl.textContent,
};
}"""
)
assert geometry["height"] > 0, geometry
assert geometry["boxScrollWidth"] <= geometry["boxClientWidth"], geometry
assert geometry["textScrollWidth"] <= geometry["textClientWidth"], geometry
assert geometry["left"] >= 0, geometry
assert geometry["right"] <= MOBILE_VIEWPORT["width"] + 1, geometry
assert geometry["docScroll"] <= geometry["innerWidth"] + 1, geometry
assert PEAK_CLAUSE.search(geometry["text"]) or not MEMORY_KNOWABLE, geometry
+9 -4
View File
@@ -183,6 +183,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import PEAK_CLAUSE, counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -267,8 +268,12 @@ WALL_BUDGET_S = 600.0
#: The settle's counts line (sources.js's fmtSyncResult — the first
#: part is always "N added"; the rest are conditional, " · "-joined —
#: the shape the pre-phase announcer spoke, byte-identical).
RESULT_LINE = re.compile(r"^\d+ added( · \d+ (updated|unchanged|pruned))*$")
#: the shape the pre-phase announcer spoke, byte-identical) plus the
#: optional phase-187 peak clause — the counts are still anchored, so
#: a mutated counts line still fails this shape.
RESULT_LINE = re.compile(
rf"^\d+ added( · \d+ (updated|unchanged|pruned))*{PEAK_CLAUSE}$"
)
#: A broken git source — the failure trigger (test 8): a ``file://``
#: URL with no repository behind it (the URL shape passes add-time
@@ -1146,7 +1151,7 @@ def test_settled_state_restores_the_pill(
# counts line (the fmtSyncResult shape the pre-phase announcer
# spoke, byte-identical).
expect(page.locator("#sync-progress")).to_be_visible()
expect(page.locator("#sync-progress-text")).to_have_text(f"{N_FILES} added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(f"{N_FILES} added"))
# The label parts are restored — the settle label carries no count
# or chunk part.
count = page.locator(".sync-label-count")
@@ -1243,7 +1248,7 @@ def test_settled_cancel_shows_the_result_line(
# line (byte-identical to the pre-phase announcer's string)…
expect(page.locator("#sync-progress")).to_be_visible(timeout=30_000)
expect(page.locator("#sync-progress-text")).to_have_text(
"Sync cancelled", timeout=30_000
counts_line("Sync cancelled"), timeout=30_000
)
# …the button settles retry-ready…
expect(page.locator("#sync-label")).to_have_text("Sync sources", timeout=30_000)
+10 -3
View File
@@ -130,6 +130,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line, strip_peak_clause
REPO = Path(__file__).resolve().parents[2]
@@ -497,12 +498,18 @@ def _sync_from_sources_page(page: Page, base_url: str) -> dict[str, Any]:
def _expect_counts_line(page: Page, expected: str) -> None:
"""The counts line — the owner-visible sync report on the single sync text
surface (phase 150; sources.js fmtSyncResult)."""
expect(page.locator("#sync-progress-text")).to_have_text(expected, timeout=15_000)
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(expected), timeout=15_000)
def _counts_line(page: Page) -> str:
"""The counts line as text — the string test 4 compares across the apps."""
return str(page.locator("#sync-progress-text").inner_text())
"""The counts line as text — the string test 4 compares across the apps.
Phase 187 (task 03): the peak clause is stripped first. The byte-
identity this test pins is the KB's (the same ten files report the same
counts under both cadences); the megabytes belong to the process that
ran them and differ between the two module apps by design.
"""
return strip_peak_clause(str(page.locator("#sync-progress-text").inner_text()))
def _open_level(page: Page, *names: str) -> None:
+2 -1
View File
@@ -98,6 +98,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -497,7 +498,7 @@ def test_fresh_sync_settles_same_day_shape(
# the status is the source of truth for the settle; the UI's 2 s
# poll renders it at most one tick later).
expect(page.locator("#sync-progress-text")).to_have_text(
f"{len(SYNC_FILES)} added", timeout=SYNC_TIMEOUT_MS
counts_line(f"{len(SYNC_FILES)} added"), timeout=SYNC_TIMEOUT_MS
)
btn = page.locator("#sync-btn")
expect(btn).to_be_enabled()
+2 -1
View File
@@ -124,6 +124,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -807,7 +808,7 @@ def test_sync_button_names_the_summary_phase(
expect(btn).to_be_enabled()
expect(btn).not_to_have_attribute("aria-busy")
# The counts line rides the box (phase 150 — the sole sync surface).
expect(page.locator("#sync-progress-text")).to_have_text(f"{N_FILES} added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(f"{N_FILES} added"))
# ---------------------------------------------------------------------------
+7 -6
View File
@@ -132,6 +132,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -647,11 +648,11 @@ def test_upload_toast_then_navigate_away(
# "(n/m)" count is IN the full string (the never-clipped pin),
# and the button title carries the same full string (A4 — one
# source, two surfaces).
expect(page.locator("#sync-progress-text")).to_have_text(
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(
re.compile(
rf"^Syncing… {re.escape(UPLOAD_NAME)}/.+\.md \(\d+/{N_FILES}\)"
r"(?: · chunk \d+/\d+)?$"
),
)),
timeout=SETTLE_TIMEOUT_MS,
)
# The box is the VISIBLE running surface (the count's old
@@ -686,7 +687,7 @@ def test_upload_toast_then_navigate_away(
expect(btn).to_be_enabled()
expect(btn).not_to_have_attribute("aria-busy")
# The counts line rides the box (phase 150 — the sole sync surface).
expect(page.locator("#sync-progress-text")).to_have_text(f"{N_FILES} added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(f"{N_FILES} added"))
# Phase 97: the catalog is the drill-down tree — drill source →
# folder `docs` before the per-level row asserts (the flat total
# no longer exists in one tbody).
@@ -844,11 +845,11 @@ def test_sync_live_file_label(
# "(n/m)" count is IN the full string (the never-clipped pin),
# and the button title carries the same full string (A4 — one
# source, two surfaces).
expect(page.locator("#sync-progress-text")).to_have_text(
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(
re.compile(
rf"^Syncing… {re.escape(SYNC_SOURCE_DIR)}/.+\.md \(\d+/{N_FILES}\)"
r"(?: · chunk \d+/\d+)?$"
),
)),
timeout=SETTLE_TIMEOUT_MS,
)
# The box is the VISIBLE running surface (the count's old
@@ -885,7 +886,7 @@ def test_sync_live_file_label(
expect(btn).to_be_enabled()
expect(btn).not_to_have_attribute("aria-busy")
# The counts line rides the box (phase 150 — the sole sync surface).
expect(page.locator("#sync-progress-text")).to_have_text(f"{N_FILES} added")
expect(page.locator("#sync-progress-text")).to_have_text(counts_line(f"{N_FILES} added"))
# Phase 97: the catalog is the drill-down tree — drill source →
# folder `notes` before the per-level row asserts (the flat total
# no longer exists in one tbody).
+4 -1
View File
@@ -111,6 +111,7 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.sync_helpers import counts_line
REPO = Path(__file__).resolve().parents[2]
@@ -574,7 +575,9 @@ def test_ignore_list_then_sync_scans(
expect(page.locator("#sync-label")).to_have_text(
SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS
)
expect(page.locator("#sync-progress-text")).to_have_text("2 added", timeout=SYNC_TIMEOUT_MS)
expect(page.locator("#sync-progress-text")).to_have_text(
counts_line("2 added"), timeout=SYNC_TIMEOUT_MS
)
expect(page.locator("#sync-btn")).to_be_enabled()
expect(page.locator("#sync-btn")).not_to_have_attribute("aria-busy")
+44 -1
View File
@@ -4,12 +4,19 @@ Runs the real import pipeline (walk → chunk → embed → upsert) into the
local compose Postgres, then checks the DB state *and* the API shape a
browser would consume. Embeddings come from a deterministic in-process
fake, so no network is needed.
Phase 187 (task 02) pins the PLAN §9 cron-safe summary line's ONE new
trailing field (``rss_mb=``) on both the initial import and the
idempotent re-run, with every earlier field byte-identical.
"""
from __future__ import annotations
import asyncio
import logging
import re
from pathlib import Path
import pytest
from sqlalchemy import func, select, text
from app.models import Chunk, Document
@@ -36,9 +43,22 @@ EXPECTED_DOCS = {
}
def test_import_fixtures_end_to_end(admin_client, db) -> None:
#: Phase 187 (task 02): the ONE rendering :func:`app.core.memory.fmt_mb`
#: allows for an MB field — ``?`` when the platform cannot know, else a
#: positive integer (never ``None``, never a float or a negative).
MB = r"(?:\?|[1-9][0-9]*)"
def test_import_fixtures_end_to_end(
admin_client, db, caplog: pytest.LogCaptureFixture
) -> None:
db.execute(text("TRUNCATE chunks, documents, query_log"))
db.commit()
# Phase 187 (task 02): the PLAN §9 cron-safe summary line gains a
# trailing ``rss_mb=`` — captured for BOTH runs below (the initial
# import and the idempotent re-run), the level set once so every
# ``import_sources`` call of this test logs at INFO.
caplog.set_level(logging.INFO, logger="app.importer")
llm = FakeEmbedder()
summary = asyncio.run(import_sources([FIXTURES], llm, session=db))
@@ -111,5 +131,28 @@ def test_import_fixtures_end_to_end(admin_client, db) -> None:
assert s2.unchanged == 13 and s2.added == 0
assert len(llm.calls) == calls_before # unchanged → no embedding requests
# Phase 187 (task 02): every run's summary line ends with the ONE
# appended memory field, and the pre-phase fields are byte-identical
# (LOCKED A2 — append-only, so every existing grep keeps working).
summary_lines = [
r.getMessage()
for r in caplog.records
if r.getMessage().startswith("import: summary files=")
]
assert len(summary_lines) == 2, summary_lines
for line in summary_lines:
assert re.search(rf" rss_mb={MB}$", line), line
assert summary_lines[0].startswith(
"import: summary files=13 added=13 updated=0 unchanged=0 pruned=0 errors=0 "
)
assert re.search(
rf"formats=md:5,container:1,j2:1,json:1,network:1,py:1,txt:1,volume:1,yaml:1"
rf" rss_mb={MB}$",
summary_lines[0],
), summary_lines[0]
assert summary_lines[1].startswith(
"import: summary files=13 added=0 updated=0 unchanged=13 pruned=0 errors=0 "
)
db.execute(text("TRUNCATE chunks, documents, query_log"))
db.commit()
+377 -14
View File
@@ -86,12 +86,20 @@ empty, the final file counts kept, the chunk position cleared) and a
second run after the cancel completes to ``success`` (the restart is
clean — the stale cancel event did not bleed in, and only the
completed run bumps the sources version).
Phase 187 (task 02): every sync LIFECYCLE line carries the process
memory — ``sync: started … rss_mb=N``, and ``… rss_mb=N rss_peak_mb=M``
on all three terminals (success, cancelled, failed) plus the importer's
``import: summary …``. The fields are APPENDED (LOCKED A2), so every
pre-phase prefix below is byte-identical and the pins here assert BOTH
halves of that contract.
"""
from __future__ import annotations
import asyncio
import logging
import os
import re
import subprocess
import textwrap
import threading
@@ -109,6 +117,7 @@ from sqlalchemy.orm import Session
from app.api import sync as sync_api
from app.config import Settings
from app.core.memory import fmt_mb, rss_mb, rss_peak_mb
from app.db import SessionLocal, db_available
from app.main import app as fastapi_app
from app.models import Document, GitSource
@@ -130,14 +139,21 @@ def _fresh_sync_state() -> Iterator[None]:
Phase 134 (task 01): the module ``_cancel_event`` joins the reset —
a stale (set) event from a previous test's cancelled run would
otherwise bleed into the next run's boundary checks."""
otherwise bleed into the next run's boundary checks.
Phase 187 (task 03): so does the memory-sampling guard
(``_rss_sampled_at``) — module-global state beside ``_status``.
Leaving it set would make one test's sample answer for the next
one's "did the run read the number?" assertion."""
sync_api._status = sync_api.SyncStatus()
sync_api._task = None
sync_api._cancel_event = asyncio.Event()
sync_api._rss_sampled_at = float("-inf")
yield
sync_api._status = sync_api.SyncStatus()
sync_api._task = None
sync_api._cancel_event = asyncio.Event()
sync_api._rss_sampled_at = float("-inf")
@pytest.fixture(autouse=True)
@@ -307,6 +323,21 @@ def _wait_phase(client: TestClient, want: str, timeout: float = 5.0) -> dict:
time.sleep(0.02)
#: Phase 187 (task 02): the ONE rendering :func:`app.core.memory.fmt_mb`
#: allows for an MB field — ``?`` when the platform cannot know (no
#: ``/proc``, no ``resource``), else a positive integer (never ``None``,
#: never a float or a negative).
MB = r"(?:\?|[1-9][0-9]*)"
def _log_lines(caplog: pytest.LogCaptureFixture, prefix: str) -> list[str]:
"""Captured messages starting with ``prefix`` — the lifecycle lines
are pinned by their PRE-phase prefix, so the append-only contract
(LOCKED A2) is visible in the assertion itself.
"""
return [r.getMessage() for r in caplog.records if r.getMessage().startswith(prefix)]
class FakeImportSources:
"""Records every ``import_sources`` call; returns a canned summary."""
@@ -546,6 +577,10 @@ def test_admin_sync_success_reports_full_detail(
"current_summary": None,
"summaries_done": 0,
"summaries_total": 0,
# Phase 187 (task 03): the memory keys — null/null on a run that
# never happened (never 0, and the page renders no peak clause).
"rss_mb": None,
"rss_peak_mb": None,
}
r = sync_client.post("/api/sync")
@@ -1567,7 +1602,11 @@ def test_summaries_phase_reports_the_affected_count_as_total(
def test_git_failure_marks_failed_and_skips_import(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
repo_url = f"file://{tmp_path / 'bad.git'}"
_seed(db, repo_url)
@@ -1592,9 +1631,9 @@ def test_git_failure_marks_failed_and_skips_import(
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "failed")
with caplog.at_level(logging.INFO, logger="app.api.sync"):
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "failed")
assert "bad.git" in body["error"] # the failing repo is named
assert "fatal: repository not found" in body["error"]
assert body["detail"] == {}
@@ -1613,6 +1652,21 @@ def test_git_failure_marks_failed_and_skips_import(
# Phase 53: a FAILED sync never bumps — the version is untouched.
assert current_sources_version(db) == 0
# Phase 187 (task 02): the failed funnel keeps its traceback AND
# gains ONE sanitized one-line summary (LOCKED A2 — appended, so the
# pre-phase ``sync: failed`` label is untouched). Asserted before the
# restart below, which fails a SECOND time.
tracebacks = [
r for r in caplog.records if r.getMessage() == "sync: failed" and r.exc_info
]
assert len(tracebacks) == 1, "logger.exception must still write the traceback"
failed = _log_lines(caplog, "sync: failed state=failed")
assert len(failed) == 1, failed
assert failed[0].startswith("sync: failed state=failed files_done=0 files_total=0")
assert re.search(
rf" rss_mb={MB} rss_peak_mb={MB} error=.*bad\.git", failed[0]
), failed[0]
# A failed run leaves the system restartable: a new POST is accepted.
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "failed")
@@ -1855,7 +1909,11 @@ def test_missing_local_dir_fails_loudly_and_imports_nothing(
def test_import_error_is_reported_with_credentials_masked(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
@@ -1892,9 +1950,9 @@ def test_import_error_is_reported_with_credentials_masked(
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "failed")
with caplog.at_level(logging.INFO, logger="app.api.sync"):
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "failed")
assert "*****@aipi.reeseapps.com" in body["error"] # credentials masked
assert "user:secret" not in body["error"]
assert "connection refused" in body["error"] # the reason survives
@@ -1905,6 +1963,16 @@ def test_import_error_is_reported_with_credentials_masked(
assert body["current_summary"] is None
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
# Phase 187 (task 02): the one-line failed summary logs the SAME
# masked string the status field carries — the raw exception text
# never reaches the log through this line (the ``logger.exception``
# traceback is the pre-phase surface, unchanged).
failed = _log_lines(caplog, "sync: failed state=failed")
assert len(failed) == 1, failed
assert "*****@aipi.reeseapps.com" in failed[0]
assert "user:secret" not in failed[0]
assert re.search(rf" rss_mb={MB} rss_peak_mb={MB} error=", failed[0]), failed[0]
# --- phase 41: model probe (fail fast before any clone) --------------------
@@ -2350,12 +2418,29 @@ def test_cancel_mid_run_lands_in_cancelled_then_resyncs_clean(
assert body["files_done"] == 1 and body["files_total"] == 3
# ... and the chunk position clears (the phase-129 convention).
assert body["current_chunk"] == 0 and body["chunks_total"] == 0
# PLAN §9: the ample-logging line for the converted cancel.
assert any(
r.getMessage()
== "sync: cancelled by user state=cancelled files_done=1 files_total=3"
for r in caplog.records
# Phase 187 (task 03): the CANCELLED terminal keeps the memory pair
# too — a cancelled run is exactly the run an operator wants the
# number for (the walk stopped, the box is still hot), and the
# phase-64 keep-final-values convention covers it like the counts.
if _memory_is_knowable():
assert body["rss_mb"] is not None and body["rss_peak_mb"] is not None, body
assert body["rss_peak_mb"] >= body["rss_mb"] - _MB_SLACK, body
else:
assert body["rss_mb"] is None and body["rss_peak_mb"] is None, body
# PLAN §9: the ample-logging line for the converted cancel. Phase
# 187 (task 02) APPENDS the two memory fields (LOCKED A2) — the
# pre-phase prefix is byte-identical, the tail is `?` or a positive
# integer, and NOTHING else was inserted or reworded.
cancelled = _log_lines(caplog, "sync: cancelled by user ")
assert len(cancelled) == 1, cancelled
assert cancelled[0].startswith(
"sync: cancelled by user state=cancelled files_done=1 files_total=3"
)
assert re.fullmatch(
rf"sync: cancelled by user state=cancelled files_done=1 files_total=3"
rf" rss_mb={MB} rss_peak_mb={MB}",
cancelled[0],
), cancelled[0]
# The cancelled run never reached the version bump.
assert current_sources_version(db) == 0
@@ -2518,3 +2603,281 @@ def test_git_row_branch_mismatch_button_path_fails_loud(
assert proc.returncode == 0 and proc.stdout.strip() == "main"
assert (dest / "a.md").is_file()
assert not (dest / "b.md").exists()
# --- phase 187 (task 02): the lifecycle lines carry the memory numbers ----
def test_success_lifecycle_lines_carry_memory_fields(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
"""The owner incident (2026-10-15) turned "how much RAM did that sync
use?" into archaeology across three boots' ``systemd`` teardown lines
— with one boot missing entirely. From now on ONE grep answers it:
``sync: started … rss_mb=N`` and ``sync: done detail=… rss_mb=N
rss_peak_mb=M``. Both fields are APPENDED after the existing ones
(LOCKED A2), so the pre-phase prefix stays byte-identical; the value
is a positive integer MB or ``?`` where the platform cannot know
(:mod:`app.core.memory` never raises).
"""
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
monkeypatch.setattr(
sync_api, "import_sources", FakeImportSources(ImportSummary(files=1, added=1))
)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
_login(sync_client)
with caplog.at_level(logging.INFO, logger="app.api.sync"):
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
assert body["state"] == "success"
started = _log_lines(caplog, "sync: started ")
assert len(started) == 1, started
assert started[0].startswith("sync: started repos=1 origin=db git=1 local=0")
assert re.fullmatch(
rf"sync: started repos=1 origin=db git=1 local=0 rss_mb={MB}", started[0]
), started[0]
done = _log_lines(caplog, "sync: done ")
assert len(done) == 1, done
# The pre-existing prefix, the detail dict still first, the two new
# fields trailing — nothing renamed, reordered or re-typed.
assert done[0].startswith("sync: done detail={'files': 1,")
assert re.search(rf" rss_mb={MB} rss_peak_mb={MB}$", done[0]), done[0]
# --- phase 187 (task 03): the status carries the memory pair --------------
#
# The same numbers the lifecycle lines carry, on the surface the Sources
# page polls: two ADDITIVE keys (rss_mb / rss_peak_mb, int | null),
# null until a run samples them and the FINAL pair in every terminal
# (the phase-64 keep-final-values convention). The exposure guard is the
# route's own admin gate — the Sources view is admin-only (PLAN B5), so
# no memory number ever leaves the app on an anonymous surface.
#: The status payload's key set BEFORE phase 187 — the additive pin's
#: left-hand side (the payload is exactly this set plus the memory pair,
#: forever).
_PRE_187_STATUS_KEYS = frozenset(
{
"state",
"started_at",
"finished_at",
"detail",
"error",
"current_file",
"files_done",
"files_total",
"current_chunk",
"chunks_total",
"images_done",
"images_total",
"phase",
"current_summary",
"summaries_done",
"summaries_total",
}
)
#: Phase 187 (task 03): the two additive memory keys.
_MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
#: The pair is read from two DIFFERENT kernel counters and the high-water
#: refreshes LAZILY, so a raw ``rss_peak_mb()`` really can sit under a raw
#: ``rss_mb()`` (measured: ``rss_mb=164`` against ``rss_peak_mb=162``) —
#: the nuance ``app/core/memory.py`` documents. Task 04 moved the fix to
#: the ONE place the pair is published (:func:`app.api.sync._sample_rss`,
#: which lifts a lagging peak to the current reading), so the slack here is
#: ZERO and the assertion is the invariant itself: a reported peak below
#: the run's own current footprint is a false number, not a smaller one.
_MB_SLACK = 0
def _memory_is_knowable() -> bool:
"""Does THIS platform answer the two helpers? Linux (the deploy
target: ``/proc`` + ``resource``) always does; a runner without
either gets null/null and the value pins step aside — the SHAPE pins
never do, because ``null`` is the honest answer there too."""
return rss_mb() is not None and rss_peak_mb() is not None
def _stub_a_one_file_run(
monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""The shortest honest run: one git row, a stub clone that writes one
file, a stubbed import (the runner's own numbers are what's under
test, not the importer's)."""
_seed(db, f"file://{tmp_path / 'repo.git'}")
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
monkeypatch.setattr(
sync_api, "import_sources", FakeImportSources(ImportSummary(files=1, added=1))
)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
monkeypatch.setattr(sync_api, "generate_folder_summaries", FakeFolderSummaries())
def test_memory_status_keys_are_the_only_addition_and_null_before_a_run(
sync_client: TestClient,
) -> None:
"""The additive pin on the wire: the payload's key set is EXACTLY the
pre-phase-187 set plus ``rss_mb`` / ``rss_peak_mb``, and a run that
never happened answers ``null`` / ``null`` — not 0, not a plausible
number. That null is also what makes the Sources page render the
counts line with no peak clause after an app restart."""
_login(sync_client)
body = sync_client.get("/api/sync/status").json()
assert set(body) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS, set(body)
assert body["rss_mb"] is None
assert body["rss_peak_mb"] is None
# The pre-existing shapes, unchanged: the state word, the two ISO
# slots, the counts, the phase machine.
assert body["state"] == "idle"
assert body["started_at"] is None and body["finished_at"] is None
assert body["detail"] == {} and body["error"] is None
assert body["files_done"] == 0 and body["files_total"] == 0
assert body["current_chunk"] == 0 and body["chunks_total"] == 0
assert body["images_done"] == 0 and body["images_total"] == 0
assert body["phase"] is None and body["current_summary"] is None
assert body["summaries_done"] == 0 and body["summaries_total"] == 0
def test_success_terminal_status_keeps_the_final_memory_pair(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
"""After a run the payload answers the incident's question directly:
both keys are integers here (Linux), the peak is the run's
high-water next to the final counts (within the documented MB
slack), and the pair SURVIVES the run — a later poll, i.e. the page
reloaded after the sync, still shows it. The ``sync: done`` line
logs the SAME pair: one reading, two surfaces."""
_stub_a_one_file_run(monkeypatch, db, tmp_path)
_login(sync_client)
with caplog.at_level(logging.INFO, logger="app.api.sync"):
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
assert body["state"] == "success"
current, peak = body["rss_mb"], body["rss_peak_mb"]
assert current is None or isinstance(current, int)
assert peak is None or isinstance(peak, int)
if _memory_is_knowable():
assert current is not None and peak is not None, body
assert current > 0 and peak > 0, body
assert peak >= current - _MB_SLACK, (
"the published pair broke its peak >= current invariant — the"
" sampling point must lift a lazily-refreshed high-water to the"
f" current reading (slack {_MB_SLACK} MB):"
f" current={current} peak={peak}"
)
# The terminal keeps the pair (no re-sample, no reset between the
# run and the next poll — the page can be reloaded at leisure).
again = sync_client.get("/api/sync/status").json()
assert (again["rss_mb"], again["rss_peak_mb"]) == (current, peak), again
# Log / payload agreement: the line the operator greps and the key
# the page renders come off the same reading.
done = _log_lines(caplog, "sync: done ")
assert len(done) == 1, done
assert done[0].endswith(
f"rss_mb={fmt_mb(current)} rss_peak_mb={fmt_mb(peak)}"
), done[0]
# The pre-existing payload is untouched by all of this.
assert set(again) == _PRE_187_STATUS_KEYS | _MEMORY_KEYS
assert again["files_done"] == 0 or isinstance(again["files_done"], int)
def test_failed_terminal_keeps_the_memory_pair(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""A run that dies is exactly the run whose memory matters, so the
FAILED funnel samples too: the payload keeps the pair beside the
sanitized error (the task-02 line logs the same pair)."""
repo_url = f"file://{tmp_path / 'missing.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_stub_probe(monkeypatch)
def broken_clone(url: str, dest: Path | str, branch: str | None = None) -> Path:
raise GitSyncError(f"git clone failed for {url}")
monkeypatch.setattr(sync_api, "clone_or_pull", broken_clone)
monkeypatch.setattr(
sync_api,
"import_sources",
FakeImportSources(ImportSummary()), # must never run
)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "failed")
assert "missing.git" in body["error"]
assert body["detail"] == {}
if _memory_is_knowable():
assert body["rss_mb"] is not None and body["rss_peak_mb"] is not None, body
assert body["rss_peak_mb"] >= body["rss_mb"] - _MB_SLACK, body
else: # a platform that cannot know still answers, with null
assert body["rss_mb"] is None and body["rss_peak_mb"] is None, body
def test_memory_pair_is_never_an_anonymous_surface(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""A4's exposure guard on the wire: the status route is
``require_admin``, and the new keys ride that gate — an anonymous
visitor gets the pre-phase 403 with the pre-phase body, with or
without a run in flight (the Sources view is the only place the
number renders, and it is admin-only)."""
anonymous = TestClient(fastapi_app)
r = anonymous.get("/api/sync/status")
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
_stub_a_one_file_run(monkeypatch, db, tmp_path)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "success")
r = anonymous.get("/api/sync/status")
assert r.status_code == 403, "no memory number on an anonymous surface"
assert r.json() == {"detail": "admin only"}
+285
View File
@@ -0,0 +1,285 @@
"""Unit tests: ``app.core.memory`` — process RSS helpers (phase 187, task 01).
Every branch of the four helpers, with the filesystem stubbed through the
module's path constants (``PROC_STATUS_PATH`` / ``CGROUP_PEAK_PATH`` →
``tmp_path`` files, so ``open`` is never mocked globally) and ``resource``
stubbed through ``memory._resource`` (the guarded-import binding):
* the REAL ``/proc/self/status`` read on a Linux runner — ``rss_mb()`` is a
positive int and provably comes from the ``VmRSS`` parse, not the
fallback (the fallback is pinned to a sentinel that never surfaces);
* the kB→MB floor arithmetic (1500 kB → 1 MB);
* missing file, unparseable value, value-less ``VmRSS:`` line, and a file
without any ``VmRSS`` line — each falls back to ``rss_peak_mb()``;
* the non-Linux path: ``/proc`` open raises AND ``getrusage`` is patched →
the proxy value; both unavailable → ``None``;
* ``rss_peak_mb`` arithmetic, the no-``resource`` ``None``, and the
never-raise wrapper when ``getrusage`` itself raises;
* ``cgroup_peak_mb`` present (bytes→MB), absent, garbage, empty → the
latter three are ``None``;
* ``fmt_mb(None) == "?"``, ``fmt_mb(0) == "0"`` (0 is a value, not
missing), plain ints as digits;
* the purity contract: stdlib imports only — no ``app.`` import — and the
docstring carries the "Pure stdlib" line the phase completion criteria
require.
"""
from __future__ import annotations
import ast
import importlib
import inspect
import sys
from types import SimpleNamespace
from typing import Any
import pytest
import app.core.memory as memory
#: Stub whose ``getrusage`` answers with a fixed ``ru_maxrss`` (kB on
#: Linux) so fallback assertions are deterministic.
def _fake_resource(maxrss_kb: int) -> Any:
return SimpleNamespace(
RUSAGE_SELF=0,
getrusage=lambda who: SimpleNamespace(ru_maxrss=maxrss_kb),
)
#: Stub whose ``getrusage`` raises — the never-raise contract probe.
def _raising_resource(exc: Exception) -> Any:
def getrusage(who: int) -> Any:
raise exc
return SimpleNamespace(RUSAGE_SELF=0, getrusage=getrusage)
# ---------------------------------------------------------------------------
# rss_mb — the real /proc read (Linux) and the stubbed failure modes
# ---------------------------------------------------------------------------
@pytest.mark.skipif(not sys.platform.startswith("linux"), reason="needs /proc/self/status")
def test_rss_mb_reads_real_vm_rss_on_linux(monkeypatch: pytest.MonkeyPatch) -> None:
# Pin the FALLBACK to a negative sentinel: a positive int can therefore
# only come from the real VmRSS parse, not from getrusage.
monkeypatch.setattr(memory, "rss_peak_mb", lambda: -1)
value = memory.rss_mb()
assert value is not None
assert isinstance(value, int)
assert value > 0
def test_rss_mb_parses_vm_rss_kb_flooring_to_mb(
monkeypatch: pytest.MonkeyPatch, tmp_path: Any
) -> None:
status = tmp_path / "status"
status.write_text("Name:\tpython\nVmRSS:\t 1500 kB\nVmSize:\t999999 kB\n")
monkeypatch.setattr(memory, "PROC_STATUS_PATH", str(status))
monkeypatch.setattr(memory, "_resource", _fake_resource(999_999))
assert memory.rss_mb() == 1 # 1500 kB // 1024 — integer floor, not round
def test_rss_mb_missing_file_falls_back_to_getrusage(
monkeypatch: pytest.MonkeyPatch, tmp_path: Any
) -> None:
monkeypatch.setattr(memory, "PROC_STATUS_PATH", str(tmp_path / "nope"))
monkeypatch.setattr(memory, "_resource", _fake_resource(9216)) # 9 MB
assert memory.rss_mb() == 9
def test_rss_mb_unparseable_value_falls_back(
monkeypatch: pytest.MonkeyPatch, tmp_path: Any
) -> None:
status = tmp_path / "status"
status.write_text("VmRSS:\tnot-a-number kB\n")
monkeypatch.setattr(memory, "PROC_STATUS_PATH", str(status))
monkeypatch.setattr(memory, "_resource", _fake_resource(2048))
assert memory.rss_mb() == 2
def test_rss_mb_value_less_line_falls_back(
monkeypatch: pytest.MonkeyPatch, tmp_path: Any
) -> None:
# "VmRSS:" with no value: the split()[1] raises IndexError internally,
# the suppress turns it into the rss_peak_mb() proxy.
status = tmp_path / "status"
status.write_text("VmRSS:\n")
monkeypatch.setattr(memory, "PROC_STATUS_PATH", str(status))
monkeypatch.setattr(memory, "_resource", _fake_resource(3072))
assert memory.rss_mb() == 3
def test_rss_mb_file_without_vm_rss_line_falls_back(
monkeypatch: pytest.MonkeyPatch, tmp_path: Any
) -> None:
status = tmp_path / "status"
status.write_text("Name:\tpython\nVmSize:\t999999 kB\nVmHWM:\t888888 kB\n")
monkeypatch.setattr(memory, "PROC_STATUS_PATH", str(status))
monkeypatch.setattr(memory, "_resource", _fake_resource(1024))
assert memory.rss_mb() == 1
def test_rss_mb_returns_none_when_proc_and_resource_both_unavailable(
monkeypatch: pytest.MonkeyPatch, tmp_path: Any
) -> None:
monkeypatch.setattr(memory, "PROC_STATUS_PATH", str(tmp_path / "nope"))
monkeypatch.setattr(memory, "_resource", None)
assert memory.rss_mb() is None
# ---------------------------------------------------------------------------
# rss_peak_mb — arithmetic, the guarded import, the never-raise wrapper
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
("maxrss_kb", "expected_mb"),
[(1024, 1), (1023, 0), (1500, 1), (4_294_967_296, 4_194_304)],
)
def test_rss_peak_mb_arithmetic(
monkeypatch: pytest.MonkeyPatch, maxrss_kb: int, expected_mb: int
) -> None:
monkeypatch.setattr(memory, "_resource", _fake_resource(maxrss_kb))
assert memory.rss_peak_mb() == expected_mb
def test_rss_peak_mb_none_without_resource_module(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setattr(memory, "_resource", None)
assert memory.rss_peak_mb() is None
def test_guarded_import_degrades_to_none_when_resource_is_missing(
tmp_path: Any,
) -> None:
# The guarded-import branch itself: a platform WITHOUT the Unix-only
# `resource` module must still import this module, and every helper
# must answer None (no /proc, no proxy). `sys.modules[name] = None`
# makes `import resource` raise ImportError, so re-executing the
# module exercises the except arm for real (no pragma).
saved = sys.modules.get("resource")
sys.modules["resource"] = None # type: ignore[assignment]
try:
importlib.reload(memory)
assert memory._resource is None
assert memory.rss_peak_mb() is None
memory.PROC_STATUS_PATH = str(tmp_path / "nope") # reset by reload
assert memory.rss_mb() is None
finally:
if saved is not None:
sys.modules["resource"] = saved
else: # pragma: no cover - non-Unix runners never had it
sys.modules.pop("resource", None)
importlib.reload(memory) # restore the real bindings for later tests
def test_module_reloads_with_real_resource_after_the_missing_module_probe() -> None:
# Sibling pin: the finally-reload above restored the module, so the
# guarded import took the happy arm again (order-independent: this
# only passes if _resource is the live module on this platform).
if not sys.platform.startswith("linux"):
pytest.skip("resource exists on Linux")
assert memory._resource is not None
def test_rss_peak_mb_never_raises_when_getrusage_raises(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setattr(memory, "_resource", _raising_resource(OSError("boom")))
assert memory.rss_peak_mb() is None
def test_rss_peak_mb_real_call_on_linux_yields_positive_int() -> None:
if not sys.platform.startswith("linux"):
pytest.skip("resource.getrusage is the Linux deploy path")
value = memory.rss_peak_mb()
assert value is not None
assert isinstance(value, int)
assert value > 0
# ---------------------------------------------------------------------------
# cgroup_peak_mb — present / absent / garbage / empty
# ---------------------------------------------------------------------------
def test_cgroup_peak_mb_present_converts_bytes_to_mb(
monkeypatch: pytest.MonkeyPatch, tmp_path: Any
) -> None:
# 2,306,867,200 bytes == 2200 MiB — the incident box's ~2.16 GB of
# cgroup page-cache-inflated "peak" (module docstring caveat).
peak = tmp_path / "memory.peak"
peak.write_text("2306867200\n")
monkeypatch.setattr(memory, "CGROUP_PEAK_PATH", str(peak))
assert memory.cgroup_peak_mb() == 2200
def test_cgroup_peak_mb_absent_is_none(
monkeypatch: pytest.MonkeyPatch, tmp_path: Any
) -> None:
monkeypatch.setattr(memory, "CGROUP_PEAK_PATH", str(tmp_path / "memory.peak"))
assert memory.cgroup_peak_mb() is None
@pytest.mark.parametrize("body", ["max", "", " \n", "1048576 kB"])
def test_cgroup_peak_mb_garbage_is_none(
monkeypatch: pytest.MonkeyPatch, tmp_path: Any, body: str
) -> None:
peak = tmp_path / "memory.peak"
peak.write_text(body)
monkeypatch.setattr(memory, "CGROUP_PEAK_PATH", str(peak))
assert memory.cgroup_peak_mb() is None
# ---------------------------------------------------------------------------
# fmt_mb — the "?" rendering
# ---------------------------------------------------------------------------
def test_fmt_mb_renders_none_as_question_mark_and_values_as_digits() -> None:
assert memory.fmt_mb(None) == "?"
assert memory.fmt_mb(0) == "0" # 0 is a VALUE, not missing
assert memory.fmt_mb(512) == "512"
# ---------------------------------------------------------------------------
# Purity contract (completion criterion: stdlib only, no app. imports)
# ---------------------------------------------------------------------------
def test_module_imports_stdlib_only_and_declares_pure_stdlib() -> None:
tree = ast.parse(inspect.getsource(memory))
imported: set[str] = set()
for node in ast.walk(tree):
if isinstance(node, ast.Import):
imported.update(alias.name.split(".")[0] for alias in node.names)
elif isinstance(node, ast.ImportFrom):
if node.level: # relative import — still inside the app package
imported.add("app")
elif node.module:
imported.add(node.module.split(".")[0])
assert "app" not in imported
# __future__, contextlib, types are static; `resource` is the guarded
# Unix-only stdlib import — everything must stay standard library.
assert imported <= {"__future__", "contextlib", "resource", "types"}, imported
assert "Pure stdlib" in (memory.__doc__ or "")
+6 -1
View File
@@ -667,7 +667,12 @@ def test_settle_and_idle_labels_go_through_the_parts_writer() -> None:
success = _fn(js, "applySyncSuccess")
assert '"Synced"' in success and "fmtSyncTime(status.finished_at)" in success
i_settle = success.find("settleSyncButton(")
i_counts = success.find("setSyncProgress(fmtSyncResult(status.detail))")
# Phase 187 (task 03): the clause builder rides the SAME writer call
# (the counts line + its trailing " · peak N MB", null rendering no
# clause), so the ordering pin reads the new argument list.
i_counts = success.find(
"setSyncProgress(fmtSyncResult(status.detail) + fmtSyncPeakClause(status))"
)
assert -1 < i_settle < i_counts, (
"the counts line (the box's result line) follows the settle label"
)
+18 -3
View File
@@ -18,6 +18,7 @@ import base64
import hashlib
import logging
import os
import re
import textwrap
from collections.abc import Callable, Sequence
from datetime import UTC, datetime
@@ -54,6 +55,11 @@ from tests.fakes import FakeEmbedder
#: (``Settings().import_extension_set`` — now seventeen formats).
DEFAULT_EXTS = frozenset({".md", ".markdown", ".txt", ".yaml", ".yml", ".json", ".py"})
#: Phase 187 (task 02): the ONE rendering :func:`app.core.memory.fmt_mb`
#: allows for an MB field — ``?`` when the platform cannot know, else a
#: positive integer (never ``None``, never a float or a negative).
MB = r"(?:\?|[1-9][0-9]*)"
class _PoisonEmbedder(FakeEmbedder):
"""Fails (like a real endpoint) on any text containing 'poison'."""
@@ -1001,7 +1007,10 @@ def test_import_summary_log_line_includes_summary_counters(
and the phase-151 ``pdf_images_deduped`` / ``pdf_images_blank_skipped``
counters (the parser's per-PDF drop counts, next to
``pdf_pictorial_pages``) sit after ``dates_updated``, before
``formats``."""
``formats``. Phase 187 (task 02) APPENDS a trailing ``rss_mb=``
(LOCKED A2 — the memory the run sits at, ``?`` when unknowable); the
pre-phase text up to and including ``formats=…`` is byte-identical.
"""
s = ImportSummary()
s.files, s.added, s.chunks, s.embed_batches = 3, 3, 5, 4
s.summaries, s.summary_errors = 2, 1
@@ -1019,7 +1028,7 @@ def test_import_summary_log_line_includes_summary_counters(
with caplog.at_level(logging.INFO, logger="app.importer"):
s.log()
line = caplog.records[-1].getMessage()
assert line == (
assert line.startswith(
"import: summary files=3 added=3 updated=0 unchanged=0 pruned=0 errors=0 "
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
"dates_updated=0 images_failed=0 svg_indexed=4 image_context_backfilled=2 "
@@ -1027,6 +1036,9 @@ def test_import_summary_log_line_includes_summary_counters(
"pdf_images_deduped=3 pdf_images_blank_skipped=1 "
"formats=yaml:2,md:1"
)
# Phase 187 (task 02): the ONE appended field, `?` (unknowable) or a
# positive integer MB — never `None`, never a negative or a float.
assert re.fullmatch(rf".* rss_mb={MB}", line), line
def test_prune_removes_files_now_excluded_by_format_filter(db, tmp_path: Path) -> None:
@@ -1082,7 +1094,10 @@ def test_formats_counter_counts_extensionless_name_token(
for r in caplog.records
if "import: summary files=" in r.getMessage()
)
assert line.endswith("formats=dockerfile:1,md:1")
# Phase 187 (task 02): ``formats=`` is no longer the LAST field
# (`` rss_mb=…`` follows), but its position and value are intact.
assert " formats=dockerfile:1,md:1" in line
assert re.search(rf"formats=dockerfile:1,md:1 rss_mb={MB}$", line), line
assert "unknown" not in line
finally:
_cleanup_source(db, root.name)
+485 -11
View File
@@ -203,6 +203,10 @@ RETIRED_ID = "-".join(("sync", "result")) # built at runtime: 3.12+ folds
HAVE_NODE = shutil.which("node") is not None
#: Phase 187 (task 02): what :func:`app.core.memory.fmt_mb` may render for
#: an MB field — ``?`` when the platform cannot know, else a positive int.
MB = r"(?:\?|[1-9][0-9]*)"
def _text(path: Path) -> str:
assert path.is_file(), f"missing frontend file: {path}"
@@ -1112,9 +1116,13 @@ def test_sources_js_renders_result_line_and_banner_directly() -> None:
for other subscribers."""
js = _text(SOURCES_JS)
success = _body(js, "applySyncSuccess")
assert "setSyncProgress(fmtSyncResult(status.detail))" in success, (
"the counts live in status.detail — a bare status renders all zeros"
)
# Phase 187 (task 03): the counts line keeps its builder and gains
# the TRAILING peak clause — a null pair appends the empty string, so
# every pre-phase line stays byte-identical.
assert (
"setSyncProgress(fmtSyncResult(status.detail) + fmtSyncPeakClause(status))"
in success
), "the counts live in status.detail — a bare status renders all zeros"
assert "loadTree()" in success, (
"the catalog re-fetches live on a successful sync (phase 97: the tree)"
)
@@ -1319,14 +1327,21 @@ def fresh_sync_status() -> Iterator[None]:
Phase 134 (task 01): the module ``_cancel_event`` joins the reset —
a stale (set) event from a previous test's cancelled run would
otherwise bleed into the next run's boundary checks."""
otherwise bleed into the next run's boundary checks.
Phase 187 (task 03): the memory-sampling guard (``_rss_sampled_at``)
joins the reset — it is module-global state beside ``_status``, and a
test that pins "the per-file hook sampled" has to know whether the
guard says "never read" or "still inside the 5 s window"."""
sync_api._status = sync_api.SyncStatus()
sync_api._task = None
sync_api._cancel_event = asyncio.Event()
sync_api._rss_sampled_at = float("-inf")
yield
sync_api._status = sync_api.SyncStatus()
sync_api._task = None
sync_api._cancel_event = asyncio.Event()
sync_api._rss_sampled_at = float("-inf")
class _GatedClone:
@@ -1567,9 +1582,11 @@ def test_idle_status_pins_full_shape_including_progress_keys(
) -> None:
"""Idle: the three phase-64 progress keys ride along as null/0/0,
the two phase-129 chunk keys as 0/0, the two phase-152 image keys
as 0/0, the four phase-98 phase keys as null/0/0/0, and EVERY
pre-existing key is unchanged — the full response dict is pinned,
so the current UI and every existing consumer keep working."""
as 0/0, the four phase-98 phase keys as null/0/0/0, the two phase-187
memory keys as null/null (NO reading is not 0 — and the Sources page
renders no peak clause for it), and EVERY pre-existing key is
unchanged — the full response dict is pinned, so the current UI and
every existing consumer keep working."""
assert sync_api.sync_status() == {
"state": "idle",
"started_at": None,
@@ -1592,6 +1609,10 @@ def test_idle_status_pins_full_shape_including_progress_keys(
"current_summary": None,
"summaries_done": 0,
"summaries_total": 0,
# Phase 187 (task 03): the memory keys — null/null idle (never
# sampled: never 0, never an invented number).
"rss_mb": None,
"rss_peak_mb": None,
}
@@ -2562,10 +2583,15 @@ def test_cancel_endpoint_stops_the_in_flight_run(
# through the closure in flight is gone; the FILE counts above
# keep their final values).
assert s["images_done"] == 0 and s["images_total"] == 0
# PLAN §9: the ample-logging line for the converted cancel.
# PLAN §9: the ample-logging line for the converted cancel. Phase
# 187 (task 02) APPENDS the memory fields (LOCKED A2) — the pre-phase
# prefix stays byte-identical, the tail is `?` or a positive int.
assert any(
r.getMessage()
== "sync: cancelled by user state=cancelled files_done=1 files_total=3"
re.fullmatch(
r"sync: cancelled by user state=cancelled files_done=1 files_total=3"
rf" rss_mb={MB} rss_peak_mb={MB}",
r.getMessage(),
)
for r in caplog.records
)
@@ -3091,7 +3117,9 @@ def test_sources_js_apply_sync_cancelled_is_neutral() -> None:
js = _text(SOURCES_JS)
body = _body(js, "applySyncCancelled")
assert 'settleSyncButton("Sync sources")' in body
assert 'setSyncProgress("Sync cancelled")' in body
# Phase 187 (task 03): the neutral copy keeps its exact wording and
# the peak clause trails it (a null pair appends the empty string).
assert 'setSyncProgress("Sync cancelled" + fmtSyncPeakClause(status))' in body
assert "hideSyncError()" in body
assert "emitSyncStatus(status)" in body
assert "showSyncModal" not in body, "cancel is intent — no modal"
@@ -3123,3 +3151,449 @@ def test_sources_js_section_header_documents_the_cancel_contract() -> None:
assert "5. sync cancelled" in header, "the tree's cancelled branch (the 5th)"
assert "8. both idle" in header, "the tree is now 8 branches"
assert "#sync-cancel-btn" in header, "the control's contract is documented"
# ---------- phase 187 (task 03): the run's memory on the status ----------
#
# The owner incident (2026-10-15): "how much RAM did that sync use?" was
# answered by reading systemd unit-teardown lines out of three boots —
# and the boot that killed the VM had none. Tasks 01/02 made the log
# carry the number; task 03 puts the SAME pair on the status object, so
# the page that started the run answers without a shell. The pins below
# hold the three things that make that honest: the pair is ADDITIVE (the
# key set is exactly the old set plus two), it is SAMPLED from ONE place
# (forced at the run start / every phase transition / every terminal,
# throttled to one read per 5 s in the per-file hook — LOCKED A4), and a
# reading that does not exist stays null (never 0, never invented).
#: The status payload's key set BEFORE phase 187 — the additive pin's
#: left-hand side: the payload is exactly this set plus
#: :data:`_MEMORY_KEYS`, nothing else, forever.
_PRE_187_STATUS_KEYS = frozenset(
{
"state",
"started_at",
"finished_at",
"detail",
"error",
"current_file",
"files_done",
"files_total",
"current_chunk",
"chunks_total",
"images_done",
"images_total",
"phase",
"current_summary",
"summaries_done",
"summaries_total",
}
)
#: Phase 187 (task 03): the two additive memory keys.
_MEMORY_KEYS = frozenset({"rss_mb", "rss_peak_mb"})
def test_status_key_set_is_the_old_set_plus_the_memory_pair(
fresh_sync_status: None,
) -> None:
"""The key-set pin (D4: "Additive only — every pre-existing key
keeps its name, type, and meaning"): the payload is EXACTLY the
pre-phase-187 set ``∪ {rss_mb, rss_peak_mb}`` — a fifth sub-position
or a renamed field fails here even when the value assertions pass."""
keys = set(sync_api.sync_status())
assert keys == _PRE_187_STATUS_KEYS | _MEMORY_KEYS, keys
assert _MEMORY_KEYS.isdisjoint(_PRE_187_STATUS_KEYS)
class _FakeClock:
"""``sync_api.time`` stubbed: the sampler is the module's only
``time.monotonic()`` reader, so the 5 s guard is assertable without
sleeping (the test moves the clock, never the wall)."""
def __init__(self, start: float = 10_000.0) -> None:
self.now = start
def monotonic(self) -> float:
return self.now
class _StubMemory:
"""The two ``app.core.memory`` reads as a scripted pair-per-read
source: sample *n* returns ``readings[n]`` (the last pair repeats)
and :attr:`reads` counts the samples, so "sampled at most once per
5 s" is a number and not a comment.
Both helpers are patched on ``app.api.sync`` (the module under test
imports them by name), which is also what proves the sampler reads
them from ONE place: a call counter that disagrees with
``_status`` is a second, unsampled write path."""
def __init__(
self,
monkeypatch: pytest.MonkeyPatch,
readings: Sequence[tuple[int | None, int | None]],
) -> None:
self.readings = list(readings)
self.reads = 0
monkeypatch.setattr(sync_api, "rss_mb", self._current)
monkeypatch.setattr(sync_api, "rss_peak_mb", self._peak)
def _pair(self) -> tuple[int | None, int | None]:
return self.readings[min(self.reads, len(self.readings) - 1)]
def _current(self) -> int | None:
return self._pair()[0]
def _peak(self) -> int | None:
value = self._pair()[1]
self.reads += 1 # the PAIR is one sample — counted when it completes
return value
def test_sample_rss_stores_both_readings_from_one_sample(
fresh_sync_status: None, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``_sample_rss`` is the ONE writer of the pair: both fields land
from the same read, so the status and the lifecycle line can never
report two different instants."""
monkeypatch.setattr(sync_api, "time", _FakeClock())
memory = _StubMemory(monkeypatch, [(412, 1_936)])
sync_api._sample_rss(force=True)
assert sync_api._status.rss_mb == 412
assert sync_api._status.rss_peak_mb == 1_936
assert memory.reads == 1
def test_sample_rss_throttles_the_unforced_call_and_force_always_reads(
fresh_sync_status: None, monkeypatch: pytest.MonkeyPatch
) -> None:
"""LOCKED A4 / D3, pinned on the guard itself: the FIRST call reads
(never sampled), the next one inside ``_RSS_SAMPLE_INTERVAL_S`` is a
NO-OP, one past the interval reads again, and ``force=True`` reads
regardless AND re-arms the window (which is what keeps the 27k-file
walk at ~600 reads)."""
clock = _FakeClock(10_000.0)
monkeypatch.setattr(sync_api, "time", clock)
memory = _StubMemory(
monkeypatch, [(100, 200), (101, 201), (102, 202), (103, 203), (104, 204)]
)
assert sync_api._RSS_SAMPLE_INTERVAL_S == 5.0, "the phase's sampling window"
sync_api._sample_rss() # never sampled → reads
assert (sync_api._status.rss_mb, sync_api._status.rss_peak_mb) == (100, 200)
clock.now += sync_api._RSS_SAMPLE_INTERVAL_S - 0.001
sync_api._sample_rss() # inside the window → no-op
assert memory.reads == 1
assert (sync_api._status.rss_mb, sync_api._status.rss_peak_mb) == (100, 200)
clock.now += 0.002 # the window closed
sync_api._sample_rss()
assert (sync_api._status.rss_mb, sync_api._status.rss_peak_mb) == (101, 201)
sync_api._sample_rss(force=True) # forced, immediately → reads
assert (sync_api._status.rss_mb, sync_api._status.rss_peak_mb) == (102, 202)
sync_api._sample_rss() # …and the forced read re-armed the window
assert memory.reads == 3, "the forced sample closed the window it opened"
clock.now += sync_api._RSS_SAMPLE_INTERVAL_S
sync_api._sample_rss()
assert (sync_api._status.rss_mb, sync_api._status.rss_peak_mb) == (103, 203)
assert memory.reads == 4
def test_sample_rss_keeps_none_as_unknown(
fresh_sync_status: None, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A platform that cannot know (no ``/proc``, no ``resource``) stays
``None`` on the status — never 0, never a plausible number. The wire
form is JSON null and the UI renders no clause for it (the same
``?``-is-not-0 rule :func:`app.core.memory.fmt_mb` holds for the
log)."""
monkeypatch.setattr(sync_api, "time", _FakeClock())
_StubMemory(monkeypatch, [(None, None)])
sync_api._sample_rss(force=True)
assert sync_api._status.rss_mb is None
assert sync_api._status.rss_peak_mb is None
body = sync_api.sync_status()
assert body["rss_mb"] is None and body["rss_peak_mb"] is None
def test_sample_rss_never_publishes_a_peak_below_the_current_reading(
fresh_sync_status: None, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The pair's invariant, pinned at the ONE sampling point (task 04).
``VmRSS`` and ``ru_maxrss`` are two kernel counters read at two
instants, both floored to whole MB, and the high-water refreshes
LAZILY — so a growing run really does report ``rss_mb`` above
``rss_peak_mb`` (measured on this repo's own suite at 164 vs 162).
A "peak" under the process's own current footprint is a FALSE
number, not a conservative one, so the sampler lifts it to the
current reading: a lower bound on the truth, never an invention.
Without this the invariant every consumer (both lifecycle lines, the
status payload, the page's clause) now relies on would be a coin
flip — and the phase's own integration gate proved it."""
monkeypatch.setattr(sync_api, "time", _FakeClock())
_StubMemory(monkeypatch, [(412, 410)]) # the lazy-kernel case
sync_api._sample_rss(force=True)
assert sync_api._status.rss_mb == 412
assert sync_api._status.rss_peak_mb == 412, "the lagging peak is raised"
assert sync_api.sync_status()["rss_peak_mb"] == 412, "the wire agrees"
def test_sample_rss_does_not_fabricate_an_unknown_peak(
fresh_sync_status: None, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The normalization's edge: an UNKNOWABLE peak stays ``None``.
Lifting a lagging reading is a bound on a real measurement; inventing
a peak from the current reading on a platform where
:func:`app.core.memory.rss_peak_mb` answered ``None`` would be the
invented number the phase forbids (``null`` renders no clause, ``0``
is a value). So the clamp needs BOTH readings to be real."""
monkeypatch.setattr(sync_api, "time", _FakeClock())
_StubMemory(monkeypatch, [(412, None)])
sync_api._sample_rss(force=True)
assert sync_api._status.rss_mb == 412
assert sync_api._status.rss_peak_mb is None
def test_run_samples_at_start_phase_transitions_and_terminal(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
"""The sampling POINTS, walked through the real runner: run start →
``import`` → ``overview`` → ``summaries`` → the success terminal, one
forced read each, and the per-file progress hook's throttled call
adds NOTHING inside the window the run start just armed (LOCKED A4).
The last pair is the number the incident was about: the terminal
KEEPS it (the phase-64 keep-final-values convention) and the
``sync: done`` line logs the very same pair the payload reports —
one reading, two surfaces, no disagreement between the page and the
log an operator could be argued with."""
clock = _FakeClock(10_000.0)
monkeypatch.setattr(sync_api, "time", clock)
# run start, import, overview, summaries, done (the hook is throttled).
memory = _StubMemory(
monkeypatch,
[(10, 20), (11, 21), (12, 22), (13, 23), (14, 4_321)],
)
clone = _GatedClone(threading.Event(), threading.Event())
fake_import = _GatedImport(
ImportSummary(files=3, added=1, unchanged=2),
threading.Event(),
threading.Event(),
)
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
with caplog.at_level(logging.INFO, logger="app.api.sync"):
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
# The run start sampled BEFORE any pipeline step: a run that
# fails in the clone already has a number on the status.
s = sync_api.sync_status()
assert s["state"] == "running"
assert (s["rss_mb"], s["rss_peak_mb"]) == (10, 20)
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
# Import phase: the transition forced the second read, and
# the hook's own call (this fake fires one file) is inside
# the window — still 2 reads, not 3 (A4 in numbers).
s = sync_api.sync_status()
assert s["phase"] == "import"
assert fake_import.hook_calls == [("repo", "notes/deep.md", 1, 3)]
assert (s["rss_mb"], s["rss_peak_mb"]) == (11, 21)
assert memory.reads == 2, "the per-file hook never bypasses the guard"
fake_import.release.set()
finally:
clone.release.set()
fake_import.release.set()
thread.join(10.0)
assert errors == [], f"the runner raised: {errors!r}"
# The terminal pair is the FINAL reading, kept next to the counts,
# and it survives the run (the page shows it after the settle).
s = sync_api.sync_status()
assert s["state"] == "success"
assert (s["rss_mb"], s["rss_peak_mb"]) == (14, 4_321)
assert s["files_done"] == 1 and s["files_total"] == 3
done = [
r.getMessage()
for r in caplog.records
if r.getMessage().startswith("sync: done detail=")
]
assert len(done) == 1, done
assert done[0].endswith("rss_mb=14 rss_peak_mb=4321"), done[0]
def test_per_file_hook_samples_once_the_window_closes(
fresh_sync_status: None,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""The other half of D3: the number on the page MOVES. The hook
fires once per file, and once the 5 s window has closed the next
file's call re-reads — so a long walk reports the memory climbing
(PLAN §7.4's never-stale contract) at ~600 reads per 27k files
instead of 27k."""
clock = _FakeClock(10_000.0)
monkeypatch.setattr(sync_api, "time", clock)
memory = _StubMemory(monkeypatch, [(100, 200), (110, 210), (140, 260)])
clone = _GatedClone(threading.Event(), threading.Event())
fake_import = _GatedImport(
ImportSummary(files=1, added=1),
threading.Event(),
threading.Event(),
)
# The fake fires ONE file, parks, then returns — so the hook is
# called once mid-run and the test calls it a second time with the
# window closed (the second file of a real walk).
_patch_sync_seams(monkeypatch, tmp_path, fake_import, clone)
thread, errors = _start_run()
try:
assert clone.started.wait(5.0), "the run never reached the clone phase"
clone.release.set()
assert fake_import.started.wait(5.0), "the run never reached the import"
s = sync_api.sync_status()
assert s["phase"] == "import"
assert (s["rss_mb"], s["rss_peak_mb"]) == (110, 210) # the transition
assert memory.reads == 2, "the hook's call inside the window read nothing"
clock.now += sync_api._RSS_SAMPLE_INTERVAL_S + 1.0
sync_api._sample_rss() # the NEXT file's hook call
assert (sync_api._status.rss_mb, sync_api._status.rss_peak_mb) == (140, 260)
fake_import.release.set()
finally:
clone.release.set()
fake_import.release.set()
thread.join(10.0)
assert errors == [], f"the runner raised: {errors!r}"
assert memory.reads >= 3
# ---------- phase 187 (task 03): the peak clause on the result line ------
def test_sources_js_peak_clause_is_a_trailing_clause_with_a_null_guard() -> None:
"""The frontend SOURCE pin (the browser behavior is
tests/e2e/test_sync_peak_memory_visible.py, phase 187 task 04):
* ONE builder, reading ``status.rss_peak_mb ?? status.rss_mb`` (the
high-water mark, the current reading as the fall-back);
* a non-number (null / undefined / garbage) renders the EMPTY string
— never ``peak ? MB``, never ``peak null MB``;
* it is APPENDED to the two result strings that already exist
(``fmtSyncResult(...)``, ``"Sync cancelled"``) through the single
box writer — no new control, no new live region, no replaced
state string;
* NOTHING else reads it: the running label, the failed box (the
banner + modal own the error, byte-identical per phase 150) and
the upload settles never grow a clause;
* the shell markup is untouched — the box is still one text span."""
js = _text(SOURCES_JS)
builder = _body(js, "fmtSyncPeakClause")
assert "status.rss_peak_mb ?? status.rss_mb" in builder
assert 'return "";' in builder, "null renders NOTHING"
assert "` · peak ${peak} MB`" in builder, "the clause's exact shape"
success = _body(js, "applySyncSuccess")
assert (
"setSyncProgress(fmtSyncResult(status.detail) + fmtSyncPeakClause(status))"
in success
)
cancelled = _body(js, "applySyncCancelled")
assert 'setSyncProgress("Sync cancelled" + fmtSyncPeakClause(status))' in cancelled
for untouched in (
"enterSyncRunningState",
"fmtSyncPhaseLabel",
"fmtSyncLabel",
"fmtSyncResult",
"setSyncLabel",
"setSyncProgress",
"applySyncFailure",
"applySyncIdle",
):
assert "fmtSyncPeakClause" not in _body(js, untouched), (
f"{untouched} must not grow a clause — the terminal result "
"line is the only surface (D4)"
)
html = _text(SHELL_HTML)
assert '<span class="sync-progress-text" id="sync-progress-text"></span>' in html
assert 'id="sync-progress-peak"' not in html, "no new element for the clause"
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_sources_js_peak_clause_renders_under_node() -> None:
"""The REAL extracted ``fmtSyncResult`` + ``fmtSyncPeakClause`` under
node, composed exactly as ``applySyncSuccess`` composes them: the
counts line keeps its exact wording and the clause trails it
(``3 added · 2 unchanged · peak 1936 MB``); a null peak falls back to
the current reading; BOTH null (idle, a pre-phase-187 run, the
upload job, a platform that cannot know) render the counts line with
NO clause — never ``peak null MB``; ``0`` is a VALUE, not missing
(the ``fmt_mb(0) == "0"`` rule, mirrored); and a negative or
non-numeric value renders nothing."""
src = (
_body(_text(SOURCES_JS), "fmtSyncResult")
+ "\n"
+ _body(_text(SOURCES_JS), "fmtSyncPeakClause")
)
driver_expr = (
"const CASES = ["
"{ detail: { added: 3, unchanged: 2 }, rss_mb: 512, rss_peak_mb: 1936 },"
"{ detail: { added: 0, unchanged: 1 }, rss_mb: 512, rss_peak_mb: null },"
"{ detail: { added: 0, unchanged: 1 }, rss_mb: null, rss_peak_mb: null },"
"{ detail: { added: 1 } },"
"{ detail: { added: 2, pruned: 4 }, rss_mb: 0, rss_peak_mb: 0 },"
"{ detail: { added: 1 }, rss_peak_mb: -5 },"
"{ detail: { added: 1 }, rss_peak_mb: 'big' },"
"];\n"
"const LINES = CASES.map("
" (s) => fmtSyncResult(s.detail) + fmtSyncPeakClause(s)\n"
");\n"
"const CANCELLED = CASES.map("
" (s) => 'Sync cancelled' + fmtSyncPeakClause(s)\n"
");\n"
"console.log(JSON.stringify({ LINES, CANCELLED }));"
)
proc = subprocess.run(
["node", "-e", f"{src}\n{driver_expr}"], capture_output=True, text=True, timeout=60
)
assert proc.returncode == 0, f"node failed: {proc.stderr}"
out = json.loads(proc.stdout)
assert out["LINES"] == [
"3 added · 2 unchanged · peak 1936 MB",
"0 added · 1 unchanged · peak 512 MB", # the current-reading fallback
"0 added · 1 unchanged", # both null → no clause at all
"1 added", # idle / pre-phase / upload status: no keys, no clause
"2 added · 4 pruned · peak 0 MB", # 0 is a value, not missing
"1 added", # a negative reading is not a number to show
"1 added", # garbage is not a number either
], out
assert out["CANCELLED"] == [
"Sync cancelled · peak 1936 MB",
"Sync cancelled · peak 512 MB",
"Sync cancelled",
"Sync cancelled",
"Sync cancelled · peak 0 MB",
"Sync cancelled",
"Sync cancelled",
], out