"""Phase 96 task 04 E2E (Playwright, mock-only): the one-shot LLM resilience — the 2026-09-11 incident shape, retried and healed. The dedicated story suite for ``96_oneshot_resilience`` (A16 — one Playwright file per phase, run in isolation): a folder whose FIRST one-shot summary reply arrives in the exact incident shape (``content=""`` + ``finish_reason="length"``) still ends up with its stored summary (the task-01 retry recovered it, visible in the ``ls`` drill-down), a folder whose replies are ALWAYS empty stays absent without failing the sync (the task-01 exhaustion + the phase-94 per-folder fail-soft contract), and a row deleted behind the app's back is self-healed by the next UNCHANGED sync with the other rows untouched (tasks 02/03 targeted fill). Run in isolation (DB must be up: ``podman compose up -d db``): uv run pytest tests/e2e/test_oneshot_llm_retry.py -v --no-cov MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the deterministic incident-shape injection in ``tests/e2e/mock_llm.py`` (phase 96, task 04): a NON-stream ``chat/completions`` request whose system prompt carries ``FOLDER_SUMMARY_MODE`` and whose ``Folder: …`` label ends with ``/e2e_empty_once`` answers the incident envelope (``content=""`` + ``finish_reason="length"`` — the mock's normal OpenAI shape) on its FIRST non-stream POST only, and a label ending with ``/e2e_empty_always`` answers it on EVERY non-stream POST. The per-label counter resets after the success it guards (the phase-67 ``_fail_posts`` pattern), so a re-run of the suite is green without manual state cleanup. The trigger strings are this suite's own folder names, so no other E2E can hit them. The e2e app server boots with ``BOR_LLM_RETRY_DELAY=0`` (the conftest pattern) so the 4-attempt exhaustion paths are instant; ``BOR_LLM_RETRIES`` is forced to the code default (3 — the conftest leak-guard pattern), so ``e2e_empty_always`` costs exactly 4 POSTs per sync that attempts it and the suite stays fast. KB fixture — a host temp dir tree (``tmp_path_factory``; the app runs on the same host) with ONE registered local source (the ``test_local_directory_sources.py`` registration + real-Sync pattern; no git anywhere): ``oneshot/`` with three ≥ 2-doc folders — ``e2e_empty_once/`` (2 docs), ``e2e_empty_always/`` (2), ``normal/`` (2). Every fixture doc carries the words ``drill down the tree`` in its body, so the scripted ``ls`` drill-down questions (the phase-94 ``DRILL_TRIGGER`` echo pattern — the mock echoes the received tool result into its grounded answer, the E2E's only lens on the LLM's context) FTS-match at least one chunk and run grounded. Test → phase mapping (Playwright Mapping Rule): 1. ``test_incident_reply_retried_and_always_empty_stays_absent`` — after the changed sync #1 (the module fixture pins the stored rows: the ``e2e_empty_once`` row EXISTS — the retry recovered it, without task 01 it would be absent — and the ``e2e_empty_always`` row does NOT — all 4 attempts empty → ``LLMError`` → per-folder fail-soft), a scripted ``ls oneshot`` turn asserts the drill-down listing: the ``e2e_empty_once/`` line carries ``: Fixture folder summary for oneshot/e2e_empty_once.`` (the retry recovered the row), the ``normal/`` line carries its summary, and the ``e2e_empty_always/`` line is ``e2e_empty_always/ — 2 documents`` with NO ``: …`` suffix — while the sync reported success (a folder-summary exhaustion never flips the run, phase 94). 2. ``test_deleted_row_self_heals_on_unchanged_sync`` — the ``normal`` stored row is deleted directly (simulating a historical failure), ``e2e_empty_once``'s row ``updated_at`` is captured, and the UNCHANGED sync #2 runs the gap gate (task 03): a second scripted ``ls oneshot`` turn shows ``normal/`` healed (``: Fixture folder summary for oneshot/normal.`` again) and ``e2e_empty_always/`` still absent (the gap-fill attempted it, exhausted, stayed absent — the sync still succeeded), and the DB pins the targeted fill: the ``normal`` row is back with its deterministic text and the ``e2e_empty_once`` row's ``updated_at`` is UNCHANGED (a full regeneration would have re-stamped it). """ from __future__ import annotations import json import os import subprocess import sys import time from collections.abc import Iterator from datetime import datetime from pathlib import Path from typing import Any import httpx import pytest from playwright.sync_api import Locator, Page, expect from sqlalchemy import select, text from app.config import Settings as _Settings from app.db import SessionLocal from app.models import FolderSummary from e2e.auth_helpers import login from e2e.conftest import ( ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http, ) REPO = Path(__file__).resolve().parents[2] # Phase 79 (task 04, full inventory): the conftest session app owns its # port in a combined run — this module app binds its own port instead # (a same-port second uvicorn dies on bind and would drive the wrong # server). Env-overridable. APP_PORT = int(os.environ.get("E2E_APP_PORT_ONESHOT", "8138")) APP_URL = f"http://127.0.0.1:{APP_PORT}" # -------------------------------------------------------------------------- # Fixture documents + the pinned drill-down listing (deterministic) # -------------------------------------------------------------------------- #: The local source — the temp directory's basename (``kind=local`` → #: the directory's basename is the source name, phase 38). SOURCE = "oneshot" #: The three ≥ 2-doc folders (the mock's trigger labels are the folder #: NAMES — the seeded KB carries them, the house marker-flow #: convention). Path order (the ``ls`` subfolder order): #: e2e_empty_always < e2e_empty_once < normal. FOLDER_ONCE = "e2e_empty_once" FOLDER_ALWAYS = "e2e_empty_always" FOLDER_NORMAL = "normal" TOTAL_DOCS = 6 # three folders x 2 docs each #: Every fixture body carries ``drill down the tree`` (the #: ``DRILL_TRIGGER`` phrase's words): every scripted question #: FTS-matches at least one chunk → HIGH gate → the ```` section #: the drill-down flow keys on (the phase-94 seed convention). DRILL_LEAD = "The drill down the tree fixture note" def _md(title: str, body: str) -> str: return f"# {title}\n\n{body}\n" #: The mock's byte-stable ``FOLDER_SUMMARY_MODE`` lines for this #: fixture (the phase-94 template — the label is the #: ``FOLDER_HEADER_PREFIX`` tail: ```` for the root, #: ``/`` for a folder). SUM_ROOT = f"Fixture folder summary for {SOURCE}." SUM_ONCE = f"Fixture folder summary for {SOURCE}/{FOLDER_ONCE}." SUM_ALWAYS = f"Fixture folder summary for {SOURCE}/{FOLDER_ALWAYS}." SUM_NORMAL = f"Fixture folder summary for {SOURCE}/{FOLDER_NORMAL}." # --- the pinned ``ls oneshot`` level (app.rag.agent's phase-94 template) #: The source root: 0 direct files, 3 subfolders (path order). LS_HEADER = f"{SOURCE} — 0 documents, 3 folders:" #: The subfolder lines — the ``: {summary}`` suffix appended ONLY when #: the subfolder's summary is stored (``render_folder_listing``; the #: renderer's 2-space indent is whitespace-normalized away by the #: ``to_contain_text`` match — the phase-94 pin convention). LINE_ALWAYS = f"{FOLDER_ALWAYS}/ — 2 documents" LINE_ALWAYS_WITH_COLON = f"{FOLDER_ALWAYS}/ — 2 documents: " LINE_ONCE = f"{FOLDER_ONCE}/ — 2 documents: {SUM_ONCE}" LINE_NORMAL = f"{FOLDER_NORMAL}/ — 2 documents: {SUM_NORMAL}" # --- the scripted drill-down turns (the phase-94 ``DRILL_TRIGGER`` # questions — the mock echoes the listing verbatim into the answer) --- LS_QUESTION_1 = f"Drill down the tree: ls {SOURCE} — what's in source {SOURCE}?" LS_QUESTION_2 = f"Drill down the tree: ls {SOURCE} — list the source folders again" # -------------------------------------------------------------------------- # Fixtures # -------------------------------------------------------------------------- @pytest.fixture(scope="module") def oneshot_dir(tmp_path_factory: pytest.TempPathFactory) -> Path: """The one-source temp tree (see the module docstring): the app server runs on the same host, so the paths are visible to it.""" root = tmp_path_factory.mktemp("bor_oneshot") src = root / SOURCE for folder, prefix in ( (FOLDER_ONCE, "once"), (FOLDER_ALWAYS, "always"), (FOLDER_NORMAL, "normal"), ): (src / folder).mkdir(parents=True) for letter, topic in (("a", "A"), ("b", "B")): (src / folder / f"{prefix}-{letter}.md").write_text( _md( f"Oneshot {prefix.title()} {letter.upper()}", f"{DRILL_LEAD} for {SOURCE} {folder} " f"{letter}: this document covers topic {topic} " f"of the {SOURCE} source tree.", ), encoding="utf-8", ) assert len(list(src.rglob("*.md"))) == TOTAL_DOCS return src @pytest.fixture(scope="module") def app_server(mock_llm: int, oneshot_dir: Path) -> Iterator[str]: """The real app under test — per-module app (the conftest pattern, cf. ``test_local_directory_sources.py``): NO ``BOR_GIT_SOURCES`` (the env fallback is git-only — the source here is a DB-registered local directory), the mock LLM, the mock-calibrated threshold, and the leak-guarded code defaults. ``BOR_LLM_RETRY_DELAY=0`` + the code-default ``BOR_LLM_RETRIES`` (the phase-67 conftest pattern): the one-shot retry waits are instant and the exhaustion budget is the REAL one (4 attempts). The session app is never started in this isolated run, so no port clash.""" env = dict(os.environ) env.pop("DEBUGPY", None) env["BOR_ENVIRONMENT"] = "e2e" env["BOR_STATIC_DIR"] = str(REPO / "frontend") env["BOR_LLM_BASE_URL"] = ( "https://aipi.reeseapps.com/v1" if USE_REAL_LLM else f"http://127.0.0.1:{mock_llm}/v1" ) # Mock-calibrated threshold (conftest pattern): every scripted # question FTS-matches the fixture docs (the ``drill down the # tree`` words), so the gate is HIGH either way. env["BOR_RELEVANCE_THRESHOLD"] = "0.30" # Phase 67 / phase 96: instant retry waits + the code-default budget # (the conftest leak-guard pattern) — the one-shot retry and the # 4-attempt exhaustion run in real time at zero delay. env["BOR_LLM_RETRY_DELAY"] = "0" env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default) env.setdefault( "BOR_DATABASE_URL", "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese", ) # Phase 16: admin auth must be set or create_app() refuses to boot. env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD env["BOR_SESSION_SECRET"] = SESSION_SECRET # The repo's .env file carries the owner's BOR_GIT_SOURCES (the app # reads it from cwd) — override it with an EMPTY value (the env var # beats the .env file): the registry must hold EXACTLY the one local # directory this suite registers. env["BOR_GIT_SOURCES"] = "" # Leak guards (conftest pattern): an operator's local (gitignored) # .env cannot leak corpus-specific settings into the app under test. env["BOR_DOCS_REPO"] = "" env["BOR_SUGGESTIONS"] = json.dumps( _Settings.model_fields["suggestions"].default ) env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default proc = subprocess.Popen( [sys.executable, "-m", "uvicorn", "app.main:app", "--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"], cwd=REPO, env=env, ) try: _wait_http(f"{APP_URL}/api/health") yield APP_URL finally: proc.terminate() try: proc.wait(timeout=10) except subprocess.TimeoutExpired: proc.kill() @pytest.fixture(scope="module") def app_url(app_server: str) -> str: return app_server def _truncate_all() -> None: """Fresh registry + KB (the E2E isolation pattern): the E2E suites share one Postgres, so a leftover source or document would pollute the ``ls`` listing the drill answers assert on byte-exactly.""" with SessionLocal() as db: db.execute( text( "TRUNCATE chunks, documents, query_log, steering_notes, " "kb_overview, git_sources, folder_summaries" ) ) db.commit() def _run_sync_http(base_url: str, timeout_s: float = 180.0) -> dict[str, Any]: """Login + ``POST /api/sync`` + poll the status endpoint until the run reaches a terminal state (the ``test_sync_button.py`` / ``test_local_directory_sources.py`` pattern, over plain httpx).""" with httpx.Client(base_url=base_url, timeout=30.0) as client: r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) assert r.status_code == 204, r.text r = client.post("/api/sync") assert r.status_code == 202, r.text deadline = time.monotonic() + timeout_s body: dict[str, Any] = {} while time.monotonic() < deadline: r = client.get("/api/sync/status") assert r.status_code == 200, r.text body = r.json() if body["state"] in ("success", "failed"): return body time.sleep(0.5) raise AssertionError(f"sync did not reach a terminal state: {body}") def _folder_rows() -> dict[str, tuple[str, datetime]]: """The source's stored folder summaries ``{folder_path: (summary, updated_at)}`` (``""`` = the source root) — the test process's direct DB access, the E2E's other established lens.""" with SessionLocal() as db: rows = db.execute( select( FolderSummary.folder_path, FolderSummary.summary, FolderSummary.updated_at, ).where(FolderSummary.source == SOURCE) ).all() return {folder: (summary, updated_at) for folder, summary, updated_at in rows} @pytest.fixture(scope="module") def synced_kb(app_server: str, oneshot_dir: Path) -> None: """The story's precondition: the KB synced under the deterministic mock's incident-shape injection. Registers the temp directory through the authenticated API (the ``test_local_directory_sources.py`` pattern) and runs the REAL in-process sync #1 (``POST /api/sync`` — walk → chunk → embed → overview → folder summaries → version bump). The sync changed the KB → FULL folder regeneration, and the mock's injection drives the incident shapes: * ``oneshot`` (the root) + ``oneshot/normal`` — normal replies, one POST each; * ``oneshot/e2e_empty_once`` — the FIRST non-stream POST answers the incident envelope (``content=""`` + ``finish_reason= "length"``), the retry (task 01) recovers the row on the second POST; * ``oneshot/e2e_empty_always`` — EVERY non-stream POST answers the empty envelope: 1 + ``BOR_LLM_RETRIES`` = 4 attempts exhausted → ``LLMError`` → the phase-94 per-folder fail-soft leaves the row ABSENT and never flips the run. The fixture pins all of that in the DB: the ``e2e_empty_once`` row EXISTS (without task 01 it would be absent — the incident's data loss), the ``e2e_empty_always`` row does NOT, and the sync still reported ``success``. """ _truncate_all() with httpx.Client(base_url=app_server, timeout=30.0) as client: r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) assert r.status_code == 204, r.text r = client.post( "/api/git-sources", json={"kind": "local", "path": str(oneshot_dir)} ) assert r.status_code == 201, r.text body = _run_sync_http(app_server) assert body["state"] == "success", body detail = body["detail"] assert detail["added"] == TOTAL_DOCS, detail assert detail["updated"] == 0, detail assert detail["pruned"] == 0, detail assert detail["overview"] is True, detail # The full regeneration under the injection: the retried row # EXISTS (the incident, healed by task 01's one-shot retry), the # exhausted row is ABSENT (the per-folder fail-soft — the sync # above still reported success), the normal rows landed. rows = _folder_rows() assert set(rows) == {"", FOLDER_ONCE, FOLDER_NORMAL}, rows assert rows[""][0] == SUM_ROOT, rows assert rows[FOLDER_ONCE][0] == SUM_ONCE, rows assert rows[FOLDER_NORMAL][0] == SUM_NORMAL, rows assert FOLDER_ALWAYS not in rows, rows # all 4 attempts empty → no row @pytest.fixture(autouse=True) def _clean(db_ready: None) -> Iterator[None]: """Per-test query_log isolation (the KB itself is module-scoped — the drill turns never change it, so the folder summaries and the registry persist across the tests of this module).""" with SessionLocal() as db: db.execute(text("TRUNCATE query_log")) db.commit() yield with SessionLocal() as db: db.execute(text("TRUNCATE query_log")) db.commit() # -------------------------------------------------------------------------- # Page helpers (the phase-94 drill-down house pattern) # -------------------------------------------------------------------------- #: Captures the raw SSE ``data:`` payloads of the /api/chat stream #: (a response clone read in the background) — wire-level assertions #: for the ``tool`` frames, independent of the UI rendering. SSE_HOOK = """ () => { if (window.__sseInstalled) return; window.__sseInstalled = true; window.__sseFrames = []; const origFetch = window.fetch; window.fetch = async function (...args) { const res = await origFetch.apply(this, args); try { const url = typeof args[0] === 'string' ? args[0] : args[0].url; if (url.includes('/api/chat')) { res.clone().text().then((bodyText) => { for (const block of bodyText.split('\\n\\n')) { const line = block.trim(); if (line.startsWith('data: ')) { window.__sseFrames.push(line.slice(6)); } } }); } } catch (e) { /* non-clonable responses: ignored */ } return res; }; } """ def _install_page_hooks(page: Page) -> None: page.evaluate(SSE_HOOK) def _frames(page: Page) -> list[dict]: """The SSE frames captured since the last submit (``_submit`` clears the buffer), once the hook's background read settles.""" deadline = time.monotonic() + 30.0 while True: raw = page.evaluate("() => window.__sseFrames || []") parsed = [json.loads(line) for line in raw if line] if any(f.get("type") == "done" for f in parsed): return parsed if time.monotonic() > deadline: raise AssertionError( f"SSE hook captured no `done` frame (frames so far: " f"{len(parsed)}) — hook install failed?" ) time.sleep(0.05) def _tool_frames(frames: list[dict]) -> list[dict]: return [f for f in frames if f.get("type") == "tool"] def _submit(page: Page, question: str) -> None: page.evaluate("window.__sseFrames = []") page.fill("#message-input", question) page.click("#send-btn") # The user bubble lands synchronously with the submit handler. expect(page.locator(".msg.user .bubble").last).to_contain_text(question) def _wait_settled(page: Page) -> None: """The turn is complete: answer text in the bubble, button recovered (the phase-48 settle wait).""" expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=60_000) expect(page.locator("#send-btn")).to_be_enabled(timeout=60_000) expect(page.locator("#send-label")).to_have_text("Send", timeout=60_000) def _last_brain(page: Page) -> Locator: return page.locator(".msg.brain").last def _assert_drill_turn( page: Page, expected_tool: dict[str, Any], expected_lines: list[str], forbidden: list[str] | None = None, ) -> None: """One scripted drill turn, fully asserted: the wire carries exactly the expected ``tool`` frame (ahead of the first ``delta``), the bubble carries the expected listing lines (the mock's echo of the tool result the model received), the forbidden substrings are ABSENT (the no-suffix assertions), and the turn was grounded (the ``done`` frame is not deflected).""" frames = _frames(page) assert _tool_frames(frames) == [expected_tool], _tool_frames(frames) first_delta = next( i for i, f in enumerate(frames) if f.get("type") == "delta" ) assert all( i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool" ) done = next(f for f in frames if f.get("type") == "done") assert done["deflected"] is False, done bubble = _last_brain(page).locator(".bubble") for line in expected_lines: expect(bubble).to_contain_text(line) text = bubble.text_content() or "" for needle in forbidden or []: assert needle not in text, text # -------------------------------------------------------------------------- # 1. Sync #1: the incident shape is retried (the row lands) and the # always-empty folder stays absent with the sync green # -------------------------------------------------------------------------- def test_incident_reply_retried_and_always_empty_stays_absent( page: Page, app_url: str, synced_kb: None, db_ready: None ) -> None: """The ``ls oneshot`` drill-down shows the stored folder summaries the sync #1 produced under the incident-shape injection: the ``e2e_empty_once/`` line carries its summary (the retry recovered the row — without task 01 the line would have NO ``: …`` suffix), the ``normal/`` line does too, and the ``e2e_empty_always/`` line is bare (exhausted → fail-soft → absent) — while the sync above reported success (the folder-stats failure never flips the run, phase 94).""" page.set_default_timeout(30_000) login(page, app_url, next="/") _install_page_hooks(page) _submit(page, LS_QUESTION_1) _wait_settled(page) lines = _last_brain(page).locator(".tool-call") expect(lines).to_have_count(1) expect(lines.nth(0)).to_contain_text(f"Listing documents in {SOURCE}") _assert_drill_turn( page, {"type": "tool", "name": "ls", "argument": SOURCE}, [ LS_HEADER, # The bare line (the row is absent — no stored summary to # append) … LINE_ALWAYS, # …and the colon-suffixed lines (the rows the retry + the # normal path stored): LINE_ONCE, LINE_NORMAL, ], # The ``e2e_empty_always/`` line must NOT carry a ``: …`` # suffix — the bare line above is a prefix of the suffixed # shape, so the absence is pinned here (the all-4-attempts # exhaustion left no row to quote). forbidden=[LINE_ALWAYS_WITH_COLON], ) # -------------------------------------------------------------------------- # 2. The gap-fill: a deleted row self-heals on the next UNCHANGED sync, # the other rows untouched # -------------------------------------------------------------------------- def test_deleted_row_self_heals_on_unchanged_sync( page: Page, app_url: str, synced_kb: None, db_ready: None ) -> None: """Delete the ``normal`` stored row behind the app's back (a historical failure), run the UNCHANGED sync #2, and assert the task-02/03 gap gate: ``missing_folder_summaries`` names the gap, ``only_missing=True`` fills EXACTLY the missing rows, and every other row stays byte-identical (text AND ``updated_at``). * ``normal`` — healed: the second ``ls oneshot`` turn carries ``: Fixture folder summary for oneshot/normal.`` again, and the DB row is back with its deterministic text; * ``e2e_empty_once`` — the gap-fill NEVER calls it (the row exists): its ``updated_at`` is UNCHANGED (a full regeneration would have re-stamped it); * ``e2e_empty_always`` — the gap-fill attempted it, exhausted (4 empty POSTs by design), stayed ABSENT — and the sync still succeeded (the fail-soft never flips the run).""" page.set_default_timeout(30_000) # The gap: delete the ``normal`` row directly (the test process has # DB access via the conftest engine — the same connection the app # uses), capturing the other row's stamp for the targeted-fill # assertion. No KB change anywhere. before = _folder_rows() assert FOLDER_NORMAL in before # the row existed (sync #1 stored it) updated_at_once = before[FOLDER_ONCE][1] with SessionLocal() as db: db.execute( text( "DELETE FROM folder_summaries " "WHERE source = :s AND folder_path = :f" ), {"s": SOURCE, "f": FOLDER_NORMAL}, ) db.commit() assert FOLDER_NORMAL not in _folder_rows() # Sync #2 — the KB is UNCHANGED (nothing in the temp tree moved), # so the gate takes the gap probe: two candidates missing # (``e2e_empty_always`` + ``normal``) → the targeted fill. body = _run_sync_http(app_url) assert body["state"] == "success", body # exhaustion never flips the run detail = body["detail"] assert detail["added"] == 0, detail assert detail["updated"] == 0, detail assert detail["pruned"] == 0, detail assert detail["overview"] is False, detail # unchanged → no overview burn # The second scripted drill turn: the healed + the surviving lines, # the always folder still bare. login(page, app_url, next="/") _install_page_hooks(page) _submit(page, LS_QUESTION_2) _wait_settled(page) lines = _last_brain(page).locator(".tool-call") expect(lines).to_have_count(1) expect(lines.nth(0)).to_contain_text(f"Listing documents in {SOURCE}") _assert_drill_turn( page, {"type": "tool", "name": "ls", "argument": SOURCE}, [ LS_HEADER, LINE_ALWAYS, LINE_ONCE, LINE_NORMAL, # healed — the suffix is back ], forbidden=[LINE_ALWAYS_WITH_COLON], ) # The DB pins the targeted fill (the E2E's other established lens): rows = _folder_rows() assert set(rows) == {"", FOLDER_ONCE, FOLDER_NORMAL}, rows assert rows[FOLDER_NORMAL][0] == SUM_NORMAL, rows # deterministic text # The targeted fill never touched the other rows — a FULL # regeneration would have re-stamped this one (the ``_upsert`` # fresh-UTC-stamp rule). assert rows[FOLDER_ONCE][1] == updated_at_once, ( rows[FOLDER_ONCE][1], updated_at_once, ) assert FOLDER_ALWAYS not in rows, rows # exhausted again → still absent