brain-of-reese/tests/e2e/test_source_name_labels.py

"""Phase 191 E2E (Playwright, mock-only): source names, labels and tags.

Run in isolation (DB must be up: ``podman compose up -d db``):

    uv run pytest tests/e2e/test_source_name_labels.py -v --no-cov

The owner-visible arm of ``TODO.md`` L4 ("Sources need a name, a label,
tag(s) so you can focus the model when prompting if you have a project
name or a standard name you reference things by"). Tasks 01–05 pinned
the table, the API, the editor markup, the selection boost and the
prompt surfacing at their own layers; this suite drives the SAME feature
through the real UI and the real HTTP surface, and it is the only place
the two halves of the phase are observed TOGETHER — a label saved on the
Sources page changing the answer the chat model gets.

KB fixture — three sources, one per registry kind, all indexed by the
REAL in-process ``POST /api/sync`` over the deterministic mock (the
``test_name_signal_read_chips.py`` seeding pattern, extended to the
git/upload kinds):

* ``forgecore`` — a REAL git repository (``git init`` + one commit in a
  temp dir, registered as ``kind="git"`` with a ``file://`` URL the way
  the phase-135 suite adds one, and CLONED by the sync into this suite's
  own ``BOR_SOURCES_DIR``);
* ``homelab`` — a plain directory registered as ``kind="local"``;
* ``notesdrop`` — a ``.tar.gz`` posted to ``POST /api/git-sources/upload``
  (the phase-49 upload path, which registers the unpacked folder as a
  ``kind="local"`` row under ``BOR_UPLOAD_DIR``).

The bodies are token-controlled (the phase-119 fixture convention): the
same ``PHRASE`` repeated a strength gradient (×5/×4/×3/×2/×1) plus one
doc (``forgecore/notes/b6.md``, the TARGET) at the SAME single repeat but
a diluting long filler, so it is the LAST-ranked of the six — OUTSIDE the
top-5 suggested tier and the first entry of the rank-6+ related row. The
fixture pins that with the app's OWN retrieval at setup (``_assert_tiers``),
including the flip it exists for (the same question with the task-04
``source_name_hit`` flag set on the target's chunks, which is what saving
the name does on a question that uses it).

The tar members carry a FRESH mtime on purpose: the phase-106 recency
boost (code default 0.0007, ~1.5× the rank-5↔6 fused gap here) reads
``Document.created_at``, and a 1970-dated archive member measures as an
old document, loses that head start and re-orders the tail — the tier the
whole suite is pinned against.

Test → contract mapping (task 06 cases a–e; one Playwright file per
story, A16):

1. ``test_editor_saves_name_label_and_tags_for_every_kind`` — (a): the
   catalogue lists all three kinds by their DERIVED name; each row's
   editor saves Name (≤120) + Label (≤200) + Tags (the token input) with
   the per-row "saved" line, the row's "named" state tag comes up, and a
   reload re-populates the fields from the server — while the identity
   cell stays the read-only mono derived name (never an input).
2. ``test_control_question_leaves_the_target_in_the_related_row`` — (b)
   control: with NO label row the named question ranks the target 6th —
   its ``source/path`` is the "Nearby docs" row and NO ``<document>``
   block of the captured system prompt carries it.
3. ``test_naming_the_source_surfaces_its_document`` — (b) + (e-prompt):
   after saving the name/label/tags through the UI the SAME question
   ranks it inside the top five — the captured system prompt carries
   ``<document source="forgecore" name="the git server"
   label="self-hosted forge" path="notes/b6.md"`` (locked attribute
   order, task 05) and the target leaves the related row.
4. ``test_a_label_row_alone_never_moves_the_tier`` — (b) negative
   control: with the SAME row saved, a question that does not use the
   name holds the pre-phase tier byte-for-byte (D2 fires on the NAME in
   the question, not on the existence of a row).
5. ``test_rename_never_breaks_the_derived_identity`` — (c): with the row
   saved, ``source``/``path`` still read the document (API + the viewer
   page + the RAG catalogue), the ALIAS resolves to nothing (404 — it is
   not an identity), and the citation/nearby-doc link still targets
   ``forgecore/notes/b6.md``.
6. ``test_saving_a_label_of_a_vanished_source_shows_the_404_note`` — (d):
   a registered-then-removed source's row answers 404 and the editor
   says so in its own status line ("That source no longer exists —
   reload the page."), and the catalogue no longer lists it after a
   reload.
7. ``test_reset_reverts_the_named_question_to_the_control`` — (e): the
   row's Reset button clears the stored label (fields + "named" tag
   gone, "saved" confirmed, reload shows empty fields) and the SAME named
   question falls back to the pre-phase tier — the target out of the
   prompt and back in the related row.
"""
from __future__ import annotations

import io
import json
import os
import re
import subprocess
import sys
import tarfile
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any

import httpx
import pytest
from playwright.sync_api import Locator, Page, expect
from sqlalchemy import select, text

from app.config import Settings as _Settings
from app.db import SessionLocal
from app.models import QueryLog
from app.rag.retriever import retrieve, select_related, select_suggested
from app.rag.source_labels import all_labels, question_name_hits, set_label
from e2e.auth_helpers import login
from e2e.conftest import (
    ADMIN_PASSWORD,
    SESSION_SECRET,
    USE_REAL_LLM,
    _wait_http,
)
from e2e.mock_llm import embed_text

REPO = Path(__file__).resolve().parents[2]

# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module app binds its own port instead.
APP_PORT = int(os.environ.get("E2E_APP_PORT_SOURCE_LABELS", "8208"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"

SOURCES_PAGE = "/git-sources.html"
RAG_PAGE = "/sources.html"

#: The three registry kinds, each indexed with documents. The names are
#: the DERIVED ones the importer stores in ``Document.source`` — the git
#: repo's basename, the directory's basename, the archive's basename.
GIT_SOURCE = "forgecore"
LOCAL_SOURCE = "homelab"
UPLOAD_SOURCE = "notesdrop"
#: The git source's single document — the phase's TARGET: the document
#: a question has to NAME the source for.
TARGET_PATH = "notes/b6.md"
TARGET = f"{GIT_SOURCE}/{TARGET_PATH}"

#: The owner's alias, saved through the UI (the placeholder copy of the
#: editor is deliberately the same shape as this).
GIT_NAME = "the git server"
GIT_LABEL = "self-hosted forge"
GIT_TAGS = ["gitea", "nightops"]

#: The UI's own copy (the ONE source of truth lives in git-sources.js —
#: re-typing the strings here is the point: a reworded note must fail
#: this suite, not silently pass).
STATUS_SAVED = "saved"
STATUS_SAVING = "Saving…"
NOTE_GONE = "That source no longer exists — reload the page."

MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"

# --------------------------------------------------------------------------
# Fixture corpus (token-controlled — see the module docstring)
# --------------------------------------------------------------------------

#: 22 neutral tokens: every doc's digest + filler material, and NOT one
#: of the question's tokens (the filler only dilutes, never matches).
PRELUDE = (
    "fixture preamble block one two three four five six seven eight nine ten "
    "eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen"
)
PHRASE = "rotate the vault snapshot backups"
FILLER = "kilo one bravo two charlie three delta four echo five foxtrot six golf seven"
#: The target's diluting filler — the one doc-level difference between it
#: and the ×1 upload doc (same phrase count, lower cosine + lower ts_rank).
LONG_FILLER = FILLER + (
    " hotel two india three juliet four kilo five lima six mike seven "
    "november eight oscar nine papa ten quebec eleven romeo twelve sierra thirteen"
)

#: ``(path, phrase repeats, filler)`` per source. The paths are all
#: ``notes/<greek>.md`` on purpose: no path component equals a question
#: token, so the phase-119 PATH name-hit bonus fires for NOTHING and the
#: only signal that can move the target is the phase-191 SOURCE-NAME one.
LOCAL_DOCS: list[tuple[str, int, str]] = [
    ("notes/a1.md", 5, FILLER),
    ("notes/a2.md", 4, FILLER),
    ("notes/a3.md", 3, FILLER),
    ("notes/a4.md", 2, FILLER),
]
UPLOAD_DOCS: list[tuple[str, int, str]] = [("notes/b5.md", 1, FILLER)]
GIT_DOCS: list[tuple[str, int, str]] = [("notes/b6.md", 1, LONG_FILLER)]
ALL_DOCS: list[tuple[str, str]] = [
    *[(LOCAL_SOURCE, path) for path, _n, _f in LOCAL_DOCS],
    *[(UPLOAD_SOURCE, path) for path, _n, _f in UPLOAD_DOCS],
    *[(GIT_SOURCE, path) for path, _n, _f in GIT_DOCS],
]


def _body(path: str, strength: int, filler: str) -> str:
    """One fixture doc: a title, the neutral prelude, ``strength`` copies
    of PHRASE, the filler. The prelude is long enough (22 tokens) that
    the mock's 24-token summary digest never reaches the phrase — the
    summary chunks stay at the bottom of both candidate lists."""
    lines = [f"# Fixture note {Path(path).stem}", "", PRELUDE, ""]
    lines += [PHRASE] * strength
    lines += ["", filler]
    return "\n".join(lines) + "\n"


#: The questions. ``CONTROL_QUESTION`` carries no alias token; the NAMED
#: one adds EXACTLY the two words of the display name that appear in NO
#: document ("git", "server" — the D2 hit token is ``server``, the ≥4-char
#: one), so the two lists are the same list and the ONLY thing that can
#: move the target is the source-name bonus.
CONTROL_QUESTION = "How often should I rotate the vault snapshot backups?"
NAMED_QUESTION = "How often should I rotate the vault snapshot backups on the git server?"

#: The engineered invariant, MEASURED on this fixture (module docstring):
#: the target is rank 6 of six — out of the top-5 suggested, first of the
#: related row — for BOTH questions while no label row exists.
CONTROL_SUGGESTED = [
    f"{LOCAL_SOURCE}/notes/a1.md",
    f"{LOCAL_SOURCE}/notes/a2.md",
    f"{LOCAL_SOURCE}/notes/a3.md",
    f"{LOCAL_SOURCE}/notes/a4.md",
    f"{UPLOAD_SOURCE}/notes/b5.md",
]
CONTROL_RELATED = [TARGET]
#: …and inside the tier (first, ahead of the strongest doc) once the
#: source is named — the +0.005 bonus is ~10× the rank-5↔6 fused gap, so
#: the target takes the head of the walk and the upload doc falls to the
#: related row.
NAMED_SUGGESTED = [TARGET, *CONTROL_SUGGESTED[:4]]
NAMED_RELATED = [f"{UPLOAD_SOURCE}/notes/b5.md"]

#: The throwaway git row of the 404 arm — a URL the app validates but
#: this suite NEVER syncs (a row with no documents at all: it exists in
#: the catalogue only while its row does, which is exactly the arm).
GONE_URL = "https://git.example.invalid/ops/throwaway.git"
GONE_SOURCE = "throwaway"


# --------------------------------------------------------------------------
# Fixtures
# --------------------------------------------------------------------------


@pytest.fixture(scope="module")
def workspace(tmp_path_factory: pytest.TempPathFactory) -> dict[str, Path]:
    """The suite's scratch tree: the app's clone/upload/image dirs (so a
    run never touches ``~/bor-sources``) plus the local source directory
    and the git repository origin, both written from the constants above."""
    root = tmp_path_factory.mktemp("bor191")
    ws = {
        "root": root,
        "checkouts": root / "checkouts",
        "uploads": root / "uploads",
        "images": root / "images",
        "local": root / LOCAL_SOURCE,
        "origin": root / "origin" / GIT_SOURCE,
    }
    for key in ("checkouts", "uploads", "images"):
        ws[key].mkdir(parents=True, exist_ok=True)
    for base, docs in ((ws["local"], LOCAL_DOCS), (ws["origin"], GIT_DOCS)):
        for path, strength, filler in docs:
            target = base / path
            target.parent.mkdir(parents=True, exist_ok=True)
            target.write_text(_body(path, strength, filler), encoding="utf-8")
    return ws


def _git(cwd: Path, *args: str) -> None:
    subprocess.run(
        ["git", *args],
        cwd=cwd,
        check=True,
        capture_output=True,
        text=True,
    )


@pytest.fixture(scope="module")
def git_repo(workspace: dict[str, Path]) -> Path:
    """One committed repository (the phase-135 fixture-repo shape: the
    sync clones it over ``file://`` — a real ``subprocess``, off the event
    loop, into this suite's own ``BOR_SOURCES_DIR``)."""
    repo = workspace["origin"]
    _git(repo, "init", "-q", "-b", "main")
    _git(repo, "config", "user.email", "e2e@example.invalid")
    _git(repo, "config", "user.name", "Phase 191 E2E")
    _git(repo, "add", "-A")
    _git(repo, "commit", "-q", "-m", "phase 191 fixture corpus")
    assert (repo / ".git").is_dir()
    return repo


@pytest.fixture(scope="module")
def app_server(
    mock_llm: int, workspace: dict[str, Path]
) -> Iterator[str]:
    """The real app under test — per-module env (the conftest pattern):
    the mock LLM, the mock-calibrated honesty gate, this suite's own
    clone/upload/image dirs, the empty ``BOR_GIT_SOURCES`` fallback (the
    registry must hold EXACTLY the three sources this suite registers),
    and the leak-guarded code defaults for every retrieval knob the
    pinned tiers depend on (``suggested_docs`` 5, ``related_max_docs`` 2,
    ``source_name_hit_bonus`` 0.005 — the boost under test rides the
    PRODUCTION default, never an operator's ``.env``)."""
    env = dict(os.environ)
    env.pop("DEBUGPY", None)
    env["BOR_ENVIRONMENT"] = "e2e"
    env["BOR_STATIC_DIR"] = str(REPO / "frontend")
    env["BOR_LLM_BASE_URL"] = (
        "https://aipi.reeseapps.com/v1"
        if USE_REAL_LLM
        else f"http://127.0.0.1:{mock_llm}/v1"
    )
    env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
    env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
    env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
    env["BOR_LLM_RETRY_DELAY"] = "0"
    env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
    for key in (
        "suggested_docs",
        "related_max_docs",
        "source_name_hit_bonus",
        "name_hit_bonus",
        "recency_boost",
        "retrieval_ef_search",
    ):
        env[f"BOR_{key.upper()}"] = str(_Settings.model_fields[key].default)
    env.setdefault(
        "BOR_DATABASE_URL",
        "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
    )
    env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
    env["BOR_SESSION_SECRET"] = SESSION_SECRET
    env["BOR_GIT_SOURCES"] = ""
    env["BOR_DOCS_REPO"] = ""
    env["BOR_SUGGESTIONS"] = json.dumps(
        _Settings.model_fields["suggestions"].default
    )
    env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
    env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
    env["BOR_SOURCES_DIR"] = str(workspace["checkouts"])
    env["BOR_UPLOAD_DIR"] = str(workspace["uploads"])
    env["BOR_IMAGE_DIR"] = str(workspace["images"])
    proc = subprocess.Popen(
        [
            sys.executable, "-m", "uvicorn", "app.main:app",
            "--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning",
        ],
        cwd=REPO,
        env=env,
    )
    try:
        _wait_http(f"{APP_URL}/api/health")
        yield APP_URL
    finally:
        proc.terminate()
        try:
            proc.wait(timeout=10)
        except subprocess.TimeoutExpired:
            proc.kill()


@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
    return app_server


def _truncate_all() -> None:
    """Fresh registry + KB + labels (the E2E isolation pattern — the
    suites share one Postgres and the tiers are pinned against EXACTLY
    these six documents)."""
    with SessionLocal() as db:
        db.execute(
            text(
                "TRUNCATE chunks, documents, query_log, steering_notes, "
                "kb_overview, git_sources, folder_summaries, source_labels"
            )
        )
        db.commit()


def _wait_terminal(
    client: httpx.Client, path: str, timeout_s: float = 180.0
) -> dict[str, Any]:
    """Poll a ``SyncStatus``-shaped endpoint until the run is terminal."""
    deadline = time.monotonic() + timeout_s
    body: dict[str, Any] = {}
    while time.monotonic() < deadline:
        r = client.get(path)
        assert r.status_code == 200, r.text
        body = r.json()
        if body["state"] in ("success", "failed"):
            return body
        time.sleep(0.2)
    raise AssertionError(f"{path} never reached a terminal state: {body}")


def _selection(question: str, named_source: str | None = None) -> tuple[list[str], list[str]]:
    """The app's OWN selection walk for *question* — the task-04 flag
    applied exactly the way ``app/api/chat.py`` applies it (the pure D2
    rule over the turn's ONE ``all_labels`` read) when *named_source* is
    given. The fixture's lens on the tier the UI then shows."""
    with SessionLocal() as db:
        if named_source is not None:
            set_label(db, named_source, display_name=GIT_NAME, label=GIT_LABEL, tags=GIT_TAGS)
            db.commit()
        labels = all_labels(db)
        hits = {v.source for v in labels.values() if question_name_hits(v, question)}
        chunks = retrieve(db, question, _embed(question))
        for rc in chunks:
            if rc.document.source in hits:
                rc.source_name_hit = True
        bonus = _Settings.model_fields["source_name_hit_bonus"].default
        suggested = select_suggested(chunks, source_bonus=bonus)
        related = select_related(
            chunks,
            {d.id for d in suggested},
            _Settings.model_fields["related_max_docs"].default,
            source_bonus=bonus,
        )
    if named_source is not None:
        with SessionLocal() as db:
            db.execute(text("DELETE FROM source_labels WHERE source = :s"), {"s": named_source})
            db.commit()
    return (
        [f"{d.source}/{d.path}" for d in suggested],
        [f"{d.source}/{d.path}" for d in related],
    )


def _embed(question: str) -> list[float]:
    """The mock's token-overlap embedding (the same function the mock
    server computes — imported, not re-implemented)."""
    return embed_text(question)


def _assert_tiers() -> None:
    """Pin the fixture's engineered invariant with the app's real
    retrieval: the target is rank 6 (out of the seeded tier, first of the
    related row) for BOTH questions, and rank 1 once its source is named.
    A fixture-text regression fails HERE, at setup, with the measured
    tiers in the message — not at the wire assertions."""
    for question in (CONTROL_QUESTION, NAMED_QUESTION):
        suggested, related = _selection(question)
        assert suggested == CONTROL_SUGGESTED, (
            f"unlabelled suggested tier drifted for {question!r}: {suggested}"
        )
        assert related == CONTROL_RELATED, (
            f"unlabelled related tier drifted for {question!r}: {related}"
        )
    suggested, related = _selection(NAMED_QUESTION, named_source=GIT_SOURCE)
    assert suggested == NAMED_SUGGESTED, (
        f"the source-name bonus did not surface the target: {suggested}"
    )
    assert related == NAMED_RELATED, f"related tier after the boost: {related}"
    # …and the bonus is ONLY a name trigger: the control question with
    # the same row saved holds the pre-phase tier.
    suggested, related = _selection(CONTROL_QUESTION, named_source=GIT_SOURCE)
    assert suggested == CONTROL_SUGGESTED, (
        f"a label row moved the tier without the name: {suggested}"
    )
    assert related == CONTROL_RELATED, related


@pytest.fixture(scope="module")
def seeded_kb(app_server: str, workspace: dict[str, Path], git_repo: Path) -> None:
    """The story's precondition: three sources of the three kinds, all
    indexed by the REAL sync over the mock. Registers the local directory
    and the ``file://`` repository through ``POST /api/git-sources``,
    posts the tar.gz through the phase-49 upload route, then runs
    ``POST /api/sync`` (clone → walk → chunk → embed → summaries →
    overview → folder summaries) and pins the imported count before any
    tier is asserted."""
    _truncate_all()
    buf = io.BytesIO()
    with tarfile.open(fileobj=buf, mode="w:gz") as tf:
        for path, strength, filler in UPLOAD_DOCS:
            data = _body(path, strength, filler).encode("utf-8")
            info = tarfile.TarInfo(path)
            info.size = len(data)
            # Fresh, on purpose (module docstring): the recency boost
            # reads created_at, and the epoch would re-order the tail.
            info.mtime = int(time.time())
            tf.addfile(info, io.BytesIO(data))

    with httpx.Client(base_url=app_server, timeout=60.0) as client:
        assert client.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
        r = client.post("/api/git-sources", json={"kind": "local", "path": str(workspace["local"])})
        assert r.status_code == 201, r.text
        r = client.post("/api/git-sources", json={"kind": "git", "url": f"file://{git_repo}"})
        assert r.status_code == 201, r.text
        r = client.post(
            "/api/git-sources/upload",
            files={"file": (f"{UPLOAD_SOURCE}.tar.gz", buf.getvalue(), "application/octet-stream")},
        )
        assert r.status_code == 202, r.text
        body = _wait_terminal(client, "/api/git-sources/upload/status", 60.0)
        assert body["state"] == "success", body
        assert client.post("/api/sync").status_code == 202
        body = _wait_terminal(client, "/api/sync/status")
    assert body["state"] == "success", body
    assert body["detail"]["added"] == len(ALL_DOCS), body["detail"]
    assert body["detail"]["pruned"] == 0, body["detail"]
    # The three kinds really are three sources — the catalogue's union
    # (task 02) lists exactly them, by DERIVED name.
    with httpx.Client(base_url=app_server, timeout=30.0) as client:
        assert client.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
        listing = client.get("/api/source-labels")
        assert listing.status_code == 200, listing.text
        assert [e["source"] for e in listing.json()["labels"]] == [
            GIT_SOURCE,
            LOCAL_SOURCE,
            UPLOAD_SOURCE,
        ], listing.json()
    _assert_tiers()


@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
    """Per-test isolation of the VOLATILE state only: the KB and the
    registry are module-scoped (nothing a test does changes them), while
    every test starts from NO label row at all (the pre-phase state) and
    a clean ``query_log``."""
    with SessionLocal() as db:
        db.execute(text("TRUNCATE source_labels, query_log"))
        db.commit()
    yield
    with SessionLocal() as db:
        db.execute(text("TRUNCATE source_labels, query_log"))
        db.commit()


# --------------------------------------------------------------------------
# Page / wire helpers (the house patterns)
# --------------------------------------------------------------------------

#: Captures the raw SSE ``data:`` payloads of /api/chat (a response clone
#: read in the background) — wire-level assertions on the ``done`` frame,
#: independent of the UI rendering.
SSE_HOOK = """
() => {
  if (window.__sseInstalled) return;
  window.__sseInstalled = true;
  window.__sseFrames = [];
  const origFetch = window.fetch;
  window.fetch = async function (...args) {
    const res = await origFetch.apply(this, args);
    try {
      const url = typeof args[0] === 'string' ? args[0] : args[0].url;
      if (url.includes('/api/chat')) {
        res.clone().text().then((bodyText) => {
          for (const block of bodyText.split('\\n\\n')) {
            const line = block.trim();
            if (line.startsWith('data: ')) {
              window.__sseFrames.push(line.slice(6));
            }
          }
        });
      }
    } catch (e) { /* non-clonable responses: ignored */ }
    return res;
  };
}
"""


def _reset_capture(mock_llm: int) -> None:
    r = httpx.post(f"http://127.0.0.1:{mock_llm}/v1/e2e/captured/reset", timeout=10)
    assert r.status_code == 200, r.text


def _captured_prompts(mock_llm: int) -> list[str]:
    """The system prompts of the chat requests of the LAST turn (the
    mock's ring buffer — the wire-level lens on the EXACT prompt the app
    built; a grounded zero-read turn sends exactly one)."""
    bodies = httpx.get(
        f"http://127.0.0.1:{mock_llm}/v1/e2e/captured", timeout=10
    ).json()
    return [
        msg["content"]
        for body in bodies
        for msg in body.get("messages", [])
        if msg.get("role") == "system" and isinstance(msg.get("content"), str)
    ]


def _frames(page: Page) -> list[dict]:
    deadline = time.monotonic() + 30.0
    while True:
        raw = page.evaluate("() => window.__sseFrames || []")
        parsed = [json.loads(line) for line in raw if line]
        if any(f.get("type") == "done" for f in parsed):
            return parsed
        if time.monotonic() > deadline:
            raise AssertionError(f"SSE hook captured no `done` frame: {len(parsed)} frames")
        time.sleep(0.05)


def _ask(page: Page, question: str, mock_llm: int) -> dict[str, Any]:
    """One grounded chat turn, end to end: reset the mock's capture, send
    the question through the composer, wait for the settle, return the
    ``done`` frame. The captured prompt of the turn travels with it."""
    _reset_capture(mock_llm)
    page.evaluate("window.__sseFrames = []")
    page.fill("#message-input", question)
    page.click("#send-btn")
    expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
    expect(page.locator(".msg.brain .bubble").last).to_contain_text(
        MOCK_ANSWER_MARKER, timeout=30_000
    )
    expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
    done = [f for f in _frames(page) if f.get("type") == "done"][-1]
    assert done.get("deflected") is False, done
    prompts = _captured_prompts(mock_llm)
    assert len(prompts) == 1, f"expected one chat call, got {len(prompts)}"
    return {"done": done, "prompt": prompts[0]}


def _target_in_prompt(prompt: str) -> bool:
    """Does the seeded ``<documents>`` block carry the target document?
    (The attribute the phase-118 seed puts in front of the model — the
    tier the boost moves, observed on the wire.) Matched on the ONE
    ``<document>`` tag, so a source attribute and an unrelated path can
    never combine into a false positive."""
    return (
        re.search(
            rf'<document source="{re.escape(GIT_SOURCE)}"[^>]*path="{re.escape(TARGET_PATH)}"',
            prompt,
        )
        is not None
    )


def _related_paths(page: Page) -> list[str]:
    """The LAST bubble's de-emphasized "Nearby docs" row (rank 6+) — the
    owner-visible half of the tier (each link's text AND aria-label are
    the full ``source/path``)."""
    links = page.locator(".msg.brain").last.locator(".related-doc")
    return [links.nth(i).inner_text() for i in range(links.count())]


def _sources_page(page: Page, app_url: str, rows: int = 3) -> None:
    login(page, app_url, next=SOURCES_PAGE)
    expect(page.locator("#source-labels-tbody tr")).to_have_count(rows, timeout=30_000)


def _label_row(page: Page, source: str) -> Locator:
    row = page.locator("#source-labels-tbody tr").filter(
        has=page.locator("code.source-label-source", has_text=re.compile(rf"^{re.escape(source)}$"))
    )
    return row


def _open_editor(page: Page, source: str) -> Locator:
    row = _label_row(page, source)
    expect(row).to_have_count(1)
    editor = row.locator(".source-label-editor")
    if editor.is_hidden():
        row.locator(".source-label-toggle").click()
    expect(editor).to_be_visible()
    return row


def _fill_label(row: Locator, name: str, label: str, tags: list[str]) -> None:
    row.locator(".source-label-name-input").fill(name)
    row.locator(".source-label-label-input").fill(label)
    tags_input = row.locator(".source-label-tags-input")
    for i, tag in enumerate(tags, start=1):
        tags_input.fill(tag)
        tags_input.press("Enter")
        # The chip paints on the keypress (no round-trip), and the input
        # clears — the token-input contract of the phase-89 idiom.
        expect(row.locator(".source-label-tags li")).to_have_count(i)
        expect(tags_input).to_have_value("")
    expect(row.locator(".source-label-save")).to_be_enabled()


def _save(row: Locator) -> None:
    row.locator(".source-label-save").click()


def _status(row: Locator) -> Locator:
    return row.locator(".source-label-status")


def _last_query_log() -> QueryLog:
    with SessionLocal() as db:
        rows = db.scalars(select(QueryLog)).all()
    assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}"
    return rows[0]


# --------------------------------------------------------------------------
# 1. (a) the editor saves name + label + tags for every source kind, and
#    a reload re-populates them
# --------------------------------------------------------------------------


def test_editor_saves_name_label_and_tags_for_every_kind(
    page: Page, app_url: str, seeded_kb: None, db_ready: None
) -> None:
    """The Names/labels/tags card for a git, a local AND an uploaded
    source: save all three fields per row (the token input's chips), the
    per-row "saved" line lands, the row's "named" state tag comes up, and
    after a reload the fields re-populate from the server — while every
    row's identity cell is still the read-only mono DERIVED name (an
    input there would BE the alias-contract bug)."""
    page.set_default_timeout(30_000)
    _sources_page(page, app_url)

    # The catalogue's rows are the DERIVED names, sorted, one per kind.
    codes = page.locator("#source-labels-tbody code.source-label-source")
    expect(codes).to_have_count(3)
    for i, source in enumerate([GIT_SOURCE, LOCAL_SOURCE, UPLOAD_SOURCE]):
        expect(codes.nth(i)).to_have_text(source)
    # Nothing is stored yet: no "named" tag anywhere, no Reset anywhere.
    expect(page.locator("#source-labels-tbody .source-label-state:visible")).to_have_count(0)

    for i, source in enumerate([GIT_SOURCE, LOCAL_SOURCE, UPLOAD_SOURCE]):
        row = _open_editor(page, source)
        name = f"the {source} shelf"
        label = f"notes synced from {source}"
        tags = ["e2e", f"shelf{i}", f"kind{i}"]
        _fill_label(row, name, label, tags)
        expect(_status(row)).to_have_text("unsaved changes")
        _save(row)
        expect(_status(row)).to_have_text(STATUS_SAVING)
        expect(_status(row)).to_have_text(STATUS_SAVED, timeout=15_000)
        expect(row.locator(".source-label-state")).to_be_visible()
        expect(row.locator(".source-label-state")).to_have_text("named")
        # The chips are real elements carrying the typed text.
        chips = row.locator(".source-label-tag")
        expect(chips).to_have_count(3)
        expect(chips.nth(2)).to_have_text(f"kind{i}")
        # The identity cell never moves with the name (D1).
        expect(row.locator("code.source-label-source")).to_have_text(source)

    # Reload — the rows re-read the server, so the editor must show the
    # stored values, not the typed draft.
    page.reload()
    expect(page.locator("#source-labels-tbody .source-label-state:visible")).to_have_count(
        3, timeout=30_000
    )
    for i, source in enumerate([GIT_SOURCE, LOCAL_SOURCE, UPLOAD_SOURCE]):
        row = _open_editor(page, source)
        expect(row.locator(".source-label-name-input")).to_have_value(f"the {source} shelf")
        expect(row.locator(".source-label-label-input")).to_have_value(
            f"notes synced from {source}"
        )
        expect(row.locator(".source-label-tag")).to_have_count(3)
        expect(row.locator(".source-label-tag").nth(1)).to_have_text(f"shelf{i}")
        # A re-populated row is not dirty: Save stays disabled until an
        # edit, and Reset exists because something IS stored.
        expect(row.locator(".source-label-save")).to_be_disabled()
        expect(row.locator(".source-label-reset")).to_be_visible()

    with httpx.Client(base_url=app_url, timeout=30.0) as client:
        assert client.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
        body = client.get("/api/source-labels").json()["labels"]
        assert [e["source"] for e in body] == [GIT_SOURCE, LOCAL_SOURCE, UPLOAD_SOURCE]
        assert body[0]["display_name"] == f"the {GIT_SOURCE} shelf"
        assert body[0]["tags"] == ["e2e", "shelf0", "kind0"]


# --------------------------------------------------------------------------
# 2. (b) control — no label row: the question that WOULD name the source
#    leaves its document in the related row, out of the seeded prompt
# --------------------------------------------------------------------------


def test_control_question_leaves_the_target_in_the_related_row(
    page: Page, app_url: str, seeded_kb: None, db_ready: None, mock_llm: int
) -> None:
    """The pre-phase half of the owner-visible arm: with no ``source_labels``
    row the named question ranks the target 6th — the answer bubble's
    "Nearby docs" row carries ``forgecore/notes/b6.md`` and the captured
    system prompt carries NO ``<document>`` block for it. The turn is
    still grounded (the honesty gate is untouched by this phase)."""
    page.set_default_timeout(30_000)
    login(page, app_url, next="/")
    page.evaluate(SSE_HOOK)

    turn = _ask(page, NAMED_QUESTION, mock_llm)
    assert not _target_in_prompt(turn["prompt"]), turn["prompt"]
    assert _related_paths(page) == [TARGET]
    # The nearby-doc link is the DERIVED identity (the alias is nowhere
    # on the bubble — a link built from the display name would be the
    # D1 bug, visible right here).
    nearby = page.locator(".msg.brain").last.locator(".related-doc")
    expect(nearby).to_have_count(1)
    expect(nearby.first).to_have_attribute(
        "href", f"/document.html?source={GIT_SOURCE}&path=notes%2Fb6.md&back=%2F"
    )
    expect(nearby.first).to_have_text(TARGET)
    # The durable record agrees with the wire.
    record = _last_query_log()
    assert record.sources == ", ".join(CONTROL_SUGGESTED + CONTROL_RELATED)


# --------------------------------------------------------------------------
# 3. (b) the named question, WITH the row saved through the UI — the
#    target enters the seeded tier and the wire carries name + label
# --------------------------------------------------------------------------


def test_naming_the_source_surfaces_its_document(
    page: Page, app_url: str, seeded_kb: None, db_ready: None, mock_llm: int
) -> None:
    """The flip, driven from the Sources page: save Name "the git server" +
    Label "self-hosted forge" + the tags on the git source, ask the SAME
    question again — and the captured prompt carries the target's block
    with the additive ``name``/``label`` attributes in the locked order
    (``source`` → ``name?`` → ``label?`` → ``path`` → ``title`` →
    ``date``), while the target leaves the "Nearby docs" row. A saved
    name changes what the MODEL is told; it changes nothing about the
    document's identity (the block still names ``source``/``path``)."""
    page.set_default_timeout(30_000)
    _sources_page(page, app_url)
    row = _open_editor(page, GIT_SOURCE)
    _fill_label(row, GIT_NAME, GIT_LABEL, GIT_TAGS)
    _save(row)
    expect(_status(row)).to_have_text(STATUS_SAVED, timeout=15_000)

    page.goto(app_url + "/")
    expect(page.locator("#message-input")).to_be_visible(timeout=30_000)
    page.evaluate(SSE_HOOK)
    turn = _ask(page, NAMED_QUESTION, mock_llm)

    expected_head = (
        f'<document source="{GIT_SOURCE}" name="{GIT_NAME}" label="{GIT_LABEL}" '
        f'path="{TARGET_PATH}"'
    )
    assert expected_head in turn["prompt"], turn["prompt"]
    assert _target_in_prompt(turn["prompt"])
    # It left the related tier (the same document never appears twice).
    assert _related_paths(page) == NAMED_RELATED
    record = _last_query_log()
    assert record.sources == ", ".join(NAMED_SUGGESTED + NAMED_RELATED)
    # What the MODEL is told changed; what the OWNER is linked to did
    # not: the nearby-doc row is still a DERIVED source/path document
    # and the alias appears nowhere in the bubble but nowhere else
    # either (it is prompt-only surfacing, D4).
    nearby = page.locator(".msg.brain").last.locator(".related-doc")
    expect(nearby).to_have_count(1)
    expect(nearby.first).to_have_attribute(
        "href", "/document.html?source=notesdrop&path=notes%2Fb5.md&back=%2F"
    )
    expect(nearby.first).to_have_text(f"{UPLOAD_SOURCE}/notes/b5.md")
    # A zero-read turn chips nothing (phase 119 LOCKED A1) — and the
    # alias is prompt-only surfacing, never a citation label.
    expect(page.locator(".msg.brain .source-chip")).to_have_count(0)


# --------------------------------------------------------------------------
# 4. (b) negative control — the row alone moves nothing
# --------------------------------------------------------------------------


def test_a_label_row_alone_never_moves_the_tier(
    page: Page, app_url: str, seeded_kb: None, db_ready: None, mock_llm: int
) -> None:
    """D2 on the wire: the SAME saved row, a question that does NOT use
    the name → the pre-phase tier exactly (the target out of the prompt,
    back in the related row). The boost is a NAME trigger, not a pin."""
    page.set_default_timeout(30_000)
    _sources_page(page, app_url)
    row = _open_editor(page, GIT_SOURCE)
    _fill_label(row, GIT_NAME, GIT_LABEL, GIT_TAGS)
    _save(row)
    expect(_status(row)).to_have_text(STATUS_SAVED, timeout=15_000)

    page.goto(app_url + "/")
    expect(page.locator("#message-input")).to_be_visible(timeout=30_000)
    page.evaluate(SSE_HOOK)
    turn = _ask(page, CONTROL_QUESTION, mock_llm)
    assert not _target_in_prompt(turn["prompt"]), turn["prompt"]
    assert _related_paths(page) == CONTROL_RELATED


# --------------------------------------------------------------------------
# 5. (c) the alias never breaks the derived source/path identity
# --------------------------------------------------------------------------


def test_rename_never_breaks_the_derived_identity(
    page: Page, app_url: str, seeded_kb: None, db_ready: None
) -> None:
    """With the row saved (name + label + tags), every identity surface
    still speaks DERIVED: the content API reads the document by
    ``source``/``path`` and its payload names no alias, the same API with
    the DISPLAY NAME as the source is a plain 404 (the alias is not an
    identity), the viewer page renders the stored text under the derived
    name, and the RAG catalogue lists ``forgecore`` — never "the git
    server"."""
    page.set_default_timeout(30_000)
    _sources_page(page, app_url)
    row = _open_editor(page, GIT_SOURCE)
    _fill_label(row, GIT_NAME, GIT_LABEL, GIT_TAGS)
    _save(row)
    expect(_status(row)).to_have_text(STATUS_SAVED, timeout=15_000)

    # --- the locator the agent's read/grep/show_text tools use -------
    r = page.request.get(
        f"{app_url}/api/documents/content?source={GIT_SOURCE}&path={TARGET_PATH}"
    )
    assert r.status == 200, r.text
    payload = r.json()
    assert payload["source"] == GIT_SOURCE and payload["path"] == TARGET_PATH
    assert "Fixture note b6" in payload["content"]
    assert GIT_NAME not in json.dumps(payload) and GIT_LABEL not in json.dumps(payload)

    # --- the ALIAS is not an identity ---------------------------------
    alias = page.request.get(
        f"{app_url}/api/documents/content?source={GIT_NAME}&path={TARGET_PATH}"
    )
    assert alias.status == 404, alias.text

    # --- the viewer page (the citation's escape hatch) ----------------
    page.goto(f"{app_url}/document.html?source={GIT_SOURCE}&path={TARGET_PATH}&back=%2F")
    body = page.locator("body")
    expect(body).to_contain_text("Fixture note b6", timeout=30_000)
    expect(body).to_contain_text(GIT_SOURCE)

    # --- the KB tree (the RAG page) still lists the derived name ------
    login(page, app_url, next=RAG_PAGE)
    expect(page.locator("#folders-tbody tr")).to_have_count(3, timeout=30_000)
    links = page.locator("#folders-tbody a.folder-link")
    texts = [links.nth(i).inner_text() for i in range(links.count())]
    # The tree lists the sources in REGISTRY order (the phase-167 rule —
    # the local directory was registered first, then the repository, then
    # the upload), always by DERIVED name: no display name, anywhere.
    assert texts == [LOCAL_SOURCE, GIT_SOURCE, UPLOAD_SOURCE], texts
    # Drill the repository: the level the owner is ON is the source, the
    # files sit one level below it (the phase-167 tree).
    page.click(f'#folders-tbody a.folder-link:text-is("{GIT_SOURCE}")')
    expect(page.locator("#kb-crumb .kb-crumb-current")).to_have_text(GIT_SOURCE)
    expect(page.locator("#folders-tbody a.folder-link")).to_have_count(1)
    expect(page.locator("#docs-tbody tr")).to_have_count(0)
    page.click('#folders-tbody a.folder-link:text-is("notes")')
    expect(page.locator("#docs-tbody tr")).to_have_count(1, timeout=30_000)
    doc_row = page.locator("#docs-tbody tr").first
    # The row's identity cells: DERIVED source, DERIVED path (the link's
    # hover/accessible name) — and no alias anywhere on the level.
    expect(doc_row.locator("td").first).to_have_text(GIT_SOURCE)
    expect(doc_row.locator("a.doc-link")).to_have_attribute("title", TARGET_PATH)
    expect(doc_row.locator("a.doc-link")).to_have_attribute(
        "href", f"/document.html?source={GIT_SOURCE}&path=notes%2Fb6.md"
    )
    # The shell ships the phrase "the git server" as the editor's own
    # placeholder copy, so "no alias in the tree" is scoped to the tree —
    # the breadcrumb, the folder rows and the file rows.
    expect(page.locator("#kb-crumb")).not_to_contain_text(GIT_NAME)
    expect(page.locator("#folders-tbody")).not_to_contain_text(GIT_NAME)
    expect(page.locator("#docs-tbody")).not_to_contain_text(GIT_NAME)


# --------------------------------------------------------------------------
# 6. (d) the 404 arm, visible
# --------------------------------------------------------------------------


def test_saving_a_label_of_a_vanished_source_shows_the_404_note(
    page: Page, app_url: str, seeded_kb: None, db_ready: None
) -> None:
    """A source removed elsewhere while its editor is open: the PUT is
    the framework's 404 and the row's own status line NAMES that arm
    ("That source no longer exists — reload the page.") instead of a
    generic retry note; after a reload the catalogue no longer lists the
    row at all, and the API refuses the orphan write."""
    page.set_default_timeout(30_000)
    with httpx.Client(base_url=app_url, timeout=30.0) as client:
        assert client.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
        created = client.post("/api/git-sources", json={"kind": "git", "url": GONE_URL})
        assert created.status_code == 201, created.text
        row_id = created.json()["id"]

    _sources_page(page, app_url, rows=4)
    row = _open_editor(page, GONE_SOURCE)
    row.locator(".source-label-name-input").fill("the throwaway shelf")
    expect(_status(row)).to_have_text("unsaved changes")

    # The source goes away while the editor is open (removed elsewhere).
    with httpx.Client(base_url=app_url, timeout=30.0) as client:
        assert client.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
        assert client.delete(f"/api/git-sources/{row_id}").status_code == 204

    _save(row)
    expect(_status(row)).to_have_text(NOTE_GONE, timeout=15_000)
    expect(_status(row)).to_have_class(re.compile(r"is-error"))
    # The draft is NOT silently dropped: the row still shows what the
    # owner typed, so the note is the only claim about the save.
    expect(row.locator(".source-label-name-input")).to_have_value("the throwaway shelf")
    expect(row.locator(".source-label-state")).to_be_hidden()
    # And the note is not painted over by the dirty hint when the request
    # settles — the row is still dirty, and the note is still what the
    # owner reads. Only an edit brings the hint back.
    expect(_status(row)).to_have_text(NOTE_GONE)
    row.locator(".source-label-label-input").fill("edited after the note")
    expect(_status(row)).to_have_text("unsaved changes")

    page.reload()
    expect(page.locator("#source-labels-tbody tr")).to_have_count(3, timeout=30_000)
    expect(page.locator(f'#source-labels-tbody code:text-is("{GONE_SOURCE}")')).to_have_count(0)
    with httpx.Client(base_url=app_url, timeout=30.0) as client:
        assert client.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
        orphan = client.put(
            f"/api/source-labels/{GONE_SOURCE}",
            json={"display_name": "the throwaway shelf", "label": None, "tags": []},
        )
        assert orphan.status_code == 404, orphan.text


# --------------------------------------------------------------------------
# 7. (e) Reset reverts — the fields AND the owner-visible tier
# --------------------------------------------------------------------------


def test_reset_reverts_the_named_question_to_the_control(
    page: Page, app_url: str, seeded_kb: None, db_ready: None, mock_llm: int
) -> None:
    """The revert half of the story: name the source (the flip, asserted
    on the wire), press the row's Reset — the fields clear, the "named"
    tag and the Reset button go away, the row confirms "saved", a reload
    shows empty fields — and the SAME named question falls back to the
    pre-phase tier: the target out of the seeded prompt, back in the
    "Nearby docs" row."""
    page.set_default_timeout(30_000)
    _sources_page(page, app_url)
    row = _open_editor(page, GIT_SOURCE)
    _fill_label(row, GIT_NAME, GIT_LABEL, GIT_TAGS)
    _save(row)
    expect(_status(row)).to_have_text(STATUS_SAVED, timeout=15_000)

    page.goto(app_url + "/")
    expect(page.locator("#message-input")).to_be_visible(timeout=30_000)
    page.evaluate(SSE_HOOK)
    turn = _ask(page, NAMED_QUESTION, mock_llm)
    assert _target_in_prompt(turn["prompt"]), "the flip must be up before Reset"

    page.goto(app_url + SOURCES_PAGE)
    row = _open_editor(page, GIT_SOURCE)
    expect(row.locator(".source-label-reset")).to_be_visible()
    row.locator(".source-label-reset").click()
    expect(_status(row)).to_have_text(STATUS_SAVED, timeout=15_000)
    expect(row.locator(".source-label-name-input")).to_have_value("")
    expect(row.locator(".source-label-label-input")).to_have_value("")
    expect(row.locator(".source-label-tag")).to_have_count(0)
    expect(row.locator(".source-label-state")).to_be_hidden()
    expect(row.locator(".source-label-reset")).to_be_hidden()

    page.reload()
    row = _open_editor(page, GIT_SOURCE)
    expect(row.locator(".source-label-name-input")).to_have_value("")
    expect(row.locator(".source-label-save")).to_be_disabled()
    with httpx.Client(base_url=app_url, timeout=30.0) as client:
        assert client.post("/api/login", json={"password": ADMIN_PASSWORD}).status_code == 204
        entry = next(
            e
            for e in client.get("/api/source-labels").json()["labels"]
            if e["source"] == GIT_SOURCE
        )
        assert entry["display_name"] is None and entry["tags"] == []

    page.goto(app_url + "/")
    expect(page.locator("#message-input")).to_be_visible(timeout=30_000)
    page.evaluate(SSE_HOOK)
    turn = _ask(page, NAMED_QUESTION, mock_llm)
    assert not _target_in_prompt(turn["prompt"]), turn["prompt"]
    assert _related_paths(page) == CONTROL_RELATED