phase: 119_name_signal_read_chips
All verification complete. Final report: **Phase 119 final verification pass — all criteria verified, one stale pin fixed.** - Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry. - Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged. - New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2. - Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed). - Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met). - Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
@@ -0,0 +1,794 @@
|
||||
"""Phase 119 E2E (Playwright, mock-only): the name-signal seeds the
|
||||
right document — and chips cite READ docs only.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_name_signal_read_chips.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: the story's gates are the deterministic contracts —
|
||||
the mock's scripted-turn lenses (the tail echo, the single-read tool
|
||||
flow, the new ``FOLDER_MAP_TRIGGER`` echo) make "what reached the
|
||||
prompt / what got cited" assertable byte-exactly.
|
||||
|
||||
KB fixture — ``tests/fixtures/namekb`` (tracked; registered as a
|
||||
local-directory source through the authenticated API + the REAL
|
||||
in-process ``POST /api/sync``, the ``test_summary_seed_context.py``
|
||||
pattern), seven markdown documents in one source (``namekb``) whose
|
||||
bodies are token-controlled so the hybrid gate + the phase-119
|
||||
name-hit bonus pick the intended tiers deterministically
|
||||
(``synced_kb`` pins it with the app's REAL retrieval — a
|
||||
fixture-text regression that breaks the design fails at setup with a
|
||||
clear message, not at the wire assertions):
|
||||
|
||||
* every path carries a DIGITLESS question name token (the four
|
||||
guides' stems hold ``install``, the two forgejo docs' stems +
|
||||
folder hold ``forgejo``, ``zz-folder-map.md`` holds ``folder``) —
|
||||
so the name-hit bonus applies to ALL seven docs (a uniform
|
||||
``+0.005``) and the tier order reduces to the pure retrieval
|
||||
signal (the bonus head-start is exercised, the ordering is not
|
||||
distorted by it);
|
||||
* the four guides' bodies repeat the question phrase a strength
|
||||
gradient (×4/×3/×2/×1) so they OUT-RANK the name-hit docs on
|
||||
vector/FTS; the forgejo docs and the filler doc carry NEUTRAL
|
||||
bodies (their name is their main signal — the phase-119 owner
|
||||
scenario: "the file's name is the distinctive part");
|
||||
* engineered invariant (asserted in ``synced_kb`` for ALL FOUR
|
||||
scripted questions, measured + probe-verified, stable across
|
||||
re-imports): the name-hit doc ``forgejo/forgejo-home.md`` is the
|
||||
LAST of the top-5 suggested (the name signal lifts it into the
|
||||
tier, the weak body keeps it at the back — its ``Source:`` tail
|
||||
line is the tail echo), and the related tier is the rank-6+
|
||||
remainder (``forgejo-nginx`` + ``zz-folder-map``; for the folder
|
||||
question the two swap order — both stay rank 6+).
|
||||
|
||||
Test → contract mapping (task 06 cases a–d; one Playwright file per
|
||||
story, A16):
|
||||
1. ``test_name_hit_doc_is_last_suggested`` — (a): the distinctive
|
||||
question + ``show the end of your notes`` ⇒ the answer quotes the
|
||||
last 160 chars of the seeded ``<documents>`` block — the
|
||||
name-hit doc's SUMMARY tail (its ``Source: namekb/forgejo/
|
||||
forgejo-home.md`` pointer line; no other doc's pointer line, no
|
||||
doc's tail sentinel — the summary seed, not the full text). The
|
||||
zero-read grounded turn chips NOTHING (LOCKED A1); the related
|
||||
row renders rank 6+; the durable record carries suggested +
|
||||
related (118-A3 untouched).
|
||||
2. ``test_single_read_chips_only_the_read_doc`` — (b): the
|
||||
``use your tools`` flow (the mock's single read: ``ls`` → drill
|
||||
``ls(namekb)`` → read the FIRST file line —
|
||||
``namekb/zz-folder-map.md``, the root file) ⇒ the bubble carries
|
||||
EXACTLY ONE ``.source-chip`` — the read doc (LOCKED A1); none of
|
||||
the five suggested docs chips; the de-emphasized related row
|
||||
renders (rank 6+ deduped against the cited read doc — the read
|
||||
doc is a chip, never a "nearby doc"); the durable record carries
|
||||
suggested + related + read (118-A3 untouched).
|
||||
3. ``test_zero_read_grounded_turn_chips_nothing`` — (c): a PLAIN
|
||||
distinctive question (no tool trigger) ⇒ the ``done`` frame's
|
||||
``sources == []`` and ZERO ``.source-chip`` elements in the
|
||||
bubble — LOCKED A1's visible consequence; the answer still
|
||||
renders grounded (deflected: false); the related row renders.
|
||||
4. ``test_folder_lines_echoed_verbatim`` — (d): the distinctive
|
||||
question + ``repeat your folder map`` (the new mock trigger) ⇒
|
||||
the answer ends with the suggested-folder lines VERBATIM — the
|
||||
name-hit doc's folder line ``namekb/forgejo/: forgejo-nginx.md``
|
||||
(the line prefix + its known sibling entry) alongside the
|
||||
guides' folder line.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Locator, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document, QueryLog
|
||||
from app.rag.retriever import TRUNCATION_MARKER, retrieve, select_related, select_suggested
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
from e2e.mock_llm import TOKEN_RE, embed_text
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||
# port in a combined run — this module app binds its own port instead
|
||||
# (a same-port second uvicorn dies on bind and would drive the wrong
|
||||
# server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_NAMESIG", "8145"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
SOURCE = "namekb" # the local directory's basename = the source name
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "namekb"
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixture documents (tracked — tests/fixtures/namekb; deterministic,
|
||||
# token-controlled — see the module docstring for the design)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: 22 neutral tokens (no question tokens) — the digest + filler
|
||||
#: material of the neutral docs; the guides' strength phrase is
|
||||
#: question-token-only.
|
||||
PRELUDE = (
|
||||
"fixture preamble block one two three four five six seven eight nine ten "
|
||||
"eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen"
|
||||
)
|
||||
assert len(TOKEN_RE.findall(PRELUDE.lower())) == 22
|
||||
PHRASE = "install forgejo with the deployment steps"
|
||||
|
||||
#: (path, title, phrase repeats, filler line, tail sentinel) — the
|
||||
#: guides carry the strength gradient (×4/×3/×2/×1); the forgejo docs
|
||||
#: (name hits — distinctive digitless ``forgejo`` component) and the
|
||||
#: root filler carry NEUTRAL bodies (0 repeats).
|
||||
DOCS: list[tuple[str, str, int, str, str]] = [
|
||||
("deploy/install-guide-a.md", "Install Forgejo Guide A", 4,
|
||||
"alpha1 alpha2 alpha3 alpha4 alpha5 alpha6", "NAMEKB-TAIL-c5d6"),
|
||||
("deploy/install-guide-b.md", "Install Forgejo Guide B", 3,
|
||||
"beta1 beta2 beta3 beta4 beta5 beta6", "NAMEKB-TAIL-d7e8"),
|
||||
("deploy/install-guide-c.md", "Install Forgejo Guide C", 2,
|
||||
"gamma1 gamma2 gamma3 gamma4 gamma5 gamma6", "NAMEKB-TAIL-f9a0"),
|
||||
("deploy/install-guide-d.md", "Install Forgejo Guide D", 1,
|
||||
"delta1 delta2 delta3 delta4 delta5 delta6", "NAMEKB-TAIL-b1c2"),
|
||||
("forgejo/forgejo-home.md", "Zeta Forge One", 0,
|
||||
"kilo1 kilo2 kilo3 kilo4 kilo5 kilo6", "NAMEKB-TAIL-a1b2"),
|
||||
("forgejo/forgejo-nginx.md", "Zeta Forge Two", 0,
|
||||
"lambda1 lambda2 lambda3 lambda4 lambda5 lambda6", "NAMEKB-TAIL-b3c4"),
|
||||
("zz-folder-map.md", "Zeta Misc Three", 0,
|
||||
"micro1 micro2 micro3 micro4 micro5 micro6", "NAMEKB-TAIL-e3f4"),
|
||||
]
|
||||
|
||||
#: The engineered invariant (measured, probe-verified, stable across
|
||||
#: re-imports — pinned for ALL FOUR scripted questions): the name-hit
|
||||
#: doc is the LAST of the top-5 suggested; the guides lead in
|
||||
#: retrieval-strength order (the ×2/×1 docs swap under the md5
|
||||
#: collision noise — pinned as measured, not as the gradient order).
|
||||
SUGGESTED = [
|
||||
"deploy/install-guide-a.md",
|
||||
"deploy/install-guide-b.md",
|
||||
"deploy/install-guide-d.md",
|
||||
"deploy/install-guide-c.md",
|
||||
"forgejo/forgejo-home.md",
|
||||
]
|
||||
NAME_HIT = "forgejo/forgejo-home.md" # the (a) tail-echo target
|
||||
RELATED = ["forgejo/forgejo-nginx.md", "zz-folder-map.md"]
|
||||
#: The folder question's rank-6+ order (the filler's name hit —
|
||||
#: ``folder`` — lifts it above the sibling; both stay rank 6+).
|
||||
RELATED_FOLDER = ["zz-folder-map.md", "forgejo/forgejo-nginx.md"]
|
||||
READ_TARGET = "zz-folder-map.md" # the single-read's first file line
|
||||
SENTINELS = [sentinel for _p, _t, _i, _f, sentinel in DOCS]
|
||||
ALL_SOURCE_LINES = [f"Source: {SOURCE}/{p}" for p, _t, _i, _f, _s in DOCS]
|
||||
|
||||
#: The turn's questions (the mock's trigger phrases — see the module
|
||||
#: docstring). The base question's name tokens (``install``,
|
||||
#: ``forgejo``) ground every turn (best cosine ≈ 0.35–0.41 ≥ the E2E
|
||||
#: 0.30 threshold) and fire the D1 name hits.
|
||||
BASE_QUESTION = "How do I install forgejo?"
|
||||
TAIL_QUESTION = BASE_QUESTION + " show the end of your notes"
|
||||
TOOLS_QUESTION = BASE_QUESTION + " use your tools"
|
||||
FOLDER_QUESTION = BASE_QUESTION + " repeat your folder map"
|
||||
assert "show the end of your notes" in TAIL_QUESTION.lower()
|
||||
assert "use your tools" in TOOLS_QUESTION.lower()
|
||||
assert "repeat your folder map" in FOLDER_QUESTION.lower()
|
||||
|
||||
|
||||
def _expected_summary(content: str, path: str) -> str:
|
||||
"""The mock lite model's byte-stable digest + the code pointer line
|
||||
(mirrors ``mock_llm.compose_answer``'s ``SUMMARY_MODE`` branch —
|
||||
first 24 tokens of the document content — plus the summarizer's
|
||||
deterministic ``Source:`` line; no model output is ever trusted)."""
|
||||
digest = " ".join(TOKEN_RE.findall(content.lower())[:24])
|
||||
return f"This document covers {digest}.\nSource: {SOURCE}/{path}"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seed_dirs() -> Path:
|
||||
"""The story's local-directory source: the tracked fixture dir
|
||||
(the app server runs on the same host, so the path is visible to
|
||||
it). The directory's basename is the source name (``kind=local``,
|
||||
phase 38)."""
|
||||
assert FIXTURES.is_dir(), "tests/fixtures/namekb is missing"
|
||||
for path, _t, _i, _f, _s in DOCS:
|
||||
assert (FIXTURES / path).is_file(), f"fixture doc {path} is missing"
|
||||
return FIXTURES
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, seed_dirs: Path) -> Iterator[str]:
|
||||
"""The real app under test — per-module app (the conftest pattern,
|
||||
cf. ``test_summary_seed_context.py``): NO ``BOR_GIT_SOURCES`` (the
|
||||
env fallback is git-only — the source here is a DB-registered local
|
||||
directory), the mock LLM, the mock-calibrated threshold, and the
|
||||
leak-guarded code defaults (the suggested/related tier settings
|
||||
ride their code defaults — 5 / 2 — exactly like the production
|
||||
``.env``-free defaults). The session app is never started in this
|
||||
isolated run, so no port clash."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (conftest pattern): the strength gradient
|
||||
# keeps every scripted turn grounded (best cosine ≈ 0.35–0.41).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits + the code-default budget (the
|
||||
# conftest leak-guard pattern).
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
||||
# reads it from cwd) — override it with an EMPTY value (the env var
|
||||
# beats the .env file): the registry must hold EXACTLY the local
|
||||
# directory this suite registers (a leftover env git list would
|
||||
# pollute the KB the scripted turns run against).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
|
||||
share one Postgres, so a leftover git_sources row or document would
|
||||
pollute the retrieval the scripted turns run against (the
|
||||
name-signal margins are pinned against EXACTLY these seven
|
||||
documents)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
|
||||
"""Poll the (cookie-authenticated) status endpoint until the run
|
||||
reaches a terminal state (the test_summary_seed_context pattern,
|
||||
over plain httpx — this fixture has no browser page yet)."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
def _assert_tiers(
|
||||
question: str, related_paths: list[str]
|
||||
) -> None:
|
||||
"""Pin the name-signal design with the app's REAL hybrid retrieval
|
||||
over the mock's embeddings (deterministic): the suggested tier is
|
||||
exactly the four guides (retrieval-strength order) + the name-hit
|
||||
doc LAST (LOCKED A3 — top-5, NO floor — the name-hit bonus, D2,
|
||||
lifts it into the tier) and the related tier is the rank-6+
|
||||
remainder (``related_max_docs`` = 2). A fixture-text regression
|
||||
that breaks the design fails here, at setup, with a clear
|
||||
message."""
|
||||
with SessionLocal() as db:
|
||||
chunks = retrieve(db, question, embed_text(question))
|
||||
suggested = [f"{d.source}/{d.path}" for d in select_suggested(chunks)]
|
||||
related = [
|
||||
f"{d.source}/{d.path}"
|
||||
for d in select_related(
|
||||
chunks,
|
||||
{d.id for d in select_suggested(chunks)},
|
||||
_Settings.model_fields["related_max_docs"].default,
|
||||
)
|
||||
]
|
||||
expected = [f"{SOURCE}/{p}" for p in SUGGESTED]
|
||||
assert suggested == expected, (
|
||||
f"suggested tier drifted for {question!r}: {suggested} "
|
||||
f"(expected {expected})"
|
||||
)
|
||||
assert related == [f"{SOURCE}/{p}" for p in related_paths], (
|
||||
f"related tier drifted for {question!r}: {related}"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def synced_kb(app_server: str, seed_dirs: Path) -> None:
|
||||
"""The story's precondition: the one-source KB synced under the
|
||||
deterministic mock. Registers the fixture directory through the
|
||||
authenticated API (the ``test_local_directory_sources.py``
|
||||
pattern), runs the REAL in-process sync (``POST /api/sync`` — walk
|
||||
→ chunk → embed → summaries → overview → folder summaries →
|
||||
version bump), pins the stored content byte-identical to the
|
||||
tracked fixture files, pins LOCKED A2 end-to-end (every doc stores
|
||||
the mock's byte-stable digest + exactly one embedded
|
||||
``is_summary`` chunk), and pins the engineered invariant (the
|
||||
name-hit doc LAST of the top-5) for ALL FOUR scripted questions."""
|
||||
_truncate_all()
|
||||
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": str(seed_dirs)}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
body = _wait_sync_done_http(client)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == len(DOCS), detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
# The import stored the tracked fixture strings BYTE-IDENTICALLY
|
||||
# and, for EVERY doc, the mock's byte-stable digest: the
|
||||
# deterministic assertion surface of the whole suite.
|
||||
with SessionLocal() as db:
|
||||
for path, _title, _i, _f, _s in DOCS:
|
||||
on_disk = (FIXTURES / path).read_text(encoding="utf-8")
|
||||
stored = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE, Document.path == path
|
||||
)
|
||||
)
|
||||
assert stored is not None, f"fixture doc {path} was not imported"
|
||||
assert stored.content == on_disk, f"stored content drifted for {path}"
|
||||
assert stored.summary == _expected_summary(on_disk, path), (
|
||||
f"summary is not the mock digest for {path}: {stored.summary!r}"
|
||||
)
|
||||
schunks = [c for c in stored.chunks if c.is_summary]
|
||||
assert len(schunks) == 1 and schunks[0].position == -1, (
|
||||
f"expected exactly one is_summary chunk for {path}"
|
||||
)
|
||||
assert schunks[0].embedding is not None, (
|
||||
f"the is_summary chunk of {path} is not embedded"
|
||||
)
|
||||
_assert_tiers(TAIL_QUESTION, RELATED)
|
||||
_assert_tiers(TOOLS_QUESTION, RELATED)
|
||||
_assert_tiers(FOLDER_QUESTION, RELATED_FOLDER)
|
||||
_assert_tiers(BASE_QUESTION, RELATED)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
"""Per-test query_log isolation (the KB itself is module-scoped —
|
||||
the scripted turns never change it, so the registry and the KB
|
||||
persist across the tests of this module)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the test_summary_seed_context house patterns)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: for the ``tool`` / ``done`` frames, independent of the UI rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _frames(page: Page) -> list[dict]:
|
||||
"""The SSE frames captured since the last submit (``_submit``
|
||||
clears the buffer), once the hook's background read settles."""
|
||||
deadline = time.monotonic() + 30.0
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.evaluate("window.__sseFrames = []")
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button recovered
|
||||
(the phase-48 settle wait)."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
|
||||
def _last_brain(page: Page) -> Locator:
|
||||
return page.locator(".msg.brain").last
|
||||
|
||||
|
||||
def _last_query_log() -> QueryLog:
|
||||
with SessionLocal() as db:
|
||||
rows = db.scalars(select(QueryLog)).all()
|
||||
assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}"
|
||||
return rows[0]
|
||||
|
||||
|
||||
def _assert_chip_row(page: Page, read_paths: list[str]) -> None:
|
||||
"""The citation surface of a grounded turn (phase 119, LOCKED A1 —
|
||||
the retired phase-118 A4 union is gone): the chip row is the
|
||||
AGENT-READ docs only (a zero-read turn chips NOTHING: the
|
||||
suggested docs are seed context, not citations)."""
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(len(read_paths))
|
||||
for path in read_paths:
|
||||
expect(
|
||||
chips.filter(has_text=path), message=f"chip for {path}"
|
||||
).to_have_count(1)
|
||||
|
||||
|
||||
def _assert_related_row(page: Page, related_paths: list[str]) -> None:
|
||||
"""The de-emphasized ``related-docs`` row (phase-113 UI, untouched
|
||||
by phase 119 — never a citation chip): the rank-6+ remainder
|
||||
deduped against the cited read docs (a read related doc is a chip,
|
||||
never a "nearby doc")."""
|
||||
row = page.locator(".msg.brain .related-docs")
|
||||
expect(row).to_have_count(1)
|
||||
expect(row.first).to_have_attribute("aria-label", "Nearby docs, in case")
|
||||
links = page.locator(".msg.brain .related-docs .related-doc")
|
||||
expect(links).to_have_count(len(related_paths))
|
||||
for i, path in enumerate(related_paths):
|
||||
expect(links.nth(i)).to_contain_text(f"{SOURCE}/{path}")
|
||||
# The related links are never citation chips.
|
||||
expect(page.locator(".msg.brain .related-docs .source-chip")).to_have_count(0)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. (a) the name-signal doc is the LAST of the top-5 suggested — the
|
||||
# tail echo proves it reached the prompt as the last seed block
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_name_hit_doc_is_last_suggested(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The tail-echo question quotes the last 160 chars of the seeded
|
||||
``<documents>`` block: they end in the LAST suggested doc's SUMMARY
|
||||
— the name-hit doc's byte-stable digest tail + pointer line (the
|
||||
digitless-component rule + the D2 bonus put ``forgejo-home`` in
|
||||
the tier, its neutral body keeps it last). No other doc's pointer
|
||||
line and no doc's tail sentinel are in the echoed context — the
|
||||
summary seed, not the full text. The zero-read grounded turn
|
||||
chips nothing (LOCKED A1); the related row renders rank 6+; the
|
||||
durable record carries suggested + related (118-A3 untouched)."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, TAIL_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# No tools on this turn — the answer is the mock's direct tail echo
|
||||
# of the seeded context (the summary-seed lens).
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# Phase 119 (LOCKED A1): the grounded done frame cites the READ
|
||||
# docs only — this turn read NOTHING (the summary-only fast path),
|
||||
# so the citation surface is empty; the suggested set (name-hit
|
||||
# doc included) is seed context, not citations.
|
||||
assert done["sources"] == [], done["sources"]
|
||||
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
|
||||
# The echoed tail ends in the name-hit doc's summary: its
|
||||
# deterministic ``Source:`` pointer line (the digest is pinned in
|
||||
# ``synced_kb`` — a content preview would carry the neutral-body
|
||||
# text instead, and the pointer line only ever exists on a stored
|
||||
# summary). The bubble renders the answer as markdown, which
|
||||
# collapses the summary's newline — pin the pointer line alone
|
||||
# (it sits inside the echoed 160 chars, byte-exact).
|
||||
expect(bubble).to_contain_text(f"Source: {SOURCE}/{NAME_HIT}")
|
||||
# No OTHER doc's pointer line is in the echoed tail — the
|
||||
# name-hit doc was the LAST suggested block (the (a) invariant's
|
||||
# visible proof).
|
||||
for path, _t, _i, _f, _s in DOCS:
|
||||
if path != NAME_HIT:
|
||||
expect(bubble).not_to_contain_text(f"Source: {SOURCE}/{path}")
|
||||
# And no document's tail sentinel: the full content of no
|
||||
# suggested doc reached the model (summary seed only).
|
||||
for sentinel in SENTINELS:
|
||||
expect(bubble).not_to_contain_text(sentinel)
|
||||
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
|
||||
|
||||
# The zero-read grounded turn chips NOTHING (LOCKED A1); the
|
||||
# related row renders rank 6+ (the de-emphasized row, untouched).
|
||||
_assert_chip_row(page, [])
|
||||
_assert_related_row(page, RELATED)
|
||||
|
||||
# Durable record: grounded; suggested + related (LOCKED A3 — the
|
||||
# log records retrieval, not citations; phase 119 A1 retires the
|
||||
# chip surface only, not the record).
|
||||
row = _last_query_log()
|
||||
assert row.question == TAIL_QUESTION
|
||||
assert row.deflected is False
|
||||
assert row.sources == ", ".join(f"{SOURCE}/{p}" for p in SUGGESTED + RELATED)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. (b) chips = READ docs only — the single-read flow chips exactly
|
||||
# the one read doc; the related row renders (deduped)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_single_read_chips_only_the_read_doc(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The ``use your tools`` flow drives the mock's single read:
|
||||
``ls`` (the top level) → drill ``ls(namekb)`` → ``read`` the FIRST
|
||||
file line (the root file ``zz-folder-map.md`` — subfolders list
|
||||
first, files after). The bubble carries EXACTLY ONE
|
||||
``.source-chip`` — the read doc (LOCKED A1); none of the five
|
||||
suggested docs chips (the name-hit doc is seed context, not a
|
||||
citation); the de-emphasized related row renders the rank-6+
|
||||
remainder deduped against the cited read doc (the read doc is a
|
||||
chip, never a "nearby doc")."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, TOOLS_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# Wire level: exactly three `tool` frames — ``ls`` (the top
|
||||
# level), the drill ``ls`` scoped to the first (only) source
|
||||
# (phase 94), then ``read`` the first file line's combined
|
||||
# source/path — and all three ahead of the first `delta` frame.
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "ls", "argument": SOURCE},
|
||||
{"type": "tool", "name": "read", "argument": f"{SOURCE}/{READ_TARGET}"},
|
||||
], _tool_frames(frames)
|
||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert all(
|
||||
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# Phase 119 (LOCKED A1): done.sources = the READ docs only —
|
||||
# exactly the one read; the five suggested docs (name-hit doc
|
||||
# included) never chip.
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
(SOURCE, READ_TARGET)
|
||||
], done["sources"]
|
||||
# done.related = the rank-6+ remainder deduped against the cited
|
||||
# read doc — the read doc is rank 7 (IN the remainder) and drops
|
||||
# out: only the sibling name-hit doc remains.
|
||||
assert [(s["source"], s["path"]) for s in done["related"]] == [
|
||||
(SOURCE, "forgejo/forgejo-nginx.md")
|
||||
], done["related"]
|
||||
|
||||
# The final answer quotes the read document (the mock's
|
||||
# deterministic quote: "Read <source/path>. <first 80 chars of its
|
||||
# content>").
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(
|
||||
bubble,
|
||||
).to_contain_text(f"Read {SOURCE}/{READ_TARGET}.", timeout=30_000)
|
||||
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
|
||||
|
||||
# The UI chip row: EXACTLY the one read doc (LOCKED A1) — and
|
||||
# explicitly NONE of the suggested docs (the name-hit doc first).
|
||||
_assert_chip_row(page, [READ_TARGET])
|
||||
expect(
|
||||
page.locator(".msg.brain .source-chip", has_text="install-guide")
|
||||
).to_have_count(0)
|
||||
expect(
|
||||
page.locator(".msg.brain .source-chip", has_text=NAME_HIT)
|
||||
).to_have_count(0)
|
||||
# The related row renders rank 6+ deduped against the cited read
|
||||
# doc (phase-113 behavior untouched).
|
||||
_assert_related_row(page, ["forgejo/forgejo-nginx.md"])
|
||||
|
||||
# Durable record: grounded; suggested + related + read (deduped —
|
||||
# the read doc sits in the related slot, LOCKED A3 — phase 119 A1
|
||||
# retires the chip surface only, not the record).
|
||||
row = _last_query_log()
|
||||
assert row.question == TOOLS_QUESTION
|
||||
assert row.deflected is False
|
||||
assert row.sources == ", ".join(
|
||||
f"{SOURCE}/{p}" for p in SUGGESTED + RELATED
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 3. (c) a zero-read grounded turn chips nothing (LOCKED A1's visible
|
||||
# consequence)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_zero_read_grounded_turn_chips_nothing(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""A PLAIN distinctive question (no tool trigger) grounds on the
|
||||
name tokens: the ``done`` frame's ``sources == []`` and ZERO
|
||||
``.source-chip`` elements in the bubble — LOCKED A1's visible
|
||||
consequence (the suggested docs, name-hit doc included, are seed
|
||||
context, never citations); the answer still renders grounded
|
||||
(deflected: false); the related row renders rank 6+."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, BASE_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# LOCKED A1's wire-level consequence: a grounded turn that read
|
||||
# nothing cites nothing.
|
||||
assert done["sources"] == [], done["sources"]
|
||||
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
# Zero chips — not just "no suggested chips": the citation
|
||||
# surface is empty.
|
||||
_assert_chip_row(page, [])
|
||||
_assert_related_row(page, RELATED)
|
||||
|
||||
# Durable record: grounded; suggested + related (LOCKED A3).
|
||||
row = _last_query_log()
|
||||
assert row.question == BASE_QUESTION
|
||||
assert row.deflected is False
|
||||
assert row.sources == ", ".join(f"{SOURCE}/{p}" for p in SUGGESTED + RELATED)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 4. (d) the suggested-folder lines ride the prompt — echoed verbatim
|
||||
# by the new mock trigger
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_folder_lines_echoed_verbatim(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The distinctive question + ``repeat your folder map`` (the new
|
||||
mock trigger — the prompt-injection-echo convention of
|
||||
``<tuning>`` / ``<knowledge_base>``) ⇒ the answer ends with the
|
||||
suggested-folder lines VERBATIM (joined with ``; ``): the guides'
|
||||
folder line (its owner excluded from the entries) and the
|
||||
name-hit doc's folder line — ``namekb/forgejo/: forgejo-nginx.md``
|
||||
(the line prefix + its known sibling entry; the owner
|
||||
``forgejo-home`` is excluded — its identity is already in its
|
||||
``<document>`` block)."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, FOLDER_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
assert done["sources"] == [], done["sources"]
|
||||
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
|
||||
# The folder lines VERBATIM — both lines, suggested-folder order
|
||||
# (the guides' folder first — its first suggested doc leads; the
|
||||
# name-hit doc's folder second), entries owner-excluded.
|
||||
expect(bubble).to_contain_text(
|
||||
"(folders: "
|
||||
"namekb/deploy/: install-guide-b.md, install-guide-c.md, install-guide-d.md; "
|
||||
"namekb/forgejo/: forgejo-nginx.md)",
|
||||
)
|
||||
# The task's pin: the name-hit doc's folder line prefix + one
|
||||
# known sibling entry.
|
||||
expect(bubble).to_contain_text(f"{SOURCE}/forgejo/: forgejo-nginx.md")
|
||||
|
||||
# Zero-read grounded turn — chips nothing (LOCKED A1).
|
||||
_assert_chip_row(page, [])
|
||||
|
||||
# Durable record: grounded; suggested + related (LOCKED A3 — the
|
||||
# folder question's rank-6+ order).
|
||||
row = _last_query_log()
|
||||
assert row.question == FOLDER_QUESTION
|
||||
assert row.deflected is False
|
||||
assert row.sources == ", ".join(
|
||||
f"{SOURCE}/{p}" for p in SUGGESTED + RELATED_FOLDER
|
||||
)
|
||||
Reference in New Issue
Block a user