phase: 119_name_signal_read_chips
All verification complete. Final report: **Phase 119 final verification pass — all criteria verified, one stale pin fixed.** - Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry. - Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged. - New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2. - Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed). - Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met). - Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
@@ -436,6 +436,29 @@ Implements just enough of the aipi surface:
|
||||
phrases — the phase-71/72/94 ordering convention); verified
|
||||
2026-09-16: no existing E2E question or fixture file contains the
|
||||
phrase, so every other suite is unaffected.
|
||||
- user message containing ``repeat your folder map``
|
||||
(``FOLDER_MAP_TRIGGER``, phase 119, D3 — the suggested-folder
|
||||
context's story suite ``tests/e2e/test_name_signal_read_chips.py``)
|
||||
**and** the system prompt contains the ``<documents>`` section ->
|
||||
the composed answer ends with `` (folders: <line 1>; <line 2>; …)``
|
||||
echoing the suggested-folder lines VERBATIM — the plain lines
|
||||
between ``SUGGEST_INTRO`` and the first ``<document `` in the
|
||||
``<documents>`` section (``app.rag.prompts.build_high_prompt`` /
|
||||
``app.rag.agent.suggested_folder_lines``, phase 119 LOCKED A4:
|
||||
``<source>/<prefix>/: e1, e2, …``), joined with ``; ``. If no
|
||||
folder lines are present (empty *folder_lines* — the
|
||||
byte-identical phase-118 shape) the echo is omitted and the answer
|
||||
is otherwise unchanged. The same prompt-injection-echo convention
|
||||
as the ``<tuning>`` / ``<knowledge_base>`` triggers above (a SUFFIX
|
||||
appended after both, so the folders suffix is the last thing
|
||||
rendered): phase 119 D3's observability — the folder lines land in
|
||||
the model's prompt only on grounded turns, and the echo is the
|
||||
E2E's deterministic lens on their EXACT text (line identity shape
|
||||
+ entry order + the owning-doc exclusion + the caps). It composes
|
||||
with any base answer (it does not compete with a marker branch);
|
||||
the trigger phrase is disjoint from every other trigger's; verified
|
||||
2026-09-16: no existing E2E question or fixture file contains the
|
||||
phrase, so every other suite is unaffected.
|
||||
|
||||
Failure injection (phase 67, LLM retry, TODO.md L3) — deterministic
|
||||
dead-endpoint behavior for the retry E2E suite (``tests/e2e/
|
||||
@@ -675,6 +698,17 @@ TABLE_TRIGGER = "show me a table"
|
||||
#: phrase, so every other suite is unaffected.
|
||||
HISTORY_TRIGGER = "echo my history"
|
||||
|
||||
#: Phase 119 (D3, the suggested-folder context's story suite): a user
|
||||
#: message containing this substring (case-insensitive) — combined
|
||||
#: with the ``<documents>`` section in the system prompt (a grounded
|
||||
#: turn) — gets the composed answer suffixed with the folder lines
|
||||
#: VERBATIM (`` (folders: <line 1>; <line 2>; …)``, joined with
|
||||
#: ``; ``; omitted when the section carries no folder lines) — see the
|
||||
#: module docstring. Disjoint from every other trigger phrase; verified
|
||||
#: 2026-09-16: no existing E2E question or fixture file contains the
|
||||
#: phrase, so every other suite is unaffected.
|
||||
FOLDER_MAP_TRIGGER = "repeat your folder map"
|
||||
|
||||
TABLE_ANSWER = (
|
||||
"Here's the shape, in a table:\n"
|
||||
"\n"
|
||||
@@ -1657,6 +1691,32 @@ def first_kb_bullet(system: str) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def folder_map_lines(system: str) -> list[str]:
|
||||
"""The suggested-folder lines of the ``<documents>`` section
|
||||
(phase 119, D3, LOCKED A4) — or ``[]`` when there are none.
|
||||
|
||||
``build_high_prompt`` lays the section body out as ``SUGGEST_INTRO``
|
||||
(line 1), the folder lines (each on its own line, immediately after
|
||||
the intro line), a blank line, then the first ``<document>`` block
|
||||
— so the lines are exactly the non-blank lines between the intro
|
||||
line and the first ``<document `` line of the block. With no folder
|
||||
lines (the byte-identical phase-118 shape) there is only the blank
|
||||
line there, and this returns ``[]`` (the echo is then omitted — the
|
||||
answer is otherwise unchanged).
|
||||
"""
|
||||
block = _DOCUMENTS_BLOCK_RE.search(system)
|
||||
if not block:
|
||||
return []
|
||||
lines = block.group(0).splitlines()
|
||||
out: list[str] = []
|
||||
for line in lines[2:]: # skip the tag line + the SUGGEST_INTRO line
|
||||
if line.lstrip().startswith("<document"):
|
||||
break
|
||||
if line.strip():
|
||||
out.append(line)
|
||||
return out
|
||||
|
||||
|
||||
def _history_echo(body: dict[str, Any]) -> str:
|
||||
"""The phase-74 history echo (byte-stable, stateless over messages).
|
||||
|
||||
@@ -2093,6 +2153,18 @@ def compose_answer(body: dict[str, Any]) -> str:
|
||||
bullet = first_kb_bullet(system)
|
||||
if bullet:
|
||||
answer = f"{answer} (kb: {bullet})"
|
||||
# Suggested-folder context (phase 119, D3, LOCKED A4): when the user
|
||||
# message carries FOLDER_MAP_TRIGGER and the system prompt carries
|
||||
# the <documents> section, the answer ends with the folder lines
|
||||
# VERBATIM (joined with "; ") — the E2E's deterministic lens on the
|
||||
# exact lines build_high_prompt seeded (D3 observability; the same
|
||||
# prompt-injection-echo convention as the two echoes above). Omitted
|
||||
# when the section carries no folder lines (the byte-identical
|
||||
# phase-118 shape) — the answer is otherwise unchanged.
|
||||
if FOLDER_MAP_TRIGGER in user.lower() and "<documents>" in system:
|
||||
folder_lines = folder_map_lines(system)
|
||||
if folder_lines:
|
||||
answer = f"{answer} (folders: {'; '.join(folder_lines)})"
|
||||
return answer
|
||||
|
||||
|
||||
|
||||
@@ -60,12 +60,15 @@ Test → story mapping (Playwright Mapping Rule):
|
||||
``tool`` frames (``ls``, then ``read`` with the combined path, ahead
|
||||
of any delta), the UI shows the transient calling-tool status while a
|
||||
tool runs, the bubble shows both tool lines, the final answer quotes
|
||||
the read document, and the source chips include the read document
|
||||
(viewer link).
|
||||
the read document, and the source chips are EXACTLY the read
|
||||
document (phase 119, LOCKED A1 — chips cite read docs only; viewer
|
||||
link).
|
||||
2. ``test_tool_lines_re_render_after_reload`` — the persisted record
|
||||
(phase 14) re-renders the tool lines.
|
||||
3. ``test_plain_grounded_question_has_no_tool_frames`` — no marker → no
|
||||
``tool`` frames, the answer renders exactly as today (regression
|
||||
``tool`` frames, the answer renders exactly as today, and (phase 119,
|
||||
LOCKED A1) the zero-read grounded turn chips NOTHING (the retrieval
|
||||
doc is cited only in the durable record, never as a chip) (regression
|
||||
inside the story file).
|
||||
4. ``test_deflected_question_has_no_tool_frames`` — the tools are
|
||||
grounded-only: a deflected turn runs none.
|
||||
@@ -444,8 +447,12 @@ def test_marker_question_lists_reads_and_quotes(
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
# Phase 119 (LOCKED A1): done.sources = the READ docs only —
|
||||
# exactly the scripted read; the retrieval (seed) doc is suggested
|
||||
# context, never a citation chip (the retired phase-118 A4
|
||||
# suggested+read union is gone). The durable record below still
|
||||
# carries both (LOCKED A3, untouched).
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
(SEED_SOURCE, SEED_PATH),
|
||||
(READ_SOURCE, READ_PATH),
|
||||
]
|
||||
|
||||
@@ -465,16 +472,17 @@ def test_marker_question_lists_reads_and_quotes(
|
||||
expect(bubble).to_contain_text(ANSWER_PREFIX)
|
||||
expect(bubble).to_contain_text(ANSWER_QUOTE)
|
||||
|
||||
# Source chips: the retrieval doc AND the read doc (deduped, in
|
||||
# order) — the read chip links to the viewer.
|
||||
# Source chips: the read doc ONLY (phase 119, LOCKED A1 — the
|
||||
# retrieval doc was never read, so it never chips; the retired
|
||||
# phase-118 A4 union is gone) — the read chip links to the viewer.
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(2)
|
||||
expect(chips.nth(0)).to_contain_text(SEED_SP)
|
||||
expect(chips).to_have_count(1)
|
||||
chip_read = page.locator(".msg.brain .source-chip", has_text=READ_PATH)
|
||||
expect(chip_read).to_have_count(1)
|
||||
expect(chip_read.first).to_have_attribute("href", READ_CHIP_HREF)
|
||||
|
||||
# Durable record: grounded, both sources logged (retrieval + read).
|
||||
# Durable record: grounded, both sources logged (suggested + read —
|
||||
# LOCKED A3, untouched by phase 119 A1).
|
||||
row = _last_query_log()
|
||||
assert row.question == MARKER_QUESTION
|
||||
assert row.deflected is False
|
||||
@@ -540,14 +548,15 @@ def test_plain_grounded_question_has_no_tool_frames(
|
||||
assert _tool_frames(_frames(page)) == []
|
||||
expect(page.locator(".tool-call")).to_have_count(0)
|
||||
|
||||
# The standard grounded answer, citing the retrieval doc only — the
|
||||
# referenced JSON stays OUT of the sources (it was never read).
|
||||
# The standard grounded answer, with ZERO citation chips — the
|
||||
# turn read nothing, so (phase 119, LOCKED A1) the chip row is
|
||||
# empty: the retrieval doc is cited in the durable record only
|
||||
# (the retired phase-118 A4 union is gone); the referenced JSON
|
||||
# stays out of the record too (it was never read).
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(PLAIN_QUESTION)
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(1)
|
||||
expect(chips.first).to_contain_text(SEED_SP)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
row = _last_query_log()
|
||||
assert row.question == PLAIN_QUESTION
|
||||
|
||||
@@ -65,9 +65,11 @@ Test → story mapping (Playwright Mapping Rule):
|
||||
``I read <sp1> and <sp2>.`` line; the round cap (default 10) bounds
|
||||
the turn, no budget refusal anywhere.
|
||||
2. ``test_done_sources_include_reads`` — the source chips under the
|
||||
answer list the retrieval doc PLUS both read documents, deduped
|
||||
(the phase-37 ``done.sources`` extension contract, now with 2
|
||||
reads); the same combined list lands in ``query_log.sources``.
|
||||
answer list the BOTH read documents, deduped (phase 119, LOCKED A1:
|
||||
chips cite read docs only — the retrieval doc was never read, so it
|
||||
never chips; the retired phase-118 A4 suggested+read union is
|
||||
gone); the retrieval doc PLUS both reads still land in
|
||||
``query_log.sources`` (LOCKED A3, untouched).
|
||||
3. ``test_relist_allowed`` — the listing tool ran (its line rendered)
|
||||
and no pre-phase-45 budget refusal ("… budget left") appears
|
||||
anywhere in the message bubble or tool lines: the old
|
||||
@@ -195,14 +197,22 @@ BUDGET_REFUSAL_FRAGMENTS = (
|
||||
"budget left",
|
||||
)
|
||||
|
||||
# The combined source list the app reports (app/api/chat.py): retrieval
|
||||
# docs first, then the agent's read docs, deduped by (source, path).
|
||||
# The combined source list the DURABLE record reports
|
||||
# (app/api/chat.py, LOCKED A3 — suggested + related + read, deduped):
|
||||
# retrieval doc first, then the agent's read docs, deduped by
|
||||
# (source, path). The citation surface (done.sources, the chips) is
|
||||
# the READ set only (phase 119, LOCKED A1 — the retired phase-118 A4
|
||||
# suggested+read union is gone): the retrieval doc was never read.
|
||||
EXPECTED_SOURCES = [
|
||||
(SEED_SOURCE, SEED_PATH),
|
||||
(READ1_SOURCE, READ1_PATH),
|
||||
(READ2_SOURCE, READ2_PATH),
|
||||
]
|
||||
EXPECTED_SOURCES_LINE = ", ".join(f"{s}/{p}" for s, p in EXPECTED_SOURCES)
|
||||
READ_ONLY_SOURCES = [
|
||||
(READ1_SOURCE, READ1_PATH),
|
||||
(READ2_SOURCE, READ2_PATH),
|
||||
]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
@@ -508,7 +518,7 @@ def test_multi_read_turn(
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. done.sources / source chips: retrieval doc + BOTH reads, deduped
|
||||
# 2. done.sources / source chips: BOTH reads, deduped (phase 119 A1)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -523,23 +533,28 @@ def test_done_sources_include_reads(
|
||||
_submit(page, MULTI_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# Wire level: done.sources is the retrieval doc FIRST, then both
|
||||
# read documents — deduped (the retrieval doc was never read, the
|
||||
# reads are each read once; nothing appears twice).
|
||||
# Wire level: done.sources is the READ documents only, in read
|
||||
# order — deduped (the reads are each read once; nothing appears
|
||||
# twice). Phase 119, LOCKED A1: the retrieval doc was never read,
|
||||
# so it is NOT in the citation surface (the retired phase-118 A4
|
||||
# suggested+read union is gone); it still lands in the durable
|
||||
# record (LOCKED A3, pinned in test 1).
|
||||
frames = _frames(page)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == EXPECTED_SOURCES
|
||||
assert [
|
||||
(s["source"], s["path"]) for s in done["sources"]
|
||||
] == READ_ONLY_SOURCES, done["sources"]
|
||||
pairs = [(s["source"], s["path"]) for s in done["sources"]]
|
||||
assert len(pairs) == len(set(pairs)), "done.sources must be deduped"
|
||||
|
||||
# UI: exactly three source chips under the answer, in the same
|
||||
# order, each a viewer link — no duplicated chip.
|
||||
# UI: exactly two source chips under the answer (the read docs,
|
||||
# read order), each a viewer link — no duplicated chip, and no
|
||||
# chip for the never-read retrieval doc (phase 119 A1).
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(3)
|
||||
expect(chips.nth(0)).to_contain_text(SEED_SP)
|
||||
expect(chips.nth(1)).to_contain_text(READ1_SP)
|
||||
expect(chips.nth(2)).to_contain_text(READ2_SP)
|
||||
for i, (source, path) in enumerate(EXPECTED_SOURCES):
|
||||
expect(chips).to_have_count(2)
|
||||
expect(chips.nth(0)).to_contain_text(READ1_SP)
|
||||
expect(chips.nth(1)).to_contain_text(READ2_SP)
|
||||
for i, (source, path) in enumerate(READ_ONLY_SOURCES):
|
||||
expect(chips.nth(i)).to_have_attribute(
|
||||
"href", f"/document.html?source={source}&path={path}&back=%2F"
|
||||
)
|
||||
@@ -631,12 +646,13 @@ def test_single_tool_flow_regression(
|
||||
expect(bubble).not_to_contain_text(BOTH_READS_LINE)
|
||||
expect(bubble).not_to_contain_text(READ2_SP)
|
||||
|
||||
# done: non-deflected; sources = retrieval doc + the single read
|
||||
# (READ2 absent — it was never read).
|
||||
# done: non-deflected; sources = the single READ doc only (phase
|
||||
# 119, LOCKED A1 — READ2 absent: never read; the retrieval doc
|
||||
# absent: never read — the retired phase-118 A4 union is gone; the
|
||||
# durable record below keeps retrieval + read, LOCKED A3).
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
(SEED_SOURCE, SEED_PATH),
|
||||
(READ1_SOURCE, READ1_PATH),
|
||||
]
|
||||
|
||||
|
||||
@@ -391,8 +391,11 @@ def test_admin_generates_token_in_ui(page: Page, app_url: str, db_ready: None) -
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. The token flow: a fresh context signs in through the real gate
|
||||
# and USES the app — a grounded chat turn (mock LLM), a cited
|
||||
# document opened in the same-page modal, the role-user header
|
||||
# and USES the app — a grounded chat turn (mock LLM; phase 119,
|
||||
# LOCKED A1: the turn is the scripted summary-read flow — chips
|
||||
# cite READ docs only, and this turn's read earns the cited chip),
|
||||
# a cited document opened in the same-page modal, the role-user
|
||||
# header
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -417,15 +420,27 @@ def test_token_user_uses_the_app(
|
||||
# task-04 helper: fill #auth-gate-input → submit → gate hides).
|
||||
login_with_token(user, app_url, token)
|
||||
|
||||
# Use the app: a grounded turn against the seeded KB (mock
|
||||
# LLM) — the brain bubble renders the deterministic answer.
|
||||
_ask(user, "How is my Kubernetes cluster set up? (api-tokens-flow)")
|
||||
# Use the app: a GROUNDED turn against the seeded KB (mock
|
||||
# LLM). Phase 119 (LOCKED A1): chips cite READ docs only — a
|
||||
# plain question would chip nothing (the retired phase-118 A4
|
||||
# union is gone), so the turn is the mock's scripted
|
||||
# summary-read flow (SUMMARY_SEED_READ_TRIGGER): it reads the
|
||||
# kubernetes fixture, and that read is the chip's source.
|
||||
q = (
|
||||
"Read the suggested document: read docs/homelab/kubernetes.md — "
|
||||
"how is my Kubernetes cluster set up? (api-tokens-flow)"
|
||||
)
|
||||
user.fill("#message-input", q)
|
||||
user.click("#send-btn")
|
||||
expect(user.locator(".msg.user .bubble").last).to_contain_text(q)
|
||||
|
||||
# A cited source chip opens the document in the SAME-PAGE
|
||||
# modal (the require_user content endpoint passes for a
|
||||
# live token session).
|
||||
# The turn's cited source chip (its one READ doc — the scripted
|
||||
# read, phase 119 A1) opens the document in the SAME-PAGE modal
|
||||
# (the require_user content endpoint passes for a live token
|
||||
# session).
|
||||
chip = user.locator(".msg.brain a.source-chip").first
|
||||
expect(chip).to_be_visible(timeout=15_000)
|
||||
expect(chip).to_be_visible(timeout=30_000)
|
||||
expect(user.locator("#send-label")).to_have_text("Send")
|
||||
chip.click()
|
||||
expect(user.locator("#doc-modal")).to_be_visible()
|
||||
expect(
|
||||
|
||||
@@ -63,9 +63,6 @@ from e2e.auth_helpers import login
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "docs"
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
#: Phase 10 viewer URL + phase 13 back=/ (the restored chip must be
|
||||
#: byte-identical to the live-rendered one).
|
||||
CHIP_HREF = "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F"
|
||||
|
||||
|
||||
async def _import_fixtures(mock_port: int) -> ImportSummary:
|
||||
@@ -279,10 +276,9 @@ def test_open_chat_returns_to_history(
|
||||
# …the SAME answer text the History session saw (pixel-identical
|
||||
# restore through renderStoredMessage)…
|
||||
assert bubble.inner_text() == answer_before
|
||||
# …with its source chip restored byte-identically.
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
expect(chip).to_have_count(1)
|
||||
expect(chip.first).to_have_attribute("href", CHIP_HREF)
|
||||
# …with ZERO citation chips restored — the saved turn read
|
||||
# nothing, so its sources list is empty (phase 119, LOCKED A1;
|
||||
# the retired phase-118 A4 suggested-chip pin is gone).
|
||||
|
||||
# The conversation continues: a new turn streams fine…
|
||||
_ask(page, "How is my Kubernetes cluster set up? (hist-open-2)")
|
||||
|
||||
@@ -42,9 +42,6 @@ OFF_TOPIC = "How do I bake sourdough bread?"
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
DEFLECT_PHRASE = r"haven't done anything like that"
|
||||
STORAGE_KEY = "bor.chat.v1"
|
||||
#: Phase 10 viewer URL + phase 13 back=/ (the restored chip must be
|
||||
#: byte-identical to the live-rendered one).
|
||||
CHIP_HREF = "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F"
|
||||
|
||||
|
||||
async def _import_fixtures(mock_port: int) -> ImportSummary:
|
||||
@@ -143,22 +140,24 @@ def test_conversation_survives_reload(
|
||||
assert MOCK_ANSWER_MARKER in brain["text"]
|
||||
assert "<" not in brain["text"], "persisted brain text must be raw, not rendered HTML"
|
||||
assert brain["deflected"] is False
|
||||
assert any(s["path"] == "homelab/kubernetes.md" for s in brain["sources"])
|
||||
# Phase 119 (LOCKED A1): the turn read nothing, so its done sources
|
||||
# — and the persisted record — are EMPTY (the suggested kubernetes
|
||||
# doc is context, not a citation; the retired phase-118 A4 union is
|
||||
# gone).
|
||||
|
||||
# Refresh — the same context keeps its localStorage.
|
||||
page.reload()
|
||||
expect(page.locator("#empty-state")).to_be_hidden()
|
||||
|
||||
# Both bubbles restored: text + the source chip with the exact viewer URL.
|
||||
# Both bubbles restored: the answer text, and ZERO citation chips
|
||||
# (the zero-read turn persisted an empty sources list — phase 119,
|
||||
# LOCKED A1; the retired phase-118 A4 chip pin is gone).
|
||||
expect(page.locator(".msg.user .bubble")).to_have_count(1)
|
||||
expect(page.locator(".msg.user .bubble")).to_contain_text(QUESTION)
|
||||
bubble = page.locator(".msg.brain .bubble")
|
||||
expect(bubble).to_have_count(1)
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
expect(chip).to_have_count(1)
|
||||
expect(chip.first).to_have_attribute("href", CHIP_HREF)
|
||||
expect(chip.first).not_to_have_attribute("target") # phase 26: modal, not a new tab
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
# The restore is read-only: storage still holds the same two messages.
|
||||
assert [m["who"] for m in _stored_parsed(page)["messages"]] == ["user", "brain"]
|
||||
@@ -294,7 +293,9 @@ def test_persists_across_page_navigation(
|
||||
expect(page.locator(".msg.user .bubble").first).to_contain_text(QUESTION)
|
||||
expect(page.locator(".msg.user .bubble").nth(1)).to_contain_text(OFF_TOPIC)
|
||||
expect(page.locator(".msg.brain .bubble").first).to_contain_text(MOCK_ANSWER_MARKER)
|
||||
expect(page.locator(".msg.brain .source-chip", has_text="kubernetes.md")).to_have_count(1)
|
||||
# Zero citation chips (phase 119, LOCKED A1 — the grounded turn read
|
||||
# nothing; the retired phase-118 A4 union is gone).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
deflected = page.locator(".msg.brain.is-deflected .bubble")
|
||||
expect(deflected).to_have_count(1)
|
||||
expect(deflected.first).to_contain_text(re.compile(DEFLECT_PHRASE, re.IGNORECASE))
|
||||
|
||||
+21
-17
@@ -97,18 +97,12 @@ def test_on_topic_question_streams_grounded_answer(
|
||||
expect(bubble.first).to_contain_text(QUESTION, timeout=30_000)
|
||||
expect(bubble.first).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
|
||||
# Grounded: a kubernetes.md source chip renders under the bubble
|
||||
# (top-N docs can add more chips; the question's doc must be among them).
|
||||
# Phase 26: the chip opens the document in the SAME-PAGE modal — no new
|
||||
# tab; the encoded href stays as the no-JS / context-menu escape hatch
|
||||
# (phase 13's back=/ lets the viewer's back button return to chat).
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
expect(chip).to_have_count(1)
|
||||
expect(chip.first).to_contain_text("kubernetes.md")
|
||||
expect(chip.first).to_have_attribute(
|
||||
"href", "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F"
|
||||
)
|
||||
expect(chip.first).not_to_have_attribute("target") # phase 26: modal, not a new tab
|
||||
# Phase 119 (LOCKED A1): a zero-read grounded turn chips NOTHING —
|
||||
# the suggested kubernetes.md is seed context, not a citation chip
|
||||
# (the retired phase-118 A4 suggested+read union is gone; the
|
||||
# pre-phase-26 chip/contract pins retired with it). The grounding is
|
||||
# pinned by the query_log row below (LOCKED A3, untouched).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
# Button recovers: enabled + "Send" (never stale).
|
||||
expect(page.locator("#send-btn")).to_be_enabled()
|
||||
@@ -121,9 +115,16 @@ def test_chat_logs_query(page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
login(page, app_url, next="/")
|
||||
page.fill("#message-input", QUESTION)
|
||||
page.click("#send-btn")
|
||||
expect(
|
||||
page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
).to_have_count(1, timeout=30_000)
|
||||
# The turn settles with the mock answer (phase 119 A1: no chip to
|
||||
# wait on — a zero-read turn chips nothing; the retired phase-118
|
||||
# A4 union is gone). The button recovery is the settle sync: the
|
||||
# client re-enables Send on the done frame, and the server writes
|
||||
# the query_log row just before yielding it.
|
||||
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
|
||||
"Deterministic mock answer for E2E", timeout=30_000
|
||||
)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
# App still healthy after the turn.
|
||||
r = httpx.get(f"{app_url}/api/health", timeout=5)
|
||||
@@ -180,5 +181,8 @@ def test_sse_stream_shape(app_url: str, mock_llm: int, db_ready: None) -> None:
|
||||
assert frames[-1]["type"] == "done" # done is the final event
|
||||
assert done[0]["deflected"] is False
|
||||
assert done[0]["suggestions"] == []
|
||||
assert done[0]["sources"], "done must carry the cited sources"
|
||||
assert any(s["path"] == "homelab/kubernetes.md" for s in done[0]["sources"])
|
||||
# Phase 119 (LOCKED A1): done.sources = the READ docs only — this
|
||||
# plain turn read nothing, so the frame carries an EMPTY sources
|
||||
# list (the retired phase-118 A4 suggested+read union is gone; the
|
||||
# suggested doc's durable record lives in query_log, not the frame).
|
||||
assert done[0]["sources"] == [], done[0]["sources"]
|
||||
|
||||
@@ -358,8 +358,9 @@ def test_behavior_unchanged_smoke(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
"""AC6: layout/behavior unchanged under the new skin — an on-topic
|
||||
question streams a grounded answer, renders a source chip, and the send
|
||||
button recovers (never stale)."""
|
||||
question streams a grounded answer (phase 119, LOCKED A1: a zero-read
|
||||
turn renders ZERO source chips — the retired phase-118 A4
|
||||
suggested-chip is gone) and the send button recovers (never stale)."""
|
||||
_seed_kb(mock_llm)
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/") # phase 79: chat is require_user-gated
|
||||
@@ -373,7 +374,8 @@ def test_behavior_unchanged_smoke(
|
||||
bubble.first.wait_for(state="visible", timeout=30_000)
|
||||
expect(bubble.first).to_contain_text(QUESTION, timeout=30_000)
|
||||
expect(bubble.first).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(page.locator(".msg.brain .source-chip", has_text="kubernetes.md")).to_have_count(1)
|
||||
# Phase 119 (LOCKED A1): zero-read turn → zero citation chips.
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
# The state machine settled: button re-enabled, label back to "Send".
|
||||
expect(page.locator("#send-btn")).to_be_enabled()
|
||||
|
||||
@@ -19,10 +19,16 @@ viewer URL survives as each link's ``href`` (the no-JS / context-menu
|
||||
"open in new tab" escape hatch), so the back contract is asserted on
|
||||
that exact href and verified by navigating to it directly.
|
||||
|
||||
Phase 119 re-target (LOCKED A1): chips cite READ docs only — a plain
|
||||
question chips nothing, so test 1 drives the mock's scripted
|
||||
summary-read flow (``SUMMARY_SEED_READ_TRIGGER``): the turn ``read``s
|
||||
the kubernetes fixture, and its READ-doc chip is what carries the
|
||||
``&back=%2F`` href the story asserts.
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule):
|
||||
1. ``test_back_from_chat_returns_to_chat`` — question → source chip href
|
||||
(carries ``&back=%2F``) → viewer back link href ``/`` labeled "Chat"
|
||||
→ click → the chat page.
|
||||
1. ``test_back_from_chat_returns_to_chat`` — scripted-read question →
|
||||
the read doc's source chip href (carries ``&back=%2F``) → viewer
|
||||
back link href ``/`` labeled "Chat" → click → the chat page.
|
||||
2. ``test_back_from_sources_returns_to_sources`` — Sources table link
|
||||
href (no ``back`` param) → back link href ``/sources.html`` labeled
|
||||
"Sources" → click → the Sources page.
|
||||
@@ -49,7 +55,15 @@ from e2e.auth_helpers import login
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "docs"
|
||||
QUESTION = "How is my Kubernetes cluster set up?"
|
||||
# Phase 119 (LOCKED A1): the chip is the turn's READ doc — the scripted
|
||||
# summary-read flow (mock_llm.SUMMARY_SEED_READ_TRIGGER) reads the
|
||||
# kubernetes fixture so the story's chip (with its back=%2F href)
|
||||
# exists (a zero-read turn would chip nothing — the retired phase-118
|
||||
# A4 suggested-chip is gone).
|
||||
QUESTION = (
|
||||
"Read the suggested document: read docs/homelab/kubernetes.md — "
|
||||
"how is my Kubernetes cluster set up?"
|
||||
)
|
||||
# Seeded fixture doc (source=docs) shared by every test in this file.
|
||||
DOC_SOURCE = "docs"
|
||||
DOC_PATH = "homelab%2Fkubernetes.md"
|
||||
@@ -111,6 +125,8 @@ def test_back_from_chat_returns_to_chat(
|
||||
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
expect(chip).to_have_count(1, timeout=30_000)
|
||||
# Phase 119 A1: the read doc is the turn's ONLY chip.
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(1)
|
||||
# Chat chips carry back=/ (encoded %2F) so the viewer knows where
|
||||
# home is. Phase 26: the left click opens the same-page modal (no
|
||||
# target=_blank); this href is what the no-JS / context-menu "open
|
||||
|
||||
@@ -540,12 +540,14 @@ def test_old_correct_beats_new_similar(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None, dates_tree: Path
|
||||
) -> None:
|
||||
"""``How did I configure the backup retention policy?`` → grounded,
|
||||
and the FIRST cited source is the OLDER correct doc (2020) — the
|
||||
NEWER similar one (now, "under review") is cited second. Phase 118
|
||||
(A4): the chip row IS the suggested tier (top-5, NO floor) — this
|
||||
four-doc KB chips ALL four docs in rank order — and the related row
|
||||
is absent (no rank-6+ doc). The real hybrid retriever + the DEFAULT
|
||||
recency boost (0.0007 / 365 d) over the mock's token-overlap
|
||||
and the FIRST logged source is the OLDER correct doc (2020) — the
|
||||
NEWER similar one (now, "under review") is second. Phase 119
|
||||
(LOCKED A1): the zero-read turn chips NOTHING (the chip row is the
|
||||
READ docs only — the retired phase-118 A4 suggested-chip row is
|
||||
gone), so the ordering assertion rides the durable record (LOCKED
|
||||
A3 — all four docs, suggested rank order, untouched); the related
|
||||
row is absent (no rank-6+ doc). The real hybrid retriever + the
|
||||
DEFAULT recency boost (0.0007 / 365 d) over the mock's token-overlap
|
||||
embeddings (the module docstring records the measured fused scores:
|
||||
0.032523 vs 0.031498 — margin ≈ 0.001025 WITH the full zero-age
|
||||
boost on the newer doc)."""
|
||||
@@ -563,21 +565,20 @@ def test_old_correct_beats_new_similar(
|
||||
# top_score 0.6222 ≥ the e2e threshold 0.30).
|
||||
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
|
||||
|
||||
# Phase 118 (A4): the chip row IS the suggested tier — top-5, NO
|
||||
# floor, and this KB has exactly four docs, so ALL four are
|
||||
# suggested (chipped) in fused rank order: the OLDER correct doc
|
||||
# first, the NEWER similar one (the boost's intended beneficiary)
|
||||
# second, then the two unrelated docs (no floor filters them — the
|
||||
# LLM decides what the summaries earn).
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(4, timeout=30_000)
|
||||
assert chips.nth(0).inner_text() == f"{dates_tree.name}/{RETENTION_MD}"
|
||||
assert chips.nth(1).inner_text() == f"{dates_tree.name}/{DRAFT_MD}"
|
||||
assert chips.nth(2).inner_text() == f"{dates_tree.name}/{FORWARD_MD}"
|
||||
assert chips.nth(3).inner_text() == f"{dates_tree.name}/{OLDDOC_MD}"
|
||||
# Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation
|
||||
# chips — the four suggested docs (all of them, top-5 NO floor on a
|
||||
# four-doc KB: the OLDER correct doc first, the NEWER similar one
|
||||
# second, then the two unrelated docs) seeded the prompt but never
|
||||
# chip (the retired phase-118 A4 union is gone). Their rank order
|
||||
# is pinned by the durable record below (LOCKED A3, untouched).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
# No rank-6+ doc exists in this four-doc KB → the related row is
|
||||
# absent (the de-emphasized row renders only when it has entries).
|
||||
expect(page.locator(".msg.brain .related-docs")).to_have_count(0)
|
||||
# The button label is the settle sync: the client re-labels Send on
|
||||
# the done frame, and the server writes the query_log row just
|
||||
# before yielding it.
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
# Durable record: one row, grounded, the FULL retrieval (suggested
|
||||
# tier + related + read, deduped — here: all four docs) in rank
|
||||
|
||||
@@ -171,8 +171,9 @@ def test_summary_hit_seeds_the_summary_not_the_full_text(
|
||||
quotes the last suggested doc's summary tail (digest + pointer), and
|
||||
the sentinel (the yaml doc's last line, outside the digest) is
|
||||
ABSENT: the full source doc never reached the prompt (A6). Both
|
||||
fixture docs are suggested (two-doc KB, no floor) and both chips
|
||||
render (deflected: false)."""
|
||||
fixture docs are suggested (two-doc KB, no floor) but chip
|
||||
NOTHING — the turn read nothing, so (phase 119, LOCKED A1) the
|
||||
citation surface is empty (deflected: false)."""
|
||||
_reset_db()
|
||||
summary = _run_in_thread(_import_fixtures(mock_llm))
|
||||
assert summary.added == 2 # yaml + md control
|
||||
@@ -225,13 +226,13 @@ def test_summary_hit_seeds_the_summary_not_the_full_text(
|
||||
expect(bubble).to_contain_text(yaml_digest_line[-80:])
|
||||
expect(bubble).not_to_contain_text(SENTINEL)
|
||||
|
||||
# Grounded: both fixture docs are suggested (two-doc KB — no floor)
|
||||
# and both chips render, in rank order (md first, yaml last).
|
||||
chip = page.locator(".msg.brain .source-chip", has_text=YAML_PATH)
|
||||
expect(chip).to_have_count(1)
|
||||
expect(chip.first).to_contain_text(f"{SOURCE}/{YAML_PATH}")
|
||||
expect(page.locator(".msg.brain .source-chip", has_text=MD_PATH)).to_have_count(1)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(2)
|
||||
# Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation
|
||||
# chips — both fixture docs are suggested (two-doc KB — no floor,
|
||||
# md first, yaml last) but suggested docs are seed context, not
|
||||
# citations (the retired phase-118 A4 suggested-chip union is gone);
|
||||
# their rank order is pinned by the durable record below (LOCKED
|
||||
# A3, untouched).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
# No rank-6+ doc in a two-doc KB → the related row is absent.
|
||||
expect(page.locator(".msg.brain .related-docs")).to_have_count(0)
|
||||
|
||||
|
||||
@@ -11,9 +11,18 @@ deterministic mock embeddings (same harness as the phase-10 suite — only
|
||||
the assertions changed: chips/row links now open the SAME-PAGE modal,
|
||||
no ``expect_popup``).
|
||||
|
||||
Phase 119 re-target (LOCKED A1): chips cite READ docs only — a plain
|
||||
question chips nothing, so the chat-driven tests here run the
|
||||
mock's scripted summary-read flow (``SUMMARY_SEED_READ_TRIGGER`` —
|
||||
the house scripted-turn convention): the turn ``read``s
|
||||
``docs/homelab/kubernetes.md``, whose read result is what earns the
|
||||
kubernetes.md chip the story interacts with (the chip's viewer-link
|
||||
contract is unchanged).
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule):
|
||||
1. ``test_source_chip_opens_modal`` — chat chip → modal opens in-page
|
||||
(NO new tab, URL unchanged), title + ``.doc-md`` content + meta row.
|
||||
1. ``test_source_chip_opens_modal`` — chat chip (the turn's READ doc —
|
||||
phase 119 A1) → modal opens in-page (NO new tab, URL unchanged),
|
||||
title + ``.doc-md`` content + meta row.
|
||||
2. ``test_sources_row_opens_modal`` — Sources path link → modal, yaml in
|
||||
``<pre.doc-raw>``, mono font, URL unchanged.
|
||||
3. ``test_modal_closes_on_button_escape_and_backdrop`` — close via
|
||||
@@ -56,7 +65,14 @@ from e2e.auth_helpers import login
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "docs"
|
||||
SCREENSHOTS = REPO / ".agents" / "screenshots" # house convention for visual records
|
||||
QUESTION = "How is my Kubernetes cluster set up?"
|
||||
# Phase 119 (LOCKED A1): the chip is the turn's READ doc — the scripted
|
||||
# summary-read flow (mock_llm.SUMMARY_SEED_READ_TRIGGER) reads the
|
||||
# kubernetes fixture so the story's chip exists (the retired phase-118
|
||||
# A4 suggested-chip is gone: a zero-read turn would chip nothing).
|
||||
QUESTION = (
|
||||
"Read the suggested document: read docs/homelab/kubernetes.md — "
|
||||
"how is my Kubernetes cluster set up?"
|
||||
)
|
||||
|
||||
|
||||
async def _import_fixtures(mock_port: int) -> ImportSummary:
|
||||
@@ -98,12 +114,20 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
|
||||
|
||||
|
||||
def _ask_for_chip(page: Page, app_url: str) -> Any:
|
||||
"""Drive one chat turn and return the kubernetes.md source chip."""
|
||||
"""Drive one chat turn and return the kubernetes.md source chip.
|
||||
|
||||
The scripted summary-read flow (phase 119, LOCKED A1): the turn
|
||||
``read``s the kubernetes doc, so it is the turn's ONLY citation
|
||||
chip (chips cite read docs only — the answer is the mock's
|
||||
verbatim echo of the read result)."""
|
||||
login(page, app_url, next="/") # phase 79: chat is require_user-gated
|
||||
page.fill("#message-input", QUESTION)
|
||||
page.click("#send-btn")
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
expect(chip).to_have_count(1, timeout=30_000)
|
||||
# Phase 119 A1: the read doc is the turn's ONLY chip (exactly one
|
||||
# .source-chip under the bubble — nothing suggested-but-unread).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(1)
|
||||
return chip
|
||||
|
||||
|
||||
|
||||
@@ -455,10 +455,10 @@ def test_read_flow_lines_answer_sources_no_raw_markup(
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
# Done-state sources include the read document (retrieval doc first,
|
||||
# the agent's read doc after — the phase-37 extension contract).
|
||||
# Phase 119 (LOCKED A1): done.sources = the READ docs only —
|
||||
# exactly the scripted read; the retrieval (seed) doc was never
|
||||
# read, so it never cites (the retired phase-118 A4 union is gone).
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
(SEED_SOURCE, SEED_PATH),
|
||||
(READ_SOURCE, READ_PATH),
|
||||
]
|
||||
|
||||
|
||||
@@ -23,10 +23,12 @@ assistant message — A4).
|
||||
|
||||
The marker is checked BEFORE the mock's ``DEFLECT_MODE`` branch, so
|
||||
the echo fires on BOTH turn branches — the branch under test is
|
||||
discriminated separately (the grounded source chip / the
|
||||
``is-deflected`` bubble class). The echo answers carry no tool
|
||||
markup, so no marker tool flow is re-triggered by the now-always-
|
||||
present (user/assistant-only) history.
|
||||
discriminated separately (the persisted record's ``deflected`` flag /
|
||||
the ``is-deflected`` bubble class — phase 119, LOCKED A1: a zero-read
|
||||
grounded turn chips nothing, so the retired chip discriminator is
|
||||
gone). The echo answers carry no tool markup, so no marker tool flow
|
||||
is re-triggered by the now-always-present (user/assistant-only)
|
||||
history.
|
||||
|
||||
The file name deliberately differs from phase 50's
|
||||
``test_chat_history.py`` (save & view chat history — a different
|
||||
@@ -170,12 +172,15 @@ def test_followup_receives_history_and_thinking(
|
||||
expect(bubble).to_contain_text(f"last answer tail: {answer_tail}")
|
||||
expect(bubble).to_contain_text("thinking: yes")
|
||||
# Grounded proof — the echo fires in BOTH branches, so the branch
|
||||
# is discriminated by the kubernetes.md source chip (the deflected
|
||||
# turn carries no cited sources). Scoped to the LAST brain message:
|
||||
# turn 1 cited kubernetes.md too.
|
||||
# is discriminated by the persisted record's ``deflected`` flag
|
||||
# (phase 119, LOCKED A1: the turn read nothing, so it also chips
|
||||
# nothing — the retired chip discriminator is gone).
|
||||
brain2 = _wait_record(page, 4)["messages"][3]
|
||||
assert brain2["who"] == "brain"
|
||||
assert brain2["deflected"] is False, "the echo turn must be the grounded branch"
|
||||
expect(
|
||||
page.locator(".msg.brain").last.locator(".source-chip", has_text="kubernetes.md")
|
||||
).to_have_count(1)
|
||||
page.locator(".msg.brain").last.locator(".source-chip")
|
||||
).to_have_count(0)
|
||||
|
||||
|
||||
def test_first_question_has_no_history(
|
||||
@@ -194,11 +199,15 @@ def test_first_question_has_no_history(
|
||||
expect(bubble).to_contain_text("history: 0 prior messages", timeout=30_000)
|
||||
expect(bubble).to_contain_text("last answer tail: none")
|
||||
expect(bubble).to_contain_text("thinking: no")
|
||||
# Grounded: the echo question is on-topic (the chip proves the
|
||||
# HIGH gate, not a deflection).
|
||||
# Grounded: the echo question is on-topic (the record's deflected
|
||||
# flag proves the HIGH gate, not a deflection — phase 119, LOCKED
|
||||
# A1: the zero-read turn chips nothing, so the retired chip
|
||||
# discriminator is gone).
|
||||
record = _wait_record(page, 2)
|
||||
assert record["messages"][1]["deflected"] is False
|
||||
expect(
|
||||
page.locator(".msg.brain").last.locator(".source-chip", has_text="kubernetes.md")
|
||||
).to_have_count(1)
|
||||
page.locator(".msg.brain").last.locator(".source-chip")
|
||||
).to_have_count(0)
|
||||
|
||||
|
||||
def test_deflected_followup_receives_history(
|
||||
|
||||
+12
-12
@@ -393,18 +393,18 @@ def test_dead_then_recovered_grounded(
|
||||
_assert_no_error_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
(SEED_SOURCE, SEED_PATH)
|
||||
]
|
||||
# Phase 119 (LOCKED A1): the grounded turn read NOTHING, so
|
||||
# done.sources is empty (the suggested seed doc never chips — the
|
||||
# retired phase-118 A4 union is gone); the durable record below
|
||||
# still carries it (LOCKED A3, untouched).
|
||||
assert done["sources"] == [], done["sources"]
|
||||
|
||||
# The grounded answer completed with the source chip — the agent
|
||||
# round retried and the turn is intact.
|
||||
# The grounded answer completed (no chips on a zero-read turn —
|
||||
# phase 119 A1) — the agent round retried and the turn is intact.
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(GROUNDED_Q)
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(1)
|
||||
expect(chips.first).to_contain_text(SEED_SP)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
_assert_no_error_banner(page)
|
||||
|
||||
row = _last_query_log()
|
||||
@@ -446,12 +446,12 @@ def test_embedding_retry_completes(
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
|
||||
# The turn completed normally with the grounded answer + chip.
|
||||
# The turn completed normally with the grounded answer (zero chips
|
||||
# — the turn read nothing, so the chip row is empty under phase
|
||||
# 119, LOCKED A1; the retired phase-118 A4 union is gone).
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(1)
|
||||
expect(chips.first).to_contain_text(SEED_SP)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
_assert_no_error_banner(page)
|
||||
|
||||
|
||||
|
||||
@@ -127,8 +127,8 @@ def test_normal_answer_unaffected(
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text("Deterministic mock answer for E2E", timeout=30_000)
|
||||
expect(bubble).not_to_contain_text("LONG-ANSWER-END")
|
||||
# Grounded: the question's own document is cited as a chip.
|
||||
expect(
|
||||
page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
).to_have_count(1, timeout=30_000)
|
||||
# Grounded (the bubble + marker above) — and ZERO citation chips:
|
||||
# the turn read nothing, so (phase 119, LOCKED A1) the chip row is
|
||||
# empty (the retired phase-118 A4 suggested-chip is gone).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
expect(page.locator("#send-btn")).to_be_enabled()
|
||||
|
||||
@@ -678,8 +678,8 @@ def test_drill_down_sources_folders_files_and_read(
|
||||
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# The read document is in the turn's sources (retrieval + agent-read,
|
||||
# deduped — the grounded-turn record).
|
||||
# The read document is in the turn's sources (phase 119, LOCKED A1:
|
||||
# done.sources is the READ docs only — this turn's one read).
|
||||
assert any(
|
||||
s["path"] == TWO_A and s["source"] == ALPHA for s in done["sources"]
|
||||
), done["sources"]
|
||||
|
||||
@@ -185,9 +185,11 @@ def test_chat_table_renders(
|
||||
assert "|---|" not in bubble_text, "the |---| separator leaked into the bubble"
|
||||
assert "| Service | Port | Host |" not in bubble_text, "the raw header row leaked"
|
||||
|
||||
# Grounded retrieval: the table fixture is the top source chip.
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="homelab/tables.md")
|
||||
expect(chip).to_have_count(1)
|
||||
# Grounded (the table answer is the mock's non-deflected table
|
||||
# branch) — and ZERO citation chips: the turn read nothing, so
|
||||
# (phase 119, LOCKED A1) the chip row is empty (the retired
|
||||
# phase-118 A4 suggested-chip is gone).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,794 @@
|
||||
"""Phase 119 E2E (Playwright, mock-only): the name-signal seeds the
|
||||
right document — and chips cite READ docs only.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_name_signal_read_chips.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: the story's gates are the deterministic contracts —
|
||||
the mock's scripted-turn lenses (the tail echo, the single-read tool
|
||||
flow, the new ``FOLDER_MAP_TRIGGER`` echo) make "what reached the
|
||||
prompt / what got cited" assertable byte-exactly.
|
||||
|
||||
KB fixture — ``tests/fixtures/namekb`` (tracked; registered as a
|
||||
local-directory source through the authenticated API + the REAL
|
||||
in-process ``POST /api/sync``, the ``test_summary_seed_context.py``
|
||||
pattern), seven markdown documents in one source (``namekb``) whose
|
||||
bodies are token-controlled so the hybrid gate + the phase-119
|
||||
name-hit bonus pick the intended tiers deterministically
|
||||
(``synced_kb`` pins it with the app's REAL retrieval — a
|
||||
fixture-text regression that breaks the design fails at setup with a
|
||||
clear message, not at the wire assertions):
|
||||
|
||||
* every path carries a DIGITLESS question name token (the four
|
||||
guides' stems hold ``install``, the two forgejo docs' stems +
|
||||
folder hold ``forgejo``, ``zz-folder-map.md`` holds ``folder``) —
|
||||
so the name-hit bonus applies to ALL seven docs (a uniform
|
||||
``+0.005``) and the tier order reduces to the pure retrieval
|
||||
signal (the bonus head-start is exercised, the ordering is not
|
||||
distorted by it);
|
||||
* the four guides' bodies repeat the question phrase a strength
|
||||
gradient (×4/×3/×2/×1) so they OUT-RANK the name-hit docs on
|
||||
vector/FTS; the forgejo docs and the filler doc carry NEUTRAL
|
||||
bodies (their name is their main signal — the phase-119 owner
|
||||
scenario: "the file's name is the distinctive part");
|
||||
* engineered invariant (asserted in ``synced_kb`` for ALL FOUR
|
||||
scripted questions, measured + probe-verified, stable across
|
||||
re-imports): the name-hit doc ``forgejo/forgejo-home.md`` is the
|
||||
LAST of the top-5 suggested (the name signal lifts it into the
|
||||
tier, the weak body keeps it at the back — its ``Source:`` tail
|
||||
line is the tail echo), and the related tier is the rank-6+
|
||||
remainder (``forgejo-nginx`` + ``zz-folder-map``; for the folder
|
||||
question the two swap order — both stay rank 6+).
|
||||
|
||||
Test → contract mapping (task 06 cases a–d; one Playwright file per
|
||||
story, A16):
|
||||
1. ``test_name_hit_doc_is_last_suggested`` — (a): the distinctive
|
||||
question + ``show the end of your notes`` ⇒ the answer quotes the
|
||||
last 160 chars of the seeded ``<documents>`` block — the
|
||||
name-hit doc's SUMMARY tail (its ``Source: namekb/forgejo/
|
||||
forgejo-home.md`` pointer line; no other doc's pointer line, no
|
||||
doc's tail sentinel — the summary seed, not the full text). The
|
||||
zero-read grounded turn chips NOTHING (LOCKED A1); the related
|
||||
row renders rank 6+; the durable record carries suggested +
|
||||
related (118-A3 untouched).
|
||||
2. ``test_single_read_chips_only_the_read_doc`` — (b): the
|
||||
``use your tools`` flow (the mock's single read: ``ls`` → drill
|
||||
``ls(namekb)`` → read the FIRST file line —
|
||||
``namekb/zz-folder-map.md``, the root file) ⇒ the bubble carries
|
||||
EXACTLY ONE ``.source-chip`` — the read doc (LOCKED A1); none of
|
||||
the five suggested docs chips; the de-emphasized related row
|
||||
renders (rank 6+ deduped against the cited read doc — the read
|
||||
doc is a chip, never a "nearby doc"); the durable record carries
|
||||
suggested + related + read (118-A3 untouched).
|
||||
3. ``test_zero_read_grounded_turn_chips_nothing`` — (c): a PLAIN
|
||||
distinctive question (no tool trigger) ⇒ the ``done`` frame's
|
||||
``sources == []`` and ZERO ``.source-chip`` elements in the
|
||||
bubble — LOCKED A1's visible consequence; the answer still
|
||||
renders grounded (deflected: false); the related row renders.
|
||||
4. ``test_folder_lines_echoed_verbatim`` — (d): the distinctive
|
||||
question + ``repeat your folder map`` (the new mock trigger) ⇒
|
||||
the answer ends with the suggested-folder lines VERBATIM — the
|
||||
name-hit doc's folder line ``namekb/forgejo/: forgejo-nginx.md``
|
||||
(the line prefix + its known sibling entry) alongside the
|
||||
guides' folder line.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Locator, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document, QueryLog
|
||||
from app.rag.retriever import TRUNCATION_MARKER, retrieve, select_related, select_suggested
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
from e2e.mock_llm import TOKEN_RE, embed_text
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||
# port in a combined run — this module app binds its own port instead
|
||||
# (a same-port second uvicorn dies on bind and would drive the wrong
|
||||
# server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_NAMESIG", "8145"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
SOURCE = "namekb" # the local directory's basename = the source name
|
||||
FIXTURES = REPO / "tests" / "fixtures" / "namekb"
|
||||
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixture documents (tracked — tests/fixtures/namekb; deterministic,
|
||||
# token-controlled — see the module docstring for the design)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: 22 neutral tokens (no question tokens) — the digest + filler
|
||||
#: material of the neutral docs; the guides' strength phrase is
|
||||
#: question-token-only.
|
||||
PRELUDE = (
|
||||
"fixture preamble block one two three four five six seven eight nine ten "
|
||||
"eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen"
|
||||
)
|
||||
assert len(TOKEN_RE.findall(PRELUDE.lower())) == 22
|
||||
PHRASE = "install forgejo with the deployment steps"
|
||||
|
||||
#: (path, title, phrase repeats, filler line, tail sentinel) — the
|
||||
#: guides carry the strength gradient (×4/×3/×2/×1); the forgejo docs
|
||||
#: (name hits — distinctive digitless ``forgejo`` component) and the
|
||||
#: root filler carry NEUTRAL bodies (0 repeats).
|
||||
DOCS: list[tuple[str, str, int, str, str]] = [
|
||||
("deploy/install-guide-a.md", "Install Forgejo Guide A", 4,
|
||||
"alpha1 alpha2 alpha3 alpha4 alpha5 alpha6", "NAMEKB-TAIL-c5d6"),
|
||||
("deploy/install-guide-b.md", "Install Forgejo Guide B", 3,
|
||||
"beta1 beta2 beta3 beta4 beta5 beta6", "NAMEKB-TAIL-d7e8"),
|
||||
("deploy/install-guide-c.md", "Install Forgejo Guide C", 2,
|
||||
"gamma1 gamma2 gamma3 gamma4 gamma5 gamma6", "NAMEKB-TAIL-f9a0"),
|
||||
("deploy/install-guide-d.md", "Install Forgejo Guide D", 1,
|
||||
"delta1 delta2 delta3 delta4 delta5 delta6", "NAMEKB-TAIL-b1c2"),
|
||||
("forgejo/forgejo-home.md", "Zeta Forge One", 0,
|
||||
"kilo1 kilo2 kilo3 kilo4 kilo5 kilo6", "NAMEKB-TAIL-a1b2"),
|
||||
("forgejo/forgejo-nginx.md", "Zeta Forge Two", 0,
|
||||
"lambda1 lambda2 lambda3 lambda4 lambda5 lambda6", "NAMEKB-TAIL-b3c4"),
|
||||
("zz-folder-map.md", "Zeta Misc Three", 0,
|
||||
"micro1 micro2 micro3 micro4 micro5 micro6", "NAMEKB-TAIL-e3f4"),
|
||||
]
|
||||
|
||||
#: The engineered invariant (measured, probe-verified, stable across
|
||||
#: re-imports — pinned for ALL FOUR scripted questions): the name-hit
|
||||
#: doc is the LAST of the top-5 suggested; the guides lead in
|
||||
#: retrieval-strength order (the ×2/×1 docs swap under the md5
|
||||
#: collision noise — pinned as measured, not as the gradient order).
|
||||
SUGGESTED = [
|
||||
"deploy/install-guide-a.md",
|
||||
"deploy/install-guide-b.md",
|
||||
"deploy/install-guide-d.md",
|
||||
"deploy/install-guide-c.md",
|
||||
"forgejo/forgejo-home.md",
|
||||
]
|
||||
NAME_HIT = "forgejo/forgejo-home.md" # the (a) tail-echo target
|
||||
RELATED = ["forgejo/forgejo-nginx.md", "zz-folder-map.md"]
|
||||
#: The folder question's rank-6+ order (the filler's name hit —
|
||||
#: ``folder`` — lifts it above the sibling; both stay rank 6+).
|
||||
RELATED_FOLDER = ["zz-folder-map.md", "forgejo/forgejo-nginx.md"]
|
||||
READ_TARGET = "zz-folder-map.md" # the single-read's first file line
|
||||
SENTINELS = [sentinel for _p, _t, _i, _f, sentinel in DOCS]
|
||||
ALL_SOURCE_LINES = [f"Source: {SOURCE}/{p}" for p, _t, _i, _f, _s in DOCS]
|
||||
|
||||
#: The turn's questions (the mock's trigger phrases — see the module
|
||||
#: docstring). The base question's name tokens (``install``,
|
||||
#: ``forgejo``) ground every turn (best cosine ≈ 0.35–0.41 ≥ the E2E
|
||||
#: 0.30 threshold) and fire the D1 name hits.
|
||||
BASE_QUESTION = "How do I install forgejo?"
|
||||
TAIL_QUESTION = BASE_QUESTION + " show the end of your notes"
|
||||
TOOLS_QUESTION = BASE_QUESTION + " use your tools"
|
||||
FOLDER_QUESTION = BASE_QUESTION + " repeat your folder map"
|
||||
assert "show the end of your notes" in TAIL_QUESTION.lower()
|
||||
assert "use your tools" in TOOLS_QUESTION.lower()
|
||||
assert "repeat your folder map" in FOLDER_QUESTION.lower()
|
||||
|
||||
|
||||
def _expected_summary(content: str, path: str) -> str:
|
||||
"""The mock lite model's byte-stable digest + the code pointer line
|
||||
(mirrors ``mock_llm.compose_answer``'s ``SUMMARY_MODE`` branch —
|
||||
first 24 tokens of the document content — plus the summarizer's
|
||||
deterministic ``Source:`` line; no model output is ever trusted)."""
|
||||
digest = " ".join(TOKEN_RE.findall(content.lower())[:24])
|
||||
return f"This document covers {digest}.\nSource: {SOURCE}/{path}"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seed_dirs() -> Path:
|
||||
"""The story's local-directory source: the tracked fixture dir
|
||||
(the app server runs on the same host, so the path is visible to
|
||||
it). The directory's basename is the source name (``kind=local``,
|
||||
phase 38)."""
|
||||
assert FIXTURES.is_dir(), "tests/fixtures/namekb is missing"
|
||||
for path, _t, _i, _f, _s in DOCS:
|
||||
assert (FIXTURES / path).is_file(), f"fixture doc {path} is missing"
|
||||
return FIXTURES
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, seed_dirs: Path) -> Iterator[str]:
|
||||
"""The real app under test — per-module app (the conftest pattern,
|
||||
cf. ``test_summary_seed_context.py``): NO ``BOR_GIT_SOURCES`` (the
|
||||
env fallback is git-only — the source here is a DB-registered local
|
||||
directory), the mock LLM, the mock-calibrated threshold, and the
|
||||
leak-guarded code defaults (the suggested/related tier settings
|
||||
ride their code defaults — 5 / 2 — exactly like the production
|
||||
``.env``-free defaults). The session app is never started in this
|
||||
isolated run, so no port clash."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (conftest pattern): the strength gradient
|
||||
# keeps every scripted turn grounded (best cosine ≈ 0.35–0.41).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits + the code-default budget (the
|
||||
# conftest leak-guard pattern).
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
||||
# reads it from cwd) — override it with an EMPTY value (the env var
|
||||
# beats the .env file): the registry must hold EXACTLY the local
|
||||
# directory this suite registers (a leftover env git list would
|
||||
# pollute the KB the scripted turns run against).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
|
||||
share one Postgres, so a leftover git_sources row or document would
|
||||
pollute the retrieval the scripted turns run against (the
|
||||
name-signal margins are pinned against EXACTLY these seven
|
||||
documents)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
|
||||
"""Poll the (cookie-authenticated) status endpoint until the run
|
||||
reaches a terminal state (the test_summary_seed_context pattern,
|
||||
over plain httpx — this fixture has no browser page yet)."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
def _assert_tiers(
|
||||
question: str, related_paths: list[str]
|
||||
) -> None:
|
||||
"""Pin the name-signal design with the app's REAL hybrid retrieval
|
||||
over the mock's embeddings (deterministic): the suggested tier is
|
||||
exactly the four guides (retrieval-strength order) + the name-hit
|
||||
doc LAST (LOCKED A3 — top-5, NO floor — the name-hit bonus, D2,
|
||||
lifts it into the tier) and the related tier is the rank-6+
|
||||
remainder (``related_max_docs`` = 2). A fixture-text regression
|
||||
that breaks the design fails here, at setup, with a clear
|
||||
message."""
|
||||
with SessionLocal() as db:
|
||||
chunks = retrieve(db, question, embed_text(question))
|
||||
suggested = [f"{d.source}/{d.path}" for d in select_suggested(chunks)]
|
||||
related = [
|
||||
f"{d.source}/{d.path}"
|
||||
for d in select_related(
|
||||
chunks,
|
||||
{d.id for d in select_suggested(chunks)},
|
||||
_Settings.model_fields["related_max_docs"].default,
|
||||
)
|
||||
]
|
||||
expected = [f"{SOURCE}/{p}" for p in SUGGESTED]
|
||||
assert suggested == expected, (
|
||||
f"suggested tier drifted for {question!r}: {suggested} "
|
||||
f"(expected {expected})"
|
||||
)
|
||||
assert related == [f"{SOURCE}/{p}" for p in related_paths], (
|
||||
f"related tier drifted for {question!r}: {related}"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def synced_kb(app_server: str, seed_dirs: Path) -> None:
|
||||
"""The story's precondition: the one-source KB synced under the
|
||||
deterministic mock. Registers the fixture directory through the
|
||||
authenticated API (the ``test_local_directory_sources.py``
|
||||
pattern), runs the REAL in-process sync (``POST /api/sync`` — walk
|
||||
→ chunk → embed → summaries → overview → folder summaries →
|
||||
version bump), pins the stored content byte-identical to the
|
||||
tracked fixture files, pins LOCKED A2 end-to-end (every doc stores
|
||||
the mock's byte-stable digest + exactly one embedded
|
||||
``is_summary`` chunk), and pins the engineered invariant (the
|
||||
name-hit doc LAST of the top-5) for ALL FOUR scripted questions."""
|
||||
_truncate_all()
|
||||
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": str(seed_dirs)}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
body = _wait_sync_done_http(client)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == len(DOCS), detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
# The import stored the tracked fixture strings BYTE-IDENTICALLY
|
||||
# and, for EVERY doc, the mock's byte-stable digest: the
|
||||
# deterministic assertion surface of the whole suite.
|
||||
with SessionLocal() as db:
|
||||
for path, _title, _i, _f, _s in DOCS:
|
||||
on_disk = (FIXTURES / path).read_text(encoding="utf-8")
|
||||
stored = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE, Document.path == path
|
||||
)
|
||||
)
|
||||
assert stored is not None, f"fixture doc {path} was not imported"
|
||||
assert stored.content == on_disk, f"stored content drifted for {path}"
|
||||
assert stored.summary == _expected_summary(on_disk, path), (
|
||||
f"summary is not the mock digest for {path}: {stored.summary!r}"
|
||||
)
|
||||
schunks = [c for c in stored.chunks if c.is_summary]
|
||||
assert len(schunks) == 1 and schunks[0].position == -1, (
|
||||
f"expected exactly one is_summary chunk for {path}"
|
||||
)
|
||||
assert schunks[0].embedding is not None, (
|
||||
f"the is_summary chunk of {path} is not embedded"
|
||||
)
|
||||
_assert_tiers(TAIL_QUESTION, RELATED)
|
||||
_assert_tiers(TOOLS_QUESTION, RELATED)
|
||||
_assert_tiers(FOLDER_QUESTION, RELATED_FOLDER)
|
||||
_assert_tiers(BASE_QUESTION, RELATED)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
"""Per-test query_log isolation (the KB itself is module-scoped —
|
||||
the scripted turns never change it, so the registry and the KB
|
||||
persist across the tests of this module)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the test_summary_seed_context house patterns)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: for the ``tool`` / ``done`` frames, independent of the UI rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _frames(page: Page) -> list[dict]:
|
||||
"""The SSE frames captured since the last submit (``_submit``
|
||||
clears the buffer), once the hook's background read settles."""
|
||||
deadline = time.monotonic() + 30.0
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.evaluate("window.__sseFrames = []")
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button recovered
|
||||
(the phase-48 settle wait)."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
|
||||
def _last_brain(page: Page) -> Locator:
|
||||
return page.locator(".msg.brain").last
|
||||
|
||||
|
||||
def _last_query_log() -> QueryLog:
|
||||
with SessionLocal() as db:
|
||||
rows = db.scalars(select(QueryLog)).all()
|
||||
assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}"
|
||||
return rows[0]
|
||||
|
||||
|
||||
def _assert_chip_row(page: Page, read_paths: list[str]) -> None:
|
||||
"""The citation surface of a grounded turn (phase 119, LOCKED A1 —
|
||||
the retired phase-118 A4 union is gone): the chip row is the
|
||||
AGENT-READ docs only (a zero-read turn chips NOTHING: the
|
||||
suggested docs are seed context, not citations)."""
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(len(read_paths))
|
||||
for path in read_paths:
|
||||
expect(
|
||||
chips.filter(has_text=path), message=f"chip for {path}"
|
||||
).to_have_count(1)
|
||||
|
||||
|
||||
def _assert_related_row(page: Page, related_paths: list[str]) -> None:
|
||||
"""The de-emphasized ``related-docs`` row (phase-113 UI, untouched
|
||||
by phase 119 — never a citation chip): the rank-6+ remainder
|
||||
deduped against the cited read docs (a read related doc is a chip,
|
||||
never a "nearby doc")."""
|
||||
row = page.locator(".msg.brain .related-docs")
|
||||
expect(row).to_have_count(1)
|
||||
expect(row.first).to_have_attribute("aria-label", "Nearby docs, in case")
|
||||
links = page.locator(".msg.brain .related-docs .related-doc")
|
||||
expect(links).to_have_count(len(related_paths))
|
||||
for i, path in enumerate(related_paths):
|
||||
expect(links.nth(i)).to_contain_text(f"{SOURCE}/{path}")
|
||||
# The related links are never citation chips.
|
||||
expect(page.locator(".msg.brain .related-docs .source-chip")).to_have_count(0)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. (a) the name-signal doc is the LAST of the top-5 suggested — the
|
||||
# tail echo proves it reached the prompt as the last seed block
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_name_hit_doc_is_last_suggested(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The tail-echo question quotes the last 160 chars of the seeded
|
||||
``<documents>`` block: they end in the LAST suggested doc's SUMMARY
|
||||
— the name-hit doc's byte-stable digest tail + pointer line (the
|
||||
digitless-component rule + the D2 bonus put ``forgejo-home`` in
|
||||
the tier, its neutral body keeps it last). No other doc's pointer
|
||||
line and no doc's tail sentinel are in the echoed context — the
|
||||
summary seed, not the full text. The zero-read grounded turn
|
||||
chips nothing (LOCKED A1); the related row renders rank 6+; the
|
||||
durable record carries suggested + related (118-A3 untouched)."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, TAIL_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# No tools on this turn — the answer is the mock's direct tail echo
|
||||
# of the seeded context (the summary-seed lens).
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# Phase 119 (LOCKED A1): the grounded done frame cites the READ
|
||||
# docs only — this turn read NOTHING (the summary-only fast path),
|
||||
# so the citation surface is empty; the suggested set (name-hit
|
||||
# doc included) is seed context, not citations.
|
||||
assert done["sources"] == [], done["sources"]
|
||||
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
|
||||
# The echoed tail ends in the name-hit doc's summary: its
|
||||
# deterministic ``Source:`` pointer line (the digest is pinned in
|
||||
# ``synced_kb`` — a content preview would carry the neutral-body
|
||||
# text instead, and the pointer line only ever exists on a stored
|
||||
# summary). The bubble renders the answer as markdown, which
|
||||
# collapses the summary's newline — pin the pointer line alone
|
||||
# (it sits inside the echoed 160 chars, byte-exact).
|
||||
expect(bubble).to_contain_text(f"Source: {SOURCE}/{NAME_HIT}")
|
||||
# No OTHER doc's pointer line is in the echoed tail — the
|
||||
# name-hit doc was the LAST suggested block (the (a) invariant's
|
||||
# visible proof).
|
||||
for path, _t, _i, _f, _s in DOCS:
|
||||
if path != NAME_HIT:
|
||||
expect(bubble).not_to_contain_text(f"Source: {SOURCE}/{path}")
|
||||
# And no document's tail sentinel: the full content of no
|
||||
# suggested doc reached the model (summary seed only).
|
||||
for sentinel in SENTINELS:
|
||||
expect(bubble).not_to_contain_text(sentinel)
|
||||
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
|
||||
|
||||
# The zero-read grounded turn chips NOTHING (LOCKED A1); the
|
||||
# related row renders rank 6+ (the de-emphasized row, untouched).
|
||||
_assert_chip_row(page, [])
|
||||
_assert_related_row(page, RELATED)
|
||||
|
||||
# Durable record: grounded; suggested + related (LOCKED A3 — the
|
||||
# log records retrieval, not citations; phase 119 A1 retires the
|
||||
# chip surface only, not the record).
|
||||
row = _last_query_log()
|
||||
assert row.question == TAIL_QUESTION
|
||||
assert row.deflected is False
|
||||
assert row.sources == ", ".join(f"{SOURCE}/{p}" for p in SUGGESTED + RELATED)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. (b) chips = READ docs only — the single-read flow chips exactly
|
||||
# the one read doc; the related row renders (deduped)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_single_read_chips_only_the_read_doc(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The ``use your tools`` flow drives the mock's single read:
|
||||
``ls`` (the top level) → drill ``ls(namekb)`` → ``read`` the FIRST
|
||||
file line (the root file ``zz-folder-map.md`` — subfolders list
|
||||
first, files after). The bubble carries EXACTLY ONE
|
||||
``.source-chip`` — the read doc (LOCKED A1); none of the five
|
||||
suggested docs chips (the name-hit doc is seed context, not a
|
||||
citation); the de-emphasized related row renders the rank-6+
|
||||
remainder deduped against the cited read doc (the read doc is a
|
||||
chip, never a "nearby doc")."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, TOOLS_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# Wire level: exactly three `tool` frames — ``ls`` (the top
|
||||
# level), the drill ``ls`` scoped to the first (only) source
|
||||
# (phase 94), then ``read`` the first file line's combined
|
||||
# source/path — and all three ahead of the first `delta` frame.
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "ls", "argument": SOURCE},
|
||||
{"type": "tool", "name": "read", "argument": f"{SOURCE}/{READ_TARGET}"},
|
||||
], _tool_frames(frames)
|
||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert all(
|
||||
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
|
||||
)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# Phase 119 (LOCKED A1): done.sources = the READ docs only —
|
||||
# exactly the one read; the five suggested docs (name-hit doc
|
||||
# included) never chip.
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
(SOURCE, READ_TARGET)
|
||||
], done["sources"]
|
||||
# done.related = the rank-6+ remainder deduped against the cited
|
||||
# read doc — the read doc is rank 7 (IN the remainder) and drops
|
||||
# out: only the sibling name-hit doc remains.
|
||||
assert [(s["source"], s["path"]) for s in done["related"]] == [
|
||||
(SOURCE, "forgejo/forgejo-nginx.md")
|
||||
], done["related"]
|
||||
|
||||
# The final answer quotes the read document (the mock's
|
||||
# deterministic quote: "Read <source/path>. <first 80 chars of its
|
||||
# content>").
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(
|
||||
bubble,
|
||||
).to_contain_text(f"Read {SOURCE}/{READ_TARGET}.", timeout=30_000)
|
||||
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
|
||||
|
||||
# The UI chip row: EXACTLY the one read doc (LOCKED A1) — and
|
||||
# explicitly NONE of the suggested docs (the name-hit doc first).
|
||||
_assert_chip_row(page, [READ_TARGET])
|
||||
expect(
|
||||
page.locator(".msg.brain .source-chip", has_text="install-guide")
|
||||
).to_have_count(0)
|
||||
expect(
|
||||
page.locator(".msg.brain .source-chip", has_text=NAME_HIT)
|
||||
).to_have_count(0)
|
||||
# The related row renders rank 6+ deduped against the cited read
|
||||
# doc (phase-113 behavior untouched).
|
||||
_assert_related_row(page, ["forgejo/forgejo-nginx.md"])
|
||||
|
||||
# Durable record: grounded; suggested + related + read (deduped —
|
||||
# the read doc sits in the related slot, LOCKED A3 — phase 119 A1
|
||||
# retires the chip surface only, not the record).
|
||||
row = _last_query_log()
|
||||
assert row.question == TOOLS_QUESTION
|
||||
assert row.deflected is False
|
||||
assert row.sources == ", ".join(
|
||||
f"{SOURCE}/{p}" for p in SUGGESTED + RELATED
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 3. (c) a zero-read grounded turn chips nothing (LOCKED A1's visible
|
||||
# consequence)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_zero_read_grounded_turn_chips_nothing(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""A PLAIN distinctive question (no tool trigger) grounds on the
|
||||
name tokens: the ``done`` frame's ``sources == []`` and ZERO
|
||||
``.source-chip`` elements in the bubble — LOCKED A1's visible
|
||||
consequence (the suggested docs, name-hit doc included, are seed
|
||||
context, never citations); the answer still renders grounded
|
||||
(deflected: false); the related row renders rank 6+."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, BASE_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# LOCKED A1's wire-level consequence: a grounded turn that read
|
||||
# nothing cites nothing.
|
||||
assert done["sources"] == [], done["sources"]
|
||||
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
# Zero chips — not just "no suggested chips": the citation
|
||||
# surface is empty.
|
||||
_assert_chip_row(page, [])
|
||||
_assert_related_row(page, RELATED)
|
||||
|
||||
# Durable record: grounded; suggested + related (LOCKED A3).
|
||||
row = _last_query_log()
|
||||
assert row.question == BASE_QUESTION
|
||||
assert row.deflected is False
|
||||
assert row.sources == ", ".join(f"{SOURCE}/{p}" for p in SUGGESTED + RELATED)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 4. (d) the suggested-folder lines ride the prompt — echoed verbatim
|
||||
# by the new mock trigger
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_folder_lines_echoed_verbatim(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
"""The distinctive question + ``repeat your folder map`` (the new
|
||||
mock trigger — the prompt-injection-echo convention of
|
||||
``<tuning>`` / ``<knowledge_base>``) ⇒ the answer ends with the
|
||||
suggested-folder lines VERBATIM (joined with ``; ``): the guides'
|
||||
folder line (its owner excluded from the entries) and the
|
||||
name-hit doc's folder line — ``namekb/forgejo/: forgejo-nginx.md``
|
||||
(the line prefix + its known sibling entry; the owner
|
||||
``forgejo-home`` is excluded — its identity is already in its
|
||||
``<document>`` block)."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, FOLDER_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
assert done["sources"] == [], done["sources"]
|
||||
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
|
||||
# The folder lines VERBATIM — both lines, suggested-folder order
|
||||
# (the guides' folder first — its first suggested doc leads; the
|
||||
# name-hit doc's folder second), entries owner-excluded.
|
||||
expect(bubble).to_contain_text(
|
||||
"(folders: "
|
||||
"namekb/deploy/: install-guide-b.md, install-guide-c.md, install-guide-d.md; "
|
||||
"namekb/forgejo/: forgejo-nginx.md)",
|
||||
)
|
||||
# The task's pin: the name-hit doc's folder line prefix + one
|
||||
# known sibling entry.
|
||||
expect(bubble).to_contain_text(f"{SOURCE}/forgejo/: forgejo-nginx.md")
|
||||
|
||||
# Zero-read grounded turn — chips nothing (LOCKED A1).
|
||||
_assert_chip_row(page, [])
|
||||
|
||||
# Durable record: grounded; suggested + related (LOCKED A3 — the
|
||||
# folder question's rank-6+ order).
|
||||
row = _last_query_log()
|
||||
assert row.question == FOLDER_QUESTION
|
||||
assert row.deflected is False
|
||||
assert row.sources == ", ".join(
|
||||
f"{SOURCE}/{p}" for p in SUGGESTED + RELATED_FOLDER
|
||||
)
|
||||
@@ -472,13 +472,13 @@ def test_answer_content_intact(page: Page, app_url: str, seeded_kb: None) -> Non
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
|
||||
# The long answer streams to completion with its sources ...
|
||||
# The long answer streams to completion (phase 119, LOCKED A1: a
|
||||
# zero-read turn chips nothing — the suggested kubernetes doc never
|
||||
# chips; the retired phase-118 A4 union is gone) ...
|
||||
submit(page, LONG_QUESTION)
|
||||
wait_settled(page)
|
||||
expect(page.locator(".msg.brain .bubble").last).to_contain_text(LONG_ANSWER_END)
|
||||
expect(
|
||||
page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
).to_have_count(1)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
# ... and a thinking turn completes with its block auto-collapsed
|
||||
# (phase 17: open while streaming, closed from the first delta on).
|
||||
@@ -489,25 +489,25 @@ def test_answer_content_intact(page: Page, app_url: str, seeded_kb: None) -> Non
|
||||
expect(details.locator(".thinking-text")).to_contain_text(THINKING_FRAGMENT)
|
||||
expect(page.locator(".msg.brain .bubble").last).to_contain_text(MOCK_ANSWER_MARKER)
|
||||
|
||||
# Persistence: four messages, the thinking text + sources stored raw.
|
||||
# Persistence: four messages, the thinking text stored raw — and
|
||||
# the thinking turn's sources EMPTY (phase 119, LOCKED A1: it read
|
||||
# nothing, so the done frame — and the record — carry no sources;
|
||||
# the retired phase-118 A4 union is gone).
|
||||
raw = page.evaluate(f"() => localStorage.getItem('{STORAGE_KEY}')")
|
||||
stored = json.loads(raw)
|
||||
assert [m["who"] for m in stored["messages"]] == ["user", "brain", "user", "brain"]
|
||||
assert LONG_ANSWER_END in stored["messages"][1]["text"]
|
||||
assert THINKING_FRAGMENT in stored["messages"][3]["thinking"]
|
||||
assert any(
|
||||
s["path"] == "homelab/kubernetes.md" for s in stored["messages"][3]["sources"]
|
||||
)
|
||||
assert stored["messages"][3]["sources"] == []
|
||||
|
||||
# Restore: the long answer (with its chip) and the COLLAPSED thinking
|
||||
# block come back intact.
|
||||
# Restore: the long answer and the COLLAPSED thinking block come
|
||||
# back intact — with ZERO citation chips (both turns read nothing;
|
||||
# phase 119, LOCKED A1).
|
||||
page.reload()
|
||||
expect(page.locator(".msg.user .bubble")).to_have_count(2)
|
||||
expect(page.locator(".msg.brain .bubble")).to_have_count(2)
|
||||
expect(page.locator(".msg.brain .bubble").first).to_contain_text(LONG_ANSWER_END)
|
||||
expect(
|
||||
page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
).to_have_count(2)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
restored = page.locator(".msg.brain").last.locator("details.thinking")
|
||||
expect(restored).not_to_have_attribute("open")
|
||||
expect(restored.locator(".thinking-text")).to_contain_text(THINKING_FRAGMENT)
|
||||
|
||||
@@ -24,8 +24,10 @@ Test → story mapping (Playwright Mapping Rule):
|
||||
4. ``test_jinja_retrievable_not_deflected`` — a question carrying the
|
||||
``.j2`` sentinel FTS-matches the chunk (A8: LOW requires best cosine
|
||||
below threshold **and** zero FTS hits) → honest-positive: the answer
|
||||
bubble is not ``.is-deflected`` and a source chip names
|
||||
``templates/deploy.j2``.
|
||||
bubble is not ``.is-deflected`` — and (phase 119, LOCKED A1) the
|
||||
zero-read turn chips NOTHING (the ``templates/deploy.j2`` retrieval
|
||||
doc is suggested context, not a chip; the retired phase-118 A4
|
||||
union is gone).
|
||||
|
||||
Phase 97 adaptation: the Sources table is the DRILL-DOWN TREE — the
|
||||
rows live at their folder levels (``docs`` → ``homelab`` → ``quadlet``
|
||||
@@ -218,10 +220,16 @@ def test_jinja_retrievable_not_deflected(
|
||||
page.fill("#message-input", JINJA_QUESTION)
|
||||
page.click("#send-btn")
|
||||
|
||||
# The done event appends source chips — waiting on the .j2 chip means
|
||||
# the turn is finished and the retrieval doc reached the UI.
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="templates/deploy.j2")
|
||||
expect(chip).to_have_count(1, timeout=30_000)
|
||||
# Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation
|
||||
# chips — the .j2 retrieval doc is suggested context, not a chip
|
||||
# (the retired phase-118 A4 union is gone). The turn is finished
|
||||
# (and the grounded, non-deflected state is reached) when the
|
||||
# answer settles with the send button recovered.
|
||||
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
|
||||
"Deterministic mock answer for E2E", timeout=30_000
|
||||
)
|
||||
expect(page.locator("#send-label")).to_have_text("Send")
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
# A8: LOW requires best cosine < threshold AND zero FTS hits — the
|
||||
# question's sentinel tokens FTS-match the .j2 chunk, so the gate is
|
||||
|
||||
@@ -658,10 +658,11 @@ def test_truncated_read_frame_order_live_marker_and_llm_notice(
|
||||
assert i_tool < i_result < i_delta, (i_tool, i_result, i_delta)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# The read document is the turn's cited source (retrieval +
|
||||
# agent-read, deduped — the grounded-turn record; phase 118: the
|
||||
# target may be a suggested (summary-seeded) document — its seed
|
||||
# status never changes the cap's wire contract).
|
||||
# The read document is in the turn's cited sources (phase 119,
|
||||
# LOCKED A1: done.sources is the READ docs only — this turn's one
|
||||
# read; phase 118: the target may be a suggested (summary-seeded)
|
||||
# document — its seed status never changes the cap's wire
|
||||
# contract).
|
||||
assert any(
|
||||
s["path"] == CAPPED_REL and s["source"] == SOURCE for s in done["sources"]
|
||||
), done["sources"]
|
||||
@@ -797,6 +798,8 @@ def test_short_read_control_no_frame_no_marker(
|
||||
assert _result_frames(frames) == [], _result_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# The read document is the turn's cited source (phase 119 A1 — the
|
||||
# READ docs only; the target may be a suggested/summary-seeded doc).
|
||||
assert any(
|
||||
s["path"] == SHORT_REL and s["source"] == SOURCE for s in done["sources"]
|
||||
), done["sources"]
|
||||
|
||||
@@ -10,11 +10,15 @@ deterministic mock embeddings (same pattern as the earlier story suites).
|
||||
The four tests map the story's acceptance criteria:
|
||||
|
||||
1. multi-format fixture import — hidden doc excluded, ``/api/docs`` counts
|
||||
2. "How did I install gitlab?" — grounded (not deflected), gitlab chip,
|
||||
``query_log`` row with the gitlab doc in ``sources``
|
||||
2. "How did I install gitlab?" — grounded (not deflected), ZERO citation
|
||||
chips (phase 119, LOCKED A1 — the turn reads nothing: the suggested
|
||||
gitlab doc is context, not a chip), ``query_log`` row with the gitlab
|
||||
doc in ``sources`` (LOCKED A3, untouched)
|
||||
3. keyword-only question ("kafkabridge") beats the vector ranking — the
|
||||
corroborated-lexical gate (A8 revised 2026-09-14) grounds it end to
|
||||
end: weak cosine, but an FTS hit AND cosine >= lexical_support_floor
|
||||
end: weak cosine, but an FTS hit AND cosine >= lexical_support_floor;
|
||||
the FTS-matched doc tops the durable record (the chip pin is retired
|
||||
with the phase-118 A4 union — a zero-read turn chips nothing)
|
||||
4. "sourdough" — deflected bubble + ≥2 "Maybe try" chips
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -156,9 +160,15 @@ def test_gitlab_question_is_grounded_with_gitlab_chip(
|
||||
# Grounded: no deflected bubble at all.
|
||||
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
|
||||
|
||||
# The gitlab document is cited (a chip carrying its path).
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="container_gitlab/gitlab.md")
|
||||
expect(chip).to_have_count(1, timeout=30_000)
|
||||
# Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation
|
||||
# chips — the suggested gitlab doc is seed context, not a citation
|
||||
# (the retired phase-118 A4 union is gone); the ranking assertion
|
||||
# lives in the durable record below (LOCKED A3, untouched). The
|
||||
# button label is the settle sync: the client re-labels Send on the
|
||||
# done frame, and the server writes the query_log row just before
|
||||
# yielding it.
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
# Durable record: not deflected, and the gitlab doc is in sources.
|
||||
with SessionLocal() as db:
|
||||
@@ -186,10 +196,15 @@ def test_keyword_only_question_beats_vector_ranking(
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
|
||||
|
||||
# The FTS-matched doc is the TOP source chip (it beats the vector rank).
|
||||
first_chip = page.locator(".msg.brain .source-chip").first
|
||||
first_chip.wait_for(state="visible", timeout=30_000)
|
||||
expect(first_chip).to_contain_text("static-dns.json")
|
||||
# Phase 119 (LOCKED A1): zero-read turn → ZERO chips (the
|
||||
# FTS-matched doc beats the vector rank in the DURABLE record below
|
||||
# — LOCKED A3, untouched; the retired phase-118 A4 chip pin is
|
||||
# gone). Wait for the settle via the Send label instead of a chip
|
||||
# (the in-flight button is the enabled Stop control — only the
|
||||
# label proves the done frame landed and the query_log row was
|
||||
# written).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
|
||||
|
||||
with SessionLocal() as db:
|
||||
row = db.scalars(select(QueryLog)).one()
|
||||
|
||||
@@ -50,9 +50,11 @@ Test → phase mapping:
|
||||
(``Found …`` — the match reached the model), and the turn settles to
|
||||
idle with no error banner.
|
||||
2. ``test_search_adds_no_source_by_itself`` — context accounting
|
||||
(locked A5): the search-only flow (no read) leaves
|
||||
``done.sources`` / the source chips / ``query_log.sources`` at the
|
||||
retrieval baseline — the search adds no source by itself.
|
||||
(locked A5, phase-119 A1 shape): the search-only flow (no read)
|
||||
leaves ``done.sources`` / the source chips EMPTY (chips cite read
|
||||
docs only — the search adds no source by itself, and the retrieval
|
||||
doc was never read) while ``query_log.sources`` keeps the retrieval
|
||||
baseline (LOCKED A3, untouched).
|
||||
3. ``test_search_tool_line_re_renders_after_reload`` — the persisted
|
||||
record (phase 14 convention: the generic ``{name, argument}``
|
||||
toolAcc) re-renders the search line through the same helper.
|
||||
@@ -423,25 +425,24 @@ def test_search_adds_no_source_by_itself(
|
||||
_wait_settled(page)
|
||||
|
||||
# The search really ran (its wire frame is present) — yet the
|
||||
# search-only flow (no read) leaves done.sources at the RETRIEVAL
|
||||
# baseline: the one fixture doc, nothing added by the search.
|
||||
# search-only flow (no read) adds NO cited source: phase 119,
|
||||
# LOCKED A1 — done.sources is the READ docs only, and nothing was
|
||||
# read (the retrieval doc was never read, so it never chips — the
|
||||
# retired phase-118 A4 suggested+read union is gone).
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN}
|
||||
]
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
(SEED_SOURCE, SEED_PATH)
|
||||
]
|
||||
assert done["sources"] == [], done["sources"]
|
||||
|
||||
# UI: exactly one source chip — the retrieval doc (the search
|
||||
# renders no chip of its own).
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(1)
|
||||
expect(chips.nth(0)).to_contain_text(SEED_SP)
|
||||
# UI: ZERO source chips — the search renders no chip of its own, and
|
||||
# the never-read retrieval doc chips nothing (phase 119 A1).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
# Durable record: the sources row is unchanged by the search alone.
|
||||
# Durable record: the sources row keeps the retrieval baseline (the
|
||||
# search adds nothing to it either — LOCKED A3, untouched).
|
||||
row = _last_query_log()
|
||||
assert row.deflected is False
|
||||
assert row.sources == SEED_SP
|
||||
|
||||
@@ -25,12 +25,14 @@ click is the idempotent share of the linked row):
|
||||
cookies) opening ``/shared/<token>`` sees the full conversation
|
||||
read-only through the same record shape: title = the auto-title,
|
||||
user + brain bubbles (the same deterministic answer text the admin
|
||||
session saw), the thinking block RESTORED COLLAPSED, the source
|
||||
chips as PLAIN TEXT (zero ``a.source-chip`` — guests cannot open
|
||||
documents, the documents API is admin-only), and ZERO interactive
|
||||
controls anywhere (no composer, no Save/Share pills, no Tune/Retry,
|
||||
no button chips); the nav's admin-only links stay hidden for a
|
||||
guest;
|
||||
session saw), the thinking block RESTORED COLLAPSED, ZERO source
|
||||
chips of any kind — the shared turn read nothing, so (phase 119,
|
||||
LOCKED A1) its ``done`` sources are empty and nothing re-renders as a
|
||||
chip; where chips exist they are always PLAIN TEXT (zero
|
||||
``a.source-chip`` — guests cannot open documents, the documents API
|
||||
is admin-only) — and ZERO interactive controls anywhere (no
|
||||
composer, no Save/Share pills, no Tune/Retry, no button chips); the
|
||||
nav's admin-only links stay hidden for a guest;
|
||||
* **Share from History + unshare** — the History row's Share column:
|
||||
"Create link" → Copy + Unshare; Unshare is the inline two-step
|
||||
(no native dialog); after Yes the cell returns to "Create link",
|
||||
@@ -345,17 +347,13 @@ def test_anonymous_shared_view(
|
||||
expect(think).to_have_count(1)
|
||||
expect(think.first).not_to_have_attribute("open")
|
||||
|
||||
# Source chips are PLAIN TEXT: the on-topic turn carries its
|
||||
# source chips (the suggested docs — the hybrid retrieval's
|
||||
# summary-seed tier, phase 118), but every one as a <span>:
|
||||
# zero <a.source-chip> anywhere (a guest cannot open documents;
|
||||
# the documents API is admin-only).
|
||||
assert (
|
||||
anon.locator(".msg.brain .source-chip").count() >= 1
|
||||
), "the grounded turn must carry its source chips"
|
||||
expect(
|
||||
anon.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
).to_have_count(1)
|
||||
# Source chips: ZERO — the shared turn read nothing, so
|
||||
# (phase 119, LOCKED A1) its done sources are empty and the
|
||||
# shared page re-renders nothing (the retired phase-118 A4
|
||||
# suggested+read union is gone); and where chips DO exist they
|
||||
# are always plain text: zero <a.source-chip> anywhere (a
|
||||
# guest cannot open documents; the documents API is admin-only).
|
||||
expect(anon.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
expect(anon.locator("a.source-chip")).to_have_count(0)
|
||||
|
||||
# ZERO interactive controls anywhere in the conversation: no
|
||||
|
||||
@@ -1,22 +1,27 @@
|
||||
"""Phase 113 E2E (Playwright) — phase 118 re-targeted: the citation-surface
|
||||
contract (LOCKED A4) as VISIBLE chip counts — the summary-seed contract
|
||||
replaced the phase-112/113 usefulness bar: the chip row is the
|
||||
SUGGESTED tier (top-5 distinct docs, NO floor) + agent reads (none on
|
||||
these turns), the related row is rank 6+ (capped at
|
||||
``related_max_docs`` = 2), and a deflected turn still cites nothing
|
||||
(done.sources = [] — its weak hits are suggested for the durable record
|
||||
but are never citation chips).
|
||||
"""Phase 113 E2E (Playwright) — phase 118 re-targeted, phase 119
|
||||
re-targeted again: the citation-surface contract (phase 119, LOCKED
|
||||
A1) as VISIBLE chip counts — chips are the AGENT-READ docs only (the
|
||||
retired phase-118 A4 suggested+read union is gone): neither turn on
|
||||
this suite reads anything, so a grounded turn shows ZERO citation
|
||||
chips (the suggested tier is seed context, not citations — the
|
||||
phase-112/113 usefulness bar was retired with the full-text seeds, and
|
||||
the suggested chips were retired with the phase-118 A4 union), the
|
||||
related row is rank 6+ (capped at ``related_max_docs`` = 2, untouched),
|
||||
and a deflected turn still cites nothing (done.sources = [] — its weak
|
||||
hits are suggested for the durable record but are never citation
|
||||
chips).
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov
|
||||
|
||||
Acceptance (TODO L144–146, phase-118 shape): a grounded turn's chip row
|
||||
is the suggested tier — the 5 docs the model was seeded with (its
|
||||
"start here" set — the old "one chip per bar-clearing doc" is retired
|
||||
with the full-text seeds; the LLM decides what the summaries earn);
|
||||
a deflected turn shows zero citation chips (its weak hits, if any, live
|
||||
in the de-emphasized related row — ``.related-doc`` links, NEVER
|
||||
Acceptance (TODO L144–146, phase-119 shape): a grounded turn's chip row
|
||||
is the READ set — empty on these no-read turns (the visible consequence
|
||||
of the owner decision 2026-09-16: the suggested docs the model was
|
||||
seeded with are context, not citations — the LLM decides what the
|
||||
summaries earn, and a zero-read turn chips nothing); a deflected turn
|
||||
shows zero citation chips (its weak hits, if any, live in the
|
||||
de-emphasized related row — ``.related-doc`` links, NEVER
|
||||
``.source-chip``; L2c: never render weak hits as answer citations).
|
||||
|
||||
The fixture KB's tier shapes are deterministic under the E2E mock's
|
||||
@@ -25,14 +30,20 @@ here against the wire):
|
||||
|
||||
* **grounded question** — "What SSH aliases do I have?": best cosine
|
||||
0.352 ≥ 0.30 → grounded; the suggested tier (top-5, NO floor) is
|
||||
ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > kubernetes.md
|
||||
> backups.md (fused rank — the lexical-only cosine-0.0 docs rank when
|
||||
they rank: no floor filters them, LOCKED A3) and the done frame
|
||||
carries EXACTLY those five cited refs; the rank-6+ remainder
|
||||
(compose.container, static-dns.json) rides the related row, capped at
|
||||
two. (The four OBSERVED live shapes are unit-pinned at plan level in
|
||||
``tests/unit/test_source_chip_quality.py`` — the fixture KB
|
||||
reproduces the tiered shape live, so no docstring caveat is needed.)
|
||||
ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > backups.md
|
||||
> kubernetes.md (fused rank — the lexical-only cosine-0.0 docs rank
|
||||
when they rank: no floor filters them, LOCKED A3; phase 119, D1 —
|
||||
``aliases`` is a stem sub-component of ``ssh_aliases.txt``, so the
|
||||
doc name-hits, its name-hit row LEADS the lexical list, and the FTS
|
||||
rank shift re-orders the fused scores — the pre-phase
|
||||
kubernetes-before-backups order is retired) and the done frame
|
||||
carries NO cited refs (phase 119 A1 — nothing was read: the
|
||||
suggested tier seeds the prompt but never chips); the rank-6+
|
||||
remainder (compose.container, static-dns.json) rides the related
|
||||
row, capped at two. (The four OBSERVED live shapes are unit-pinned at
|
||||
plan level in ``tests/unit/test_source_chip_quality.py`` — the
|
||||
fixture KB reproduces the tiered shape live, so no docstring caveat
|
||||
is needed.)
|
||||
* **deflected question** — "How do I bake sourdough bread?": best cosine
|
||||
0.109 < 0.30 and zero FTS hits → honest deflection → zero cited refs
|
||||
(done.sources = []); the weak hits ARE suggested (no floor) for the
|
||||
@@ -106,14 +117,15 @@ def _ask(page: Page, message: str) -> None:
|
||||
def test_single_source_question_shows_exactly_one_citation_chip(
|
||||
page: Page, app_url: str, mock_llm: int, db_ready: None
|
||||
) -> None:
|
||||
"""The acceptance pin (phase 118, A4): a grounded question → the done
|
||||
bubble carries EXACTLY the suggested tier as ``.source-chip``s (the
|
||||
five docs the model was seeded with — top-5, NO floor: the old
|
||||
usefulness-bar "one chip" is retired with the full-text seeds) and
|
||||
the rank-6+ remainder renders only in the de-emphasized
|
||||
``.related-docs`` row (``.related-doc`` links — never
|
||||
``.source-chip``), labeled "Nearby docs, in case:". The durable
|
||||
record keeps the FULL retrieval (LOCKED A3)."""
|
||||
"""The acceptance pin (phase 119, A1): a grounded, NO-READ question →
|
||||
the done bubble carries ZERO ``.source-chip``s (the chip row is the
|
||||
agent-read docs only — nothing was read here: the five suggested
|
||||
docs the model was seeded with are context, not citations — the
|
||||
retired phase-118 A4 union is gone) and the rank-6+ remainder
|
||||
renders only in the de-emphasized ``.related-docs`` row
|
||||
(``.related-doc`` links — never ``.source-chip``), labeled "Nearby
|
||||
docs, in case:" (phase-113 behavior, untouched). The durable record
|
||||
keeps the FULL retrieval (LOCKED A3, untouched)."""
|
||||
_reset_db(mock_llm)
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/") # phase 79: chat is require_user-gated
|
||||
@@ -127,16 +139,14 @@ def test_single_source_question_shows_exactly_one_citation_chip(
|
||||
# Grounded: no deflected bubble at all.
|
||||
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
|
||||
|
||||
# The citation surface IS the suggested tier (LOCKED A4 — top-5,
|
||||
# NO floor): five chips, in fused rank order — the model was
|
||||
# seeded with exactly these five docs' summaries.
|
||||
chip = page.locator(".msg.brain .source-chip")
|
||||
expect(chip).to_have_count(5, timeout=30_000)
|
||||
expect(chip.nth(0)).to_contain_text("ssh_aliases.txt")
|
||||
expect(chip.nth(1)).to_contain_text("gitlab-compose.yaml")
|
||||
expect(chip.nth(2)).to_contain_text("uptime_probe.py")
|
||||
expect(chip.nth(3)).to_contain_text("kubernetes.md")
|
||||
expect(chip.nth(4)).to_contain_text("backups.md")
|
||||
# Phase 119 (LOCKED A1): the citation surface is the READ docs only
|
||||
# — this turn read nothing, so ZERO chips; the five suggested docs
|
||||
# (ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py >
|
||||
# backups.md > kubernetes.md — the phase-119 D1 re-rank, pinned in
|
||||
# the durable record below) seeded the prompt but never chip —
|
||||
# the retired phase-118 A4 union is gone. The suggested tier itself
|
||||
# is pinned below by the durable record (LOCKED A3, untouched).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
# The rank-6+ remainder rides the related row: a labeled,
|
||||
# de-emphasized list — one .related-doc link per doc (rank order,
|
||||
@@ -162,13 +172,19 @@ def test_single_source_question_shows_exactly_one_citation_chip(
|
||||
row_log = db.scalars(select(QueryLog)).one()
|
||||
assert row_log.question == SINGLE_SOURCE_QUESTION
|
||||
assert row_log.deflected is False
|
||||
# Suggested tier + related remainder, in the logged order.
|
||||
# Suggested tier + related remainder, in the logged order. Phase
|
||||
# 119 (D1): the stem sub-component ``aliases`` name-hits
|
||||
# ``ssh_aliases.txt`` — its name-hit row leads the lexical list,
|
||||
# the FTS rank shift re-orders the fused scores, and ``backups.md``
|
||||
# (best eff 0.016277) now out-ranks ``kubernetes.md`` (0.016036);
|
||||
# re-pinned against the post-phase walk (the record's CONTENT —
|
||||
# suggested + related + read, deduped — is LOCKED A3, untouched).
|
||||
assert row_log.sources == (
|
||||
"docs/homelab/ssh/ssh_aliases.txt, "
|
||||
"docs/homelab/container_gitlab/gitlab-compose.yaml, "
|
||||
"docs/homelab/scripts/uptime_probe.py, "
|
||||
"docs/homelab/kubernetes.md, "
|
||||
"docs/homelab/backups.md, "
|
||||
"docs/homelab/kubernetes.md, "
|
||||
"docs/homelab/quadlet/compose.container, "
|
||||
"docs/homelab/networking/static-dns.json"
|
||||
), row_log.sources
|
||||
|
||||
@@ -89,10 +89,6 @@ HESITATE_QUESTION = (
|
||||
#: pre-content pause (SLOW_PRETOKEN_TRIGGER) is running.
|
||||
THINKING_TAIL = "nothing is invented"
|
||||
|
||||
#: Phase-10 viewer URL + phase-13 back=/ (byte-identical to the chip the
|
||||
#: persistence suite pins — grounded-turn sources are unchanged by 20).
|
||||
CHIP_HREF = "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F"
|
||||
|
||||
|
||||
async def _import_fixtures(mock_port: int) -> ImportSummary:
|
||||
kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"}
|
||||
@@ -334,7 +330,10 @@ def test_full_answer_completes_after_rag_nav_midstream(
|
||||
brain = msgs[1]
|
||||
assert brain["text"] == FULL_LONG
|
||||
assert brain["deflected"] is False
|
||||
assert any(s["path"] == "homelab/kubernetes.md" for s in brain["sources"])
|
||||
# Phase 119 (LOCKED A1): the turn read nothing, so its done sources
|
||||
# — and the persisted record — are EMPTY (the retired phase-118 A4
|
||||
# union is gone).
|
||||
assert brain["sources"] == []
|
||||
|
||||
# The turn SETTLED — the phase-48 query_log row exists (a
|
||||
# cancelled turn would leave no row at all).
|
||||
@@ -408,7 +407,11 @@ def test_nav_switch_before_first_token_completes(
|
||||
assert [m["who"] for m in msgs] == ["user", "brain"]
|
||||
assert MOCK_ANSWER_MARKER in msgs[1]["text"]
|
||||
assert msgs[1]["deflected"] is False
|
||||
assert any(s["path"] == "homelab/kubernetes.md" for s in msgs[1]["sources"])
|
||||
# Phase 119 (LOCKED A1): the turn read nothing, so its done sources
|
||||
# — the stored record's sources list — are EMPTY (the retired
|
||||
# phase-118 A4 suggested-citation is gone; the full retrieval
|
||||
# stays in query_log, pinned below).
|
||||
assert msgs[1]["sources"] == []
|
||||
|
||||
# The turn settled — one finalized row (a cancelled turn would
|
||||
# leave no row at all).
|
||||
@@ -495,22 +498,25 @@ def test_completed_turn_unaffected(
|
||||
brain = before["messages"][1]
|
||||
assert MOCK_ANSWER_MARKER in brain["text"]
|
||||
assert brain["deflected"] is False
|
||||
assert any(s["path"] == "homelab/kubernetes.md" for s in brain["sources"])
|
||||
# Phase 119 (LOCKED A1): the turn read nothing, so its done sources
|
||||
# — and the persisted record — are EMPTY (the retired phase-118 A4
|
||||
# union is gone).
|
||||
assert brain["sources"] == []
|
||||
|
||||
# A trip to Sources and back (the turn finished long ago — uiState is
|
||||
# idle, so the pagehide save point must be a no-op).
|
||||
page.goto(app_url + "/sources.html")
|
||||
page.goto(app_url + "/")
|
||||
|
||||
# Full answer + source chip rendered; no error banner.
|
||||
# Full answer rendered with ZERO citation chips (the zero-read turn
|
||||
# persisted an empty sources list — phase 119, LOCKED A1; the
|
||||
# retired phase-118 A4 chip pin is gone); no error banner.
|
||||
expect(page.locator("#empty-state")).to_be_hidden()
|
||||
expect(page.locator(".msg.user .bubble")).to_contain_text(QUESTION)
|
||||
bubble = page.locator(".msg.brain .bubble")
|
||||
expect(bubble).to_have_count(1)
|
||||
expect(bubble.first).to_contain_text(MOCK_ANSWER_MARKER)
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
expect(chip).to_have_count(1)
|
||||
expect(chip.first).to_have_attribute("href", CHIP_HREF)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
_no_error_banner(page)
|
||||
|
||||
# Storage is byte-identical to the pre-navigation payload — the
|
||||
|
||||
@@ -59,19 +59,21 @@ TODO item — one Playwright file per story, A16):
|
||||
The inverse of the retired phase-24 pin: NO document's tail
|
||||
sentinel is in the echoed context — the full content of no
|
||||
suggested doc ever reached the model (case b). The grounded turn
|
||||
cites the 5 suggested docs (case e's chip surface, no read yet) and
|
||||
cites NOTHING (phase 119, LOCKED A1 — a zero-read turn chips
|
||||
nothing; the retired phase-118 A4 suggested+read union is gone) and
|
||||
renders the de-emphasized related row (rank 6+); the durable
|
||||
record carries suggested + related.
|
||||
record carries suggested + related (118-A3 untouched).
|
||||
2. ``test_read_suggested_doc_adds_full_text_and_cites`` — cases (c) +
|
||||
(e): the scripted flow ``read``s the rank-1 suggested doc — the
|
||||
read SUCCEEDS (phase 118: the seeds are summaries, not full text —
|
||||
the retired top-2 seed-read refusal is gone) and the mock's
|
||||
verbatim echo of the read result lands the tail sentinel in the
|
||||
answer (case c: the full text now arrives through the ``read``
|
||||
tool, not the seed); the citation chips = suggested + read,
|
||||
deduped — the read doc is among the suggested, so the chip row is
|
||||
exactly the 5 suggested docs, and the related row renders rank 6+
|
||||
(case e); the durable record carries suggested + related + read.
|
||||
tool, not the seed); the citation chips = the READ docs only
|
||||
(phase 119, LOCKED A1 — exactly the one read doc; none of the
|
||||
four other suggested docs chips), and the related row renders
|
||||
rank 6+ (case e); the durable record carries suggested + related +
|
||||
read (118-A3 untouched).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -503,14 +505,16 @@ def _last_query_log() -> QueryLog:
|
||||
return rows[0]
|
||||
|
||||
|
||||
def _assert_suggested_chips_and_related_row(page: Page, suggested_paths: list[str]) -> None:
|
||||
"""The citation surface of a grounded turn (LOCKED A4): the chip row
|
||||
is the suggested set (+ agent reads, deduped — asserted per test)
|
||||
and the de-emphasized ``related-docs`` row carries the rank-6+
|
||||
remainder (phase-113 UI, never a citation chip)."""
|
||||
def _assert_read_chips_and_related_row(page: Page, read_paths: list[str]) -> None:
|
||||
"""The citation surface of a grounded turn (phase 119, LOCKED A1 —
|
||||
the retired phase-118 A4 union is gone): the chip row is the
|
||||
AGENT-READ docs only (deduped, read order — a zero-read turn chips
|
||||
NOTHING: the suggested docs are seed context, not citations), and
|
||||
the de-emphasized ``related-docs`` row carries the rank-6+ remainder
|
||||
(phase-113 UI, untouched — never a citation chip)."""
|
||||
chips = page.locator(".msg.brain .source-chip")
|
||||
expect(chips).to_have_count(len(suggested_paths))
|
||||
for path in suggested_paths:
|
||||
expect(chips).to_have_count(len(read_paths))
|
||||
for path in read_paths:
|
||||
expect(
|
||||
chips.filter(has_text=path), message=f"chip for {path}"
|
||||
).to_have_count(1)
|
||||
@@ -553,11 +557,12 @@ def test_summaries_seed_the_prompt_not_the_full_text(
|
||||
assert _tool_frames(frames) == [], _tool_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# The grounded turn cites the 5 suggested docs (LOCKED A4 — no read
|
||||
# yet: the chip row IS the suggested set, in rank order).
|
||||
assert [
|
||||
(s["source"], s["path"]) for s in done["sources"]
|
||||
] == [(SOURCE, p) for p in SUGGESTED_TAIL], done["sources"]
|
||||
# Phase 119 (LOCKED A1): the grounded done frame cites the READ docs
|
||||
# only — this turn read NOTHING (the summary-only fast path), so the
|
||||
# citation surface is empty; the suggested set is seed context, not
|
||||
# citations (the retired phase-118 A4 union is gone). The durable
|
||||
# record below still carries suggested + related (118-A3 untouched).
|
||||
assert done["sources"] == [], done["sources"]
|
||||
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
|
||||
@@ -583,12 +588,13 @@ def test_summaries_seed_the_prompt_not_the_full_text(
|
||||
expect(bubble).not_to_contain_text(sentinel)
|
||||
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
|
||||
|
||||
# Case (e)'s chip surface (no read yet): chips = the 5 suggested
|
||||
# docs; the related row renders rank 6+ (the de-emphasized row).
|
||||
_assert_suggested_chips_and_related_row(page, SUGGESTED_TAIL)
|
||||
# Case (e)'s chip surface (phase 119 A1 — no read yet): ZERO chips;
|
||||
# the related row renders rank 6+ (the de-emphasized row, untouched).
|
||||
_assert_read_chips_and_related_row(page, [])
|
||||
|
||||
# Durable record: grounded; suggested + related (LOCKED A3 — the log
|
||||
# records retrieval, not citations).
|
||||
# records retrieval, not citations; phase 119 A1 retires the chip
|
||||
# union only, not the record).
|
||||
row = _last_query_log()
|
||||
assert row.question == TAIL_QUESTION
|
||||
assert row.deflected is False
|
||||
@@ -609,10 +615,10 @@ def test_read_suggested_doc_adds_full_text_and_cites(
|
||||
retired top-2 seed-read refusal is gone) and the mock's verbatim
|
||||
echo of the read result lands the document's tail sentinel in the
|
||||
answer (case c: the full text reached the model through the
|
||||
``read`` tool, not the seed). The citation chips = suggested +
|
||||
read, deduped — the read doc is among the suggested, so the chip
|
||||
row is exactly the 5 suggested docs; the related row renders rank
|
||||
6+ (case e)."""
|
||||
``read`` tool, not the seed). The citation chips = the READ docs
|
||||
only (phase 119, LOCKED A1 — exactly the one read doc; none of the
|
||||
other four suggested docs chips; the retired phase-118 A4 union is
|
||||
gone); the related row renders rank 6+ (case e, untouched)."""
|
||||
page.set_default_timeout(30_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
@@ -628,13 +634,15 @@ def test_read_suggested_doc_adds_full_text_and_cites(
|
||||
], _tool_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# LOCKED A4: done.sources = suggested + read, deduped — the read doc
|
||||
# is already among the suggested, so the citation surface is
|
||||
# exactly the 5 suggested docs (in rank order).
|
||||
# Phase 119 (LOCKED A1): done.sources = the READ docs only —
|
||||
# exactly the one scripted read; the other four suggested docs
|
||||
# never chip (the retired phase-118 A4 suggested+read union is
|
||||
# gone).
|
||||
assert [
|
||||
(s["source"], s["path"]) for s in done["sources"]
|
||||
] == [(SOURCE, p) for p in SUGGESTED_READ], done["sources"]
|
||||
# done.related = the rank-6+ remainder (deduped against the cited).
|
||||
] == [(SOURCE, READ_TARGET)], done["sources"]
|
||||
# done.related = the rank-6+ remainder (deduped against the cited —
|
||||
# the read doc is rank 1, not in the remainder).
|
||||
assert [
|
||||
(s["source"], s["path"]) for s in done["related"]
|
||||
] == [(SOURCE, p) for p in RELATED_PATHS], done["related"]
|
||||
@@ -658,13 +666,13 @@ def test_read_suggested_doc_adds_full_text_and_cites(
|
||||
if path != READ_TARGET:
|
||||
expect(bubble).not_to_contain_text(sentinel)
|
||||
|
||||
# Case (e): the UI chip row = suggested + read (deduped — the read
|
||||
# doc is among the suggested, so chips = the 5 suggested docs), and
|
||||
# the related row renders rank 6+.
|
||||
_assert_suggested_chips_and_related_row(page, SUGGESTED_READ)
|
||||
# Case (e): the UI chip row = the READ docs only (phase 119 A1 —
|
||||
# exactly the one read doc), and the related row renders rank 6+.
|
||||
_assert_read_chips_and_related_row(page, [READ_TARGET])
|
||||
|
||||
# Durable record: grounded; suggested + related + read (deduped,
|
||||
# LOCKED A3).
|
||||
# LOCKED A3 — phase 119 A1 retires the chip union only, not the
|
||||
# record).
|
||||
row = _last_query_log()
|
||||
assert row.question == READ_QUESTION
|
||||
assert row.deflected is False
|
||||
|
||||
@@ -55,9 +55,6 @@ DEFLECT_PHRASE = r"haven't done anything like that"
|
||||
#: suite keys off it (mock_llm.compose_thinking).
|
||||
THINKING_FRAGMENT = "Step 2: Check my notes"
|
||||
STORAGE_KEY = "bor.chat.v1"
|
||||
#: Phase-10 viewer URL + phase-13 back=/ (byte-identical to the chip the
|
||||
#: persistence suite pins — grounded-turn sources are unchanged by 17).
|
||||
CHIP_HREF = "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F"
|
||||
|
||||
|
||||
async def _import_fixtures(mock_port: int) -> ImportSummary:
|
||||
@@ -154,13 +151,13 @@ def test_thinking_block_streams_open_then_collapses(
|
||||
expect(bubble).not_to_have_text("", timeout=30_000)
|
||||
expect(details).not_to_have_attribute("open")
|
||||
|
||||
# Settled: full scratchpad, grounded mock answer, source chip(s),
|
||||
# and the re-enabled send button.
|
||||
# Settled: full scratchpad, grounded mock answer, ZERO citation
|
||||
# chips (phase 119, LOCKED A1 — the zero-read turn chips nothing;
|
||||
# the retired phase-118 A4 union is gone), and the re-enabled send
|
||||
# button.
|
||||
expect(details.locator(".thinking-text")).to_contain_text(THINKING_FRAGMENT)
|
||||
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
expect(chip.first).to_be_visible()
|
||||
expect(chip.first).to_have_attribute("href", CHIP_HREF)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
expect(page.locator("#send-btn")).to_be_enabled()
|
||||
expect(page.locator("#send-label")).to_have_text("Send")
|
||||
|
||||
@@ -232,10 +229,12 @@ def test_thinking_restored_after_reload(page: Page, app_url: str, seeded_kb: Non
|
||||
expect(restored).not_to_have_attribute("open") # restored COLLAPSED
|
||||
expect(restored.locator(".thinking-text")).to_have_text(captured)
|
||||
|
||||
# Answer bubble + source chip are intact (phase-14 restore path).
|
||||
# Answer bubble is intact after the restore (phase-14 restore path)
|
||||
# — with ZERO citation chips (phase 119, LOCKED A1: the zero-read
|
||||
# turn persisted an empty sources list; the retired phase-118 A4
|
||||
# union is gone).
|
||||
expect(page.locator(".msg.brain .bubble").last).to_contain_text(MOCK_ANSWER_MARKER)
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
expect(chip.first).to_have_attribute("href", CHIP_HREF)
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -252,10 +251,10 @@ def test_no_thinking_block_without_trigger(page: Page, app_url: str, seeded_kb:
|
||||
# No trigger → no thinking events → no block anywhere on the page.
|
||||
expect(page.locator("details.thinking")).to_have_count(0)
|
||||
|
||||
# The turn itself is complete and grounded, exactly as before phase 17.
|
||||
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
|
||||
expect(chip.first).to_be_visible()
|
||||
expect(chip.first).to_have_attribute("href", CHIP_HREF)
|
||||
# The turn itself is complete and grounded, exactly as before phase
|
||||
# 17 — with ZERO citation chips (phase 119, LOCKED A1: the zero-read
|
||||
# turn chips nothing; the retired phase-118 A4 union is gone).
|
||||
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user