Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 119_name_signal_read_chips
Build and Push Containers / build-and-push-app (push) Successful in 2m1s
Details
Build and Push Containers / build-and-push-db (push) Successful in 18s
Details

Browse Source 
All verification complete. Final report:

**Phase 119 final verification pass — all criteria verified, one stale pin fixed.**
- Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry.
- Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged.
- New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2.
- Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed).
- Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met).
- Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
ducoterra
2026-09-16 15:50:48 -04:00
parent 795fb56425
commit a5b63f83ad
89 changed files with 4377 additions and 655 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/e2e/mock_llm.py
+72
View File
@@ -436,6 +436,29 @@ Implements just enough of the aipi surface:
phrases — the phase-71/72/94 ordering convention); verified
2026-09-16: no existing E2E question or fixture file contains the
phrase, so every other suite is unaffected.
- user message containing ``repeat your folder map``
(``FOLDER_MAP_TRIGGER``, phase 119, D3 — the suggested-folder
context's story suite ``tests/e2e/test_name_signal_read_chips.py``)
**and** the system prompt contains the ``<documents>`` section ->
the composed answer ends with `` (folders: <line 1>; <line 2>; …)``
echoing the suggested-folder lines VERBATIM — the plain lines
between ``SUGGEST_INTRO`` and the first ``<document `` in the
``<documents>`` section (``app.rag.prompts.build_high_prompt`` /
``app.rag.agent.suggested_folder_lines``, phase 119 LOCKED A4:
``<source>/<prefix>/: e1, e2, …``), joined with ``; ``. If no
folder lines are present (empty *folder_lines* — the
byte-identical phase-118 shape) the echo is omitted and the answer
is otherwise unchanged. The same prompt-injection-echo convention
as the ``<tuning>`` / ``<knowledge_base>`` triggers above (a SUFFIX
appended after both, so the folders suffix is the last thing
rendered): phase 119 D3's observability — the folder lines land in
the model's prompt only on grounded turns, and the echo is the
E2E's deterministic lens on their EXACT text (line identity shape
+ entry order + the owning-doc exclusion + the caps). It composes
with any base answer (it does not compete with a marker branch);
the trigger phrase is disjoint from every other trigger's; verified
2026-09-16: no existing E2E question or fixture file contains the
phrase, so every other suite is unaffected.
Failure injection (phase 67, LLM retry, TODO.md L3) — deterministic
dead-endpoint behavior for the retry E2E suite (``tests/e2e/
@@ -675,6 +698,17 @@ TABLE_TRIGGER = "show me a table"
#: phrase, so every other suite is unaffected.
HISTORY_TRIGGER = "echo my history"
#: Phase 119 (D3, the suggested-folder context's story suite): a user
#: message containing this substring (case-insensitive) — combined
#: with the ``<documents>`` section in the system prompt (a grounded
#: turn) — gets the composed answer suffixed with the folder lines
#: VERBATIM (`` (folders: <line 1>; <line 2>; …)``, joined with
#: ``; ``; omitted when the section carries no folder lines) — see the
#: module docstring. Disjoint from every other trigger phrase; verified
#: 2026-09-16: no existing E2E question or fixture file contains the
#: phrase, so every other suite is unaffected.
FOLDER_MAP_TRIGGER = "repeat your folder map"
TABLE_ANSWER = (
"Here's the shape, in a table:\n"
"\n"
@@ -1657,6 +1691,32 @@ def first_kb_bullet(system: str) -> str | None:
return None
def folder_map_lines(system: str) -> list[str]:
"""The suggested-folder lines of the ``<documents>`` section
(phase 119, D3, LOCKED A4) — or ``[]`` when there are none.
``build_high_prompt`` lays the section body out as ``SUGGEST_INTRO``
(line 1), the folder lines (each on its own line, immediately after
the intro line), a blank line, then the first ``<document>`` block
— so the lines are exactly the non-blank lines between the intro
line and the first ``<document `` line of the block. With no folder
lines (the byte-identical phase-118 shape) there is only the blank
line there, and this returns ``[]`` (the echo is then omitted — the
answer is otherwise unchanged).
"""
block = _DOCUMENTS_BLOCK_RE.search(system)
if not block:
return []
lines = block.group(0).splitlines()
out: list[str] = []
for line in lines[2:]: # skip the tag line + the SUGGEST_INTRO line
if line.lstrip().startswith("<document"):
break
if line.strip():
out.append(line)
return out
def _history_echo(body: dict[str, Any]) -> str:
"""The phase-74 history echo (byte-stable, stateless over messages).
@@ -2093,6 +2153,18 @@ def compose_answer(body: dict[str, Any]) -> str:
bullet = first_kb_bullet(system)
if bullet:
answer = f"{answer} (kb: {bullet})"
# Suggested-folder context (phase 119, D3, LOCKED A4): when the user
# message carries FOLDER_MAP_TRIGGER and the system prompt carries
# the <documents> section, the answer ends with the folder lines
# VERBATIM (joined with "; ") — the E2E's deterministic lens on the
# exact lines build_high_prompt seeded (D3 observability; the same
# prompt-injection-echo convention as the two echoes above). Omitted
# when the section carries no folder lines (the byte-identical
# phase-118 shape) — the answer is otherwise unchanged.
if FOLDER_MAP_TRIGGER in user.lower() and "<documents>" in system:
folder_lines = folder_map_lines(system)
if folder_lines:
answer = f"{answer} (folders: {'; '.join(folder_lines)})"
return answer
tests/e2e/test_agent_document_tools.py
+23 -14
View File
@@ -60,12 +60,15 @@ Test → story mapping (Playwright Mapping Rule):
``tool`` frames (``ls``, then ``read`` with the combined path, ahead
of any delta), the UI shows the transient calling-tool status while a
tool runs, the bubble shows both tool lines, the final answer quotes
the read document, and the source chips include the read document
(viewer link).
the read document, and the source chips are EXACTLY the read
document (phase 119, LOCKED A1 — chips cite read docs only; viewer
link).
2. ``test_tool_lines_re_render_after_reload`` — the persisted record
(phase 14) re-renders the tool lines.
3. ``test_plain_grounded_question_has_no_tool_frames`` — no marker → no
``tool`` frames, the answer renders exactly as today (regression
``tool`` frames, the answer renders exactly as today, and (phase 119,
LOCKED A1) the zero-read grounded turn chips NOTHING (the retrieval
doc is cited only in the durable record, never as a chip) (regression
inside the story file).
4. ``test_deflected_question_has_no_tool_frames`` — the tools are
grounded-only: a deflected turn runs none.
@@ -444,8 +447,12 @@ def test_marker_question_lists_reads_and_quotes(
)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
# Phase 119 (LOCKED A1): done.sources = the READ docs only —
# exactly the scripted read; the retrieval (seed) doc is suggested
# context, never a citation chip (the retired phase-118 A4
# suggested+read union is gone). The durable record below still
# carries both (LOCKED A3, untouched).
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(SEED_SOURCE, SEED_PATH),
(READ_SOURCE, READ_PATH),
]
@@ -465,16 +472,17 @@ def test_marker_question_lists_reads_and_quotes(
expect(bubble).to_contain_text(ANSWER_PREFIX)
expect(bubble).to_contain_text(ANSWER_QUOTE)
# Source chips: the retrieval doc AND the read doc (deduped, in
# order) — the read chip links to the viewer.
# Source chips: the read doc ONLY (phase 119, LOCKED A1 — the
# retrieval doc was never read, so it never chips; the retired
# phase-118 A4 union is gone) — the read chip links to the viewer.
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(2)
expect(chips.nth(0)).to_contain_text(SEED_SP)
expect(chips).to_have_count(1)
chip_read = page.locator(".msg.brain .source-chip", has_text=READ_PATH)
expect(chip_read).to_have_count(1)
expect(chip_read.first).to_have_attribute("href", READ_CHIP_HREF)
# Durable record: grounded, both sources logged (retrieval + read).
# Durable record: grounded, both sources logged (suggested + read —
# LOCKED A3, untouched by phase 119 A1).
row = _last_query_log()
assert row.question == MARKER_QUESTION
assert row.deflected is False
@@ -540,14 +548,15 @@ def test_plain_grounded_question_has_no_tool_frames(
assert _tool_frames(_frames(page)) == []
expect(page.locator(".tool-call")).to_have_count(0)
# The standard grounded answer, citing the retrieval doc only — the
# referenced JSON stays OUT of the sources (it was never read).
# The standard grounded answer, with ZERO citation chips — the
# turn read nothing, so (phase 119, LOCKED A1) the chip row is
# empty: the retrieval doc is cited in the durable record only
# (the retired phase-118 A4 union is gone); the referenced JSON
# stays out of the record too (it was never read).
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text(PLAIN_QUESTION)
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(1)
expect(chips.first).to_contain_text(SEED_SP)
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
row = _last_query_log()
assert row.question == PLAIN_QUESTION
tests/e2e/test_agent_unlimited_tools.py
+36 -20
View File
@@ -65,9 +65,11 @@ Test → story mapping (Playwright Mapping Rule):
``I read <sp1> and <sp2>.`` line; the round cap (default 10) bounds
the turn, no budget refusal anywhere.
2. ``test_done_sources_include_reads`` — the source chips under the
answer list the retrieval doc PLUS both read documents, deduped
(the phase-37 ``done.sources`` extension contract, now with 2
reads); the same combined list lands in ``query_log.sources``.
answer list the BOTH read documents, deduped (phase 119, LOCKED A1:
chips cite read docs only — the retrieval doc was never read, so it
never chips; the retired phase-118 A4 suggested+read union is
gone); the retrieval doc PLUS both reads still land in
``query_log.sources`` (LOCKED A3, untouched).
3. ``test_relist_allowed`` — the listing tool ran (its line rendered)
and no pre-phase-45 budget refusal ("… budget left") appears
anywhere in the message bubble or tool lines: the old
@@ -195,14 +197,22 @@ BUDGET_REFUSAL_FRAGMENTS = (
"budget left",
)
# The combined source list the app reports (app/api/chat.py): retrieval
# docs first, then the agent's read docs, deduped by (source, path).
# The combined source list the DURABLE record reports
# (app/api/chat.py, LOCKED A3 — suggested + related + read, deduped):
# retrieval doc first, then the agent's read docs, deduped by
# (source, path). The citation surface (done.sources, the chips) is
# the READ set only (phase 119, LOCKED A1 — the retired phase-118 A4
# suggested+read union is gone): the retrieval doc was never read.
EXPECTED_SOURCES = [
(SEED_SOURCE, SEED_PATH),
(READ1_SOURCE, READ1_PATH),
(READ2_SOURCE, READ2_PATH),
]
EXPECTED_SOURCES_LINE = ", ".join(f"{s}/{p}" for s, p in EXPECTED_SOURCES)
READ_ONLY_SOURCES = [
(READ1_SOURCE, READ1_PATH),
(READ2_SOURCE, READ2_PATH),
]
# --------------------------------------------------------------------------
@@ -508,7 +518,7 @@ def test_multi_read_turn(
# --------------------------------------------------------------------------
# 2. done.sources / source chips: retrieval doc + BOTH reads, deduped
# 2. done.sources / source chips: BOTH reads, deduped (phase 119 A1)
# --------------------------------------------------------------------------
@@ -523,23 +533,28 @@ def test_done_sources_include_reads(
_submit(page, MULTI_QUESTION)
_wait_settled(page)
# Wire level: done.sources is the retrieval doc FIRST, then both
# read documents — deduped (the retrieval doc was never read, the
# reads are each read once; nothing appears twice).
# Wire level: done.sources is the READ documents only, in read
# order — deduped (the reads are each read once; nothing appears
# twice). Phase 119, LOCKED A1: the retrieval doc was never read,
# so it is NOT in the citation surface (the retired phase-118 A4
# suggested+read union is gone); it still lands in the durable
# record (LOCKED A3, pinned in test 1).
frames = _frames(page)
done = next(f for f in frames if f.get("type") == "done")
assert [(s["source"], s["path"]) for s in done["sources"]] == EXPECTED_SOURCES
assert [
(s["source"], s["path"]) for s in done["sources"]
] == READ_ONLY_SOURCES, done["sources"]
pairs = [(s["source"], s["path"]) for s in done["sources"]]
assert len(pairs) == len(set(pairs)), "done.sources must be deduped"
# UI: exactly three source chips under the answer, in the same
# order, each a viewer link — no duplicated chip.
# UI: exactly two source chips under the answer (the read docs,
# read order), each a viewer link — no duplicated chip, and no
# chip for the never-read retrieval doc (phase 119 A1).
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(3)
expect(chips.nth(0)).to_contain_text(SEED_SP)
expect(chips.nth(1)).to_contain_text(READ1_SP)
expect(chips.nth(2)).to_contain_text(READ2_SP)
for i, (source, path) in enumerate(EXPECTED_SOURCES):
expect(chips).to_have_count(2)
expect(chips.nth(0)).to_contain_text(READ1_SP)
expect(chips.nth(1)).to_contain_text(READ2_SP)
for i, (source, path) in enumerate(READ_ONLY_SOURCES):
expect(chips.nth(i)).to_have_attribute(
"href", f"/document.html?source={source}&path={path}&back=%2F"
)
@@ -631,12 +646,13 @@ def test_single_tool_flow_regression(
expect(bubble).not_to_contain_text(BOTH_READS_LINE)
expect(bubble).not_to_contain_text(READ2_SP)
# done: non-deflected; sources = retrieval doc + the single read
# (READ2 absent — it was never read).
# done: non-deflected; sources = the single READ doc only (phase
# 119, LOCKED A1 — READ2 absent: never read; the retrieval doc
# absent: never read — the retired phase-118 A4 union is gone; the
# durable record below keeps retrieval + read, LOCKED A3).
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(SEED_SOURCE, SEED_PATH),
(READ1_SOURCE, READ1_PATH),
]
tests/e2e/test_api_tokens.py
+24 -9
View File
@@ -391,8 +391,11 @@ def test_admin_generates_token_in_ui(page: Page, app_url: str, db_ready: None) -
# ---------------------------------------------------------------------------
# 4. The token flow: a fresh context signs in through the real gate
# and USES the app — a grounded chat turn (mock LLM), a cited
# document opened in the same-page modal, the role-user header
# and USES the app — a grounded chat turn (mock LLM; phase 119,
# LOCKED A1: the turn is the scripted summary-read flow — chips
# cite READ docs only, and this turn's read earns the cited chip),
# a cited document opened in the same-page modal, the role-user
# header
# ---------------------------------------------------------------------------
@@ -417,15 +420,27 @@ def test_token_user_uses_the_app(
# task-04 helper: fill #auth-gate-input → submit → gate hides).
login_with_token(user, app_url, token)
# Use the app: a grounded turn against the seeded KB (mock
# LLM) — the brain bubble renders the deterministic answer.
_ask(user, "How is my Kubernetes cluster set up? (api-tokens-flow)")
# Use the app: a GROUNDED turn against the seeded KB (mock
# LLM). Phase 119 (LOCKED A1): chips cite READ docs only — a
# plain question would chip nothing (the retired phase-118 A4
# union is gone), so the turn is the mock's scripted
# summary-read flow (SUMMARY_SEED_READ_TRIGGER): it reads the
# kubernetes fixture, and that read is the chip's source.
q = (
"Read the suggested document: read docs/homelab/kubernetes.md — "
"how is my Kubernetes cluster set up? (api-tokens-flow)"
)
user.fill("#message-input", q)
user.click("#send-btn")
expect(user.locator(".msg.user .bubble").last).to_contain_text(q)
# A cited source chip opens the document in the SAME-PAGE
# modal (the require_user content endpoint passes for a
# live token session).
# The turn's cited source chip (its one READ doc — the scripted
# read, phase 119 A1) opens the document in the SAME-PAGE modal
# (the require_user content endpoint passes for a live token
# session).
chip = user.locator(".msg.brain a.source-chip").first
expect(chip).to_be_visible(timeout=15_000)
expect(chip).to_be_visible(timeout=30_000)
expect(user.locator("#send-label")).to_have_text("Send")
chip.click()
expect(user.locator("#doc-modal")).to_be_visible()
expect(
tests/e2e/test_chat_history.py
+3 -7
View File
@@ -63,9 +63,6 @@ from e2e.auth_helpers import login
REPO = Path(__file__).resolve().parents[2]
FIXTURES = REPO / "tests" / "fixtures" / "docs"
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
#: Phase 10 viewer URL + phase 13 back=/ (the restored chip must be
#: byte-identical to the live-rendered one).
CHIP_HREF = "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F"
async def _import_fixtures(mock_port: int) -> ImportSummary:
@@ -279,10 +276,9 @@ def test_open_chat_returns_to_history(
# …the SAME answer text the History session saw (pixel-identical
# restore through renderStoredMessage)…
assert bubble.inner_text() == answer_before
# …with its source chip restored byte-identically.
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
expect(chip).to_have_count(1)
expect(chip.first).to_have_attribute("href", CHIP_HREF)
# …with ZERO citation chips restored — the saved turn read
# nothing, so its sources list is empty (phase 119, LOCKED A1;
# the retired phase-118 A4 suggested-chip pin is gone).
# The conversation continues: a new turn streams fine…
_ask(page, "How is my Kubernetes cluster set up? (hist-open-2)")
tests/e2e/test_chat_persistence.py
+11 -10
View File
@@ -42,9 +42,6 @@ OFF_TOPIC = "How do I bake sourdough bread?"
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
DEFLECT_PHRASE = r"haven't done anything like that"
STORAGE_KEY = "bor.chat.v1"
#: Phase 10 viewer URL + phase 13 back=/ (the restored chip must be
#: byte-identical to the live-rendered one).
CHIP_HREF = "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F"
async def _import_fixtures(mock_port: int) -> ImportSummary:
@@ -143,22 +140,24 @@ def test_conversation_survives_reload(
assert MOCK_ANSWER_MARKER in brain["text"]
assert "<" not in brain["text"], "persisted brain text must be raw, not rendered HTML"
assert brain["deflected"] is False
assert any(s["path"] == "homelab/kubernetes.md" for s in brain["sources"])
# Phase 119 (LOCKED A1): the turn read nothing, so its done sources
# — and the persisted record — are EMPTY (the suggested kubernetes
# doc is context, not a citation; the retired phase-118 A4 union is
# gone).
# Refresh — the same context keeps its localStorage.
page.reload()
expect(page.locator("#empty-state")).to_be_hidden()
# Both bubbles restored: text + the source chip with the exact viewer URL.
# Both bubbles restored: the answer text, and ZERO citation chips
# (the zero-read turn persisted an empty sources list — phase 119,
# LOCKED A1; the retired phase-118 A4 chip pin is gone).
expect(page.locator(".msg.user .bubble")).to_have_count(1)
expect(page.locator(".msg.user .bubble")).to_contain_text(QUESTION)
bubble = page.locator(".msg.brain .bubble")
expect(bubble).to_have_count(1)
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
expect(chip).to_have_count(1)
expect(chip.first).to_have_attribute("href", CHIP_HREF)
expect(chip.first).not_to_have_attribute("target") # phase 26: modal, not a new tab
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# The restore is read-only: storage still holds the same two messages.
assert [m["who"] for m in _stored_parsed(page)["messages"]] == ["user", "brain"]
@@ -294,7 +293,9 @@ def test_persists_across_page_navigation(
expect(page.locator(".msg.user .bubble").first).to_contain_text(QUESTION)
expect(page.locator(".msg.user .bubble").nth(1)).to_contain_text(OFF_TOPIC)
expect(page.locator(".msg.brain .bubble").first).to_contain_text(MOCK_ANSWER_MARKER)
expect(page.locator(".msg.brain .source-chip", has_text="kubernetes.md")).to_have_count(1)
# Zero citation chips (phase 119, LOCKED A1 — the grounded turn read
# nothing; the retired phase-118 A4 union is gone).
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
deflected = page.locator(".msg.brain.is-deflected .bubble")
expect(deflected).to_have_count(1)
expect(deflected.first).to_contain_text(re.compile(DEFLECT_PHRASE, re.IGNORECASE))
tests/e2e/test_chat_rag.py
+21 -17
View File
@@ -97,18 +97,12 @@ def test_on_topic_question_streams_grounded_answer(
expect(bubble.first).to_contain_text(QUESTION, timeout=30_000)
expect(bubble.first).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
# Grounded: a kubernetes.md source chip renders under the bubble
# (top-N docs can add more chips; the question's doc must be among them).
# Phase 26: the chip opens the document in the SAME-PAGE modal — no new
# tab; the encoded href stays as the no-JS / context-menu escape hatch
# (phase 13's back=/ lets the viewer's back button return to chat).
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
expect(chip).to_have_count(1)
expect(chip.first).to_contain_text("kubernetes.md")
expect(chip.first).to_have_attribute(
"href", "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F"
)
expect(chip.first).not_to_have_attribute("target") # phase 26: modal, not a new tab
# Phase 119 (LOCKED A1): a zero-read grounded turn chips NOTHING —
# the suggested kubernetes.md is seed context, not a citation chip
# (the retired phase-118 A4 suggested+read union is gone; the
# pre-phase-26 chip/contract pins retired with it). The grounding is
# pinned by the query_log row below (LOCKED A3, untouched).
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# Button recovers: enabled + "Send" (never stale).
expect(page.locator("#send-btn")).to_be_enabled()
@@ -121,9 +115,16 @@ def test_chat_logs_query(page: Page, app_url: str, mock_llm: int, db_ready: None
login(page, app_url, next="/")
page.fill("#message-input", QUESTION)
page.click("#send-btn")
expect(
page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
).to_have_count(1, timeout=30_000)
# The turn settles with the mock answer (phase 119 A1: no chip to
# wait on — a zero-read turn chips nothing; the retired phase-118
# A4 union is gone). The button recovery is the settle sync: the
# client re-enables Send on the done frame, and the server writes
# the query_log row just before yielding it.
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
"Deterministic mock answer for E2E", timeout=30_000
)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
# App still healthy after the turn.
r = httpx.get(f"{app_url}/api/health", timeout=5)
@@ -180,5 +181,8 @@ def test_sse_stream_shape(app_url: str, mock_llm: int, db_ready: None) -> None:
assert frames[-1]["type"] == "done" # done is the final event
assert done[0]["deflected"] is False
assert done[0]["suggestions"] == []
assert done[0]["sources"], "done must carry the cited sources"
assert any(s["path"] == "homelab/kubernetes.md" for s in done[0]["sources"])
# Phase 119 (LOCKED A1): done.sources = the READ docs only — this
# plain turn read nothing, so the frame carries an EMPTY sources
# list (the retired phase-118 A4 suggested+read union is gone; the
# suggested doc's durable record lives in query_log, not the frame).
assert done[0]["sources"] == [], done[0]["sources"]
tests/e2e/test_dark_tech_theme.py
+5 -3
View File
@@ -358,8 +358,9 @@ def test_behavior_unchanged_smoke(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""AC6: layout/behavior unchanged under the new skin — an on-topic
question streams a grounded answer, renders a source chip, and the send
button recovers (never stale)."""
question streams a grounded answer (phase 119, LOCKED A1: a zero-read
turn renders ZERO source chips — the retired phase-118 A4
suggested-chip is gone) and the send button recovers (never stale)."""
_seed_kb(mock_llm)
page.set_default_timeout(30_000)
login(page, app_url, next="/") # phase 79: chat is require_user-gated
@@ -373,7 +374,8 @@ def test_behavior_unchanged_smoke(
bubble.first.wait_for(state="visible", timeout=30_000)
expect(bubble.first).to_contain_text(QUESTION, timeout=30_000)
expect(bubble.first).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
expect(page.locator(".msg.brain .source-chip", has_text="kubernetes.md")).to_have_count(1)
# Phase 119 (LOCKED A1): zero-read turn → zero citation chips.
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# The state machine settled: button re-enabled, label back to "Send".
expect(page.locator("#send-btn")).to_be_enabled()
tests/e2e/test_document_back_navigation.py
+20 -4
View File
@@ -19,10 +19,16 @@ viewer URL survives as each link's ``href`` (the no-JS / context-menu
"open in new tab" escape hatch), so the back contract is asserted on
that exact href and verified by navigating to it directly.
Phase 119 re-target (LOCKED A1): chips cite READ docs only — a plain
question chips nothing, so test 1 drives the mock's scripted
summary-read flow (``SUMMARY_SEED_READ_TRIGGER``): the turn ``read``s
the kubernetes fixture, and its READ-doc chip is what carries the
``&back=%2F`` href the story asserts.
Test → story mapping (Playwright Mapping Rule):
1. ``test_back_from_chat_returns_to_chat`` — question → source chip href
(carries ``&back=%2F``) → viewer back link href ``/`` labeled "Chat"
→ click → the chat page.
1. ``test_back_from_chat_returns_to_chat`` — scripted-read question →
the read doc's source chip href (carries ``&back=%2F``) → viewer
back link href ``/`` labeled "Chat" → click → the chat page.
2. ``test_back_from_sources_returns_to_sources`` — Sources table link
href (no ``back`` param) → back link href ``/sources.html`` labeled
"Sources" → click → the Sources page.
@@ -49,7 +55,15 @@ from e2e.auth_helpers import login
REPO = Path(__file__).resolve().parents[2]
FIXTURES = REPO / "tests" / "fixtures" / "docs"
QUESTION = "How is my Kubernetes cluster set up?"
# Phase 119 (LOCKED A1): the chip is the turn's READ doc — the scripted
# summary-read flow (mock_llm.SUMMARY_SEED_READ_TRIGGER) reads the
# kubernetes fixture so the story's chip (with its back=%2F href)
# exists (a zero-read turn would chip nothing — the retired phase-118
# A4 suggested-chip is gone).
QUESTION = (
"Read the suggested document: read docs/homelab/kubernetes.md — "
"how is my Kubernetes cluster set up?"
)
# Seeded fixture doc (source=docs) shared by every test in this file.
DOC_SOURCE = "docs"
DOC_PATH = "homelab%2Fkubernetes.md"
@@ -111,6 +125,8 @@ def test_back_from_chat_returns_to_chat(
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
expect(chip).to_have_count(1, timeout=30_000)
# Phase 119 A1: the read doc is the turn's ONLY chip.
expect(page.locator(".msg.brain .source-chip")).to_have_count(1)
# Chat chips carry back=/ (encoded %2F) so the viewer knows where
# home is. Phase 26: the left click opens the same-page modal (no
# target=_blank); this href is what the no-JS / context-menu "open
tests/e2e/test_document_dates.py
+19 -18
View File
@@ -540,12 +540,14 @@ def test_old_correct_beats_new_similar(
page: Page, app_url: str, mock_llm: int, db_ready: None, dates_tree: Path
) -> None:
"""``How did I configure the backup retention policy?`` → grounded,
and the FIRST cited source is the OLDER correct doc (2020) — the
NEWER similar one (now, "under review") is cited second. Phase 118
(A4): the chip row IS the suggested tier (top-5, NO floor) — this
four-doc KB chips ALL four docs in rank order — and the related row
is absent (no rank-6+ doc). The real hybrid retriever + the DEFAULT
recency boost (0.0007 / 365 d) over the mock's token-overlap
and the FIRST logged source is the OLDER correct doc (2020) — the
NEWER similar one (now, "under review") is second. Phase 119
(LOCKED A1): the zero-read turn chips NOTHING (the chip row is the
READ docs only — the retired phase-118 A4 suggested-chip row is
gone), so the ordering assertion rides the durable record (LOCKED
A3 — all four docs, suggested rank order, untouched); the related
row is absent (no rank-6+ doc). The real hybrid retriever + the
DEFAULT recency boost (0.0007 / 365 d) over the mock's token-overlap
embeddings (the module docstring records the measured fused scores:
0.032523 vs 0.031498 — margin ≈ 0.001025 WITH the full zero-age
boost on the newer doc)."""
@@ -563,21 +565,20 @@ def test_old_correct_beats_new_similar(
# top_score 0.6222 ≥ the e2e threshold 0.30).
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
# Phase 118 (A4): the chip row IS the suggested tier — top-5, NO
# floor, and this KB has exactly four docs, so ALL four are
# suggested (chipped) in fused rank order: the OLDER correct doc
# first, the NEWER similar one (the boost's intended beneficiary)
# second, then the two unrelated docs (no floor filters them — the
# LLM decides what the summaries earn).
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(4, timeout=30_000)
assert chips.nth(0).inner_text() == f"{dates_tree.name}/{RETENTION_MD}"
assert chips.nth(1).inner_text() == f"{dates_tree.name}/{DRAFT_MD}"
assert chips.nth(2).inner_text() == f"{dates_tree.name}/{FORWARD_MD}"
assert chips.nth(3).inner_text() == f"{dates_tree.name}/{OLDDOC_MD}"
# Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation
# chips — the four suggested docs (all of them, top-5 NO floor on a
# four-doc KB: the OLDER correct doc first, the NEWER similar one
# second, then the two unrelated docs) seeded the prompt but never
# chip (the retired phase-118 A4 union is gone). Their rank order
# is pinned by the durable record below (LOCKED A3, untouched).
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# No rank-6+ doc exists in this four-doc KB → the related row is
# absent (the de-emphasized row renders only when it has entries).
expect(page.locator(".msg.brain .related-docs")).to_have_count(0)
# The button label is the settle sync: the client re-labels Send on
# the done frame, and the server writes the query_log row just
# before yielding it.
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
# Durable record: one row, grounded, the FULL retrieval (suggested
# tier + related + read, deduped — here: all four docs) in rank
tests/e2e/test_document_summaries.py
+10 -9
View File
@@ -171,8 +171,9 @@ def test_summary_hit_seeds_the_summary_not_the_full_text(
quotes the last suggested doc's summary tail (digest + pointer), and
the sentinel (the yaml doc's last line, outside the digest) is
ABSENT: the full source doc never reached the prompt (A6). Both
fixture docs are suggested (two-doc KB, no floor) and both chips
render (deflected: false)."""
fixture docs are suggested (two-doc KB, no floor) but chip
NOTHING — the turn read nothing, so (phase 119, LOCKED A1) the
citation surface is empty (deflected: false)."""
_reset_db()
summary = _run_in_thread(_import_fixtures(mock_llm))
assert summary.added == 2 # yaml + md control
@@ -225,13 +226,13 @@ def test_summary_hit_seeds_the_summary_not_the_full_text(
expect(bubble).to_contain_text(yaml_digest_line[-80:])
expect(bubble).not_to_contain_text(SENTINEL)
# Grounded: both fixture docs are suggested (two-doc KB — no floor)
# and both chips render, in rank order (md first, yaml last).
chip = page.locator(".msg.brain .source-chip", has_text=YAML_PATH)
expect(chip).to_have_count(1)
expect(chip.first).to_contain_text(f"{SOURCE}/{YAML_PATH}")
expect(page.locator(".msg.brain .source-chip", has_text=MD_PATH)).to_have_count(1)
expect(page.locator(".msg.brain .source-chip")).to_have_count(2)
# Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation
# chips — both fixture docs are suggested (two-doc KB — no floor,
# md first, yaml last) but suggested docs are seed context, not
# citations (the retired phase-118 A4 suggested-chip union is gone);
# their rank order is pinned by the durable record below (LOCKED
# A3, untouched).
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# No rank-6+ doc in a two-doc KB → the related row is absent.
expect(page.locator(".msg.brain .related-docs")).to_have_count(0)
tests/e2e/test_document_viewer.py
+28 -4
View File
@@ -11,9 +11,18 @@ deterministic mock embeddings (same harness as the phase-10 suite — only
the assertions changed: chips/row links now open the SAME-PAGE modal,
no ``expect_popup``).
Phase 119 re-target (LOCKED A1): chips cite READ docs only — a plain
question chips nothing, so the chat-driven tests here run the
mock's scripted summary-read flow (``SUMMARY_SEED_READ_TRIGGER`` —
the house scripted-turn convention): the turn ``read``s
``docs/homelab/kubernetes.md``, whose read result is what earns the
kubernetes.md chip the story interacts with (the chip's viewer-link
contract is unchanged).
Test → story mapping (Playwright Mapping Rule):
1. ``test_source_chip_opens_modal`` — chat chip → modal opens in-page
(NO new tab, URL unchanged), title + ``.doc-md`` content + meta row.
1. ``test_source_chip_opens_modal`` — chat chip (the turn's READ doc —
phase 119 A1) → modal opens in-page (NO new tab, URL unchanged),
title + ``.doc-md`` content + meta row.
2. ``test_sources_row_opens_modal`` — Sources path link → modal, yaml in
``<pre.doc-raw>``, mono font, URL unchanged.
3. ``test_modal_closes_on_button_escape_and_backdrop`` — close via
@@ -56,7 +65,14 @@ from e2e.auth_helpers import login
REPO = Path(__file__).resolve().parents[2]
FIXTURES = REPO / "tests" / "fixtures" / "docs"
SCREENSHOTS = REPO / ".agents" / "screenshots" # house convention for visual records
QUESTION = "How is my Kubernetes cluster set up?"
# Phase 119 (LOCKED A1): the chip is the turn's READ doc — the scripted
# summary-read flow (mock_llm.SUMMARY_SEED_READ_TRIGGER) reads the
# kubernetes fixture so the story's chip exists (the retired phase-118
# A4 suggested-chip is gone: a zero-read turn would chip nothing).
QUESTION = (
"Read the suggested document: read docs/homelab/kubernetes.md — "
"how is my Kubernetes cluster set up?"
)
async def _import_fixtures(mock_port: int) -> ImportSummary:
@@ -98,12 +114,20 @@ def _reset_db(mock_port: int, seed: bool) -> ImportSummary | None:
def _ask_for_chip(page: Page, app_url: str) -> Any:
"""Drive one chat turn and return the kubernetes.md source chip."""
"""Drive one chat turn and return the kubernetes.md source chip.
The scripted summary-read flow (phase 119, LOCKED A1): the turn
``read``s the kubernetes doc, so it is the turn's ONLY citation
chip (chips cite read docs only — the answer is the mock's
verbatim echo of the read result)."""
login(page, app_url, next="/") # phase 79: chat is require_user-gated
page.fill("#message-input", QUESTION)
page.click("#send-btn")
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
expect(chip).to_have_count(1, timeout=30_000)
# Phase 119 A1: the read doc is the turn's ONLY chip (exactly one
# .source-chip under the bubble — nothing suggested-but-unread).
expect(page.locator(".msg.brain .source-chip")).to_have_count(1)
return chip
tests/e2e/test_harness_aligned_tools.py
+3 -3
View File
@@ -455,10 +455,10 @@ def test_read_flow_lines_answer_sources_no_raw_markup(
)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
# Done-state sources include the read document (retrieval doc first,
# the agent's read doc after — the phase-37 extension contract).
# Phase 119 (LOCKED A1): done.sources = the READ docs only —
# exactly the scripted read; the retrieval (seed) doc was never
# read, so it never cites (the retired phase-118 A4 union is gone).
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(SEED_SOURCE, SEED_PATH),
(READ_SOURCE, READ_PATH),
]
tests/e2e/test_llm_history.py
+22 -13
View File
@@ -23,10 +23,12 @@ assistant message — A4).
The marker is checked BEFORE the mock's ``DEFLECT_MODE`` branch, so
the echo fires on BOTH turn branches — the branch under test is
discriminated separately (the grounded source chip / the
``is-deflected`` bubble class). The echo answers carry no tool
markup, so no marker tool flow is re-triggered by the now-always-
present (user/assistant-only) history.
discriminated separately (the persisted record's ``deflected`` flag /
the ``is-deflected`` bubble class — phase 119, LOCKED A1: a zero-read
grounded turn chips nothing, so the retired chip discriminator is
gone). The echo answers carry no tool markup, so no marker tool flow
is re-triggered by the now-always-present (user/assistant-only)
history.
The file name deliberately differs from phase 50's
``test_chat_history.py`` (save & view chat history — a different
@@ -170,12 +172,15 @@ def test_followup_receives_history_and_thinking(
expect(bubble).to_contain_text(f"last answer tail: {answer_tail}")
expect(bubble).to_contain_text("thinking: yes")
# Grounded proof — the echo fires in BOTH branches, so the branch
# is discriminated by the kubernetes.md source chip (the deflected
# turn carries no cited sources). Scoped to the LAST brain message:
# turn 1 cited kubernetes.md too.
# is discriminated by the persisted record's ``deflected`` flag
# (phase 119, LOCKED A1: the turn read nothing, so it also chips
# nothing — the retired chip discriminator is gone).
brain2 = _wait_record(page, 4)["messages"][3]
assert brain2["who"] == "brain"
assert brain2["deflected"] is False, "the echo turn must be the grounded branch"
expect(
page.locator(".msg.brain").last.locator(".source-chip", has_text="kubernetes.md")
).to_have_count(1)
page.locator(".msg.brain").last.locator(".source-chip")
).to_have_count(0)
def test_first_question_has_no_history(
@@ -194,11 +199,15 @@ def test_first_question_has_no_history(
expect(bubble).to_contain_text("history: 0 prior messages", timeout=30_000)
expect(bubble).to_contain_text("last answer tail: none")
expect(bubble).to_contain_text("thinking: no")
# Grounded: the echo question is on-topic (the chip proves the
# HIGH gate, not a deflection).
# Grounded: the echo question is on-topic (the record's deflected
# flag proves the HIGH gate, not a deflection — phase 119, LOCKED
# A1: the zero-read turn chips nothing, so the retired chip
# discriminator is gone).
record = _wait_record(page, 2)
assert record["messages"][1]["deflected"] is False
expect(
page.locator(".msg.brain").last.locator(".source-chip", has_text="kubernetes.md")
).to_have_count(1)
page.locator(".msg.brain").last.locator(".source-chip")
).to_have_count(0)
def test_deflected_followup_receives_history(
tests/e2e/test_llm_retry.py
+12 -12
View File
@@ -393,18 +393,18 @@ def test_dead_then_recovered_grounded(
_assert_no_error_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(SEED_SOURCE, SEED_PATH)
]
# Phase 119 (LOCKED A1): the grounded turn read NOTHING, so
# done.sources is empty (the suggested seed doc never chips — the
# retired phase-118 A4 union is gone); the durable record below
# still carries it (LOCKED A3, untouched).
assert done["sources"] == [], done["sources"]
# The grounded answer completed with the source chip — the agent
# round retried and the turn is intact.
# The grounded answer completed (no chips on a zero-read turn —
# phase 119 A1) — the agent round retried and the turn is intact.
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text(GROUNDED_Q)
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(1)
expect(chips.first).to_contain_text(SEED_SP)
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
_assert_no_error_banner(page)
row = _last_query_log()
@@ -446,12 +446,12 @@ def test_embedding_retry_completes(
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
# The turn completed normally with the grounded answer + chip.
# The turn completed normally with the grounded answer (zero chips
# — the turn read nothing, so the chip row is empty under phase
# 119, LOCKED A1; the retired phase-118 A4 union is gone).
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(1)
expect(chips.first).to_contain_text(SEED_SP)
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
_assert_no_error_banner(page)
tests/e2e/test_long_answers.py
+4 -4
View File
@@ -127,8 +127,8 @@ def test_normal_answer_unaffected(
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text("Deterministic mock answer for E2E", timeout=30_000)
expect(bubble).not_to_contain_text("LONG-ANSWER-END")
# Grounded: the question's own document is cited as a chip.
expect(
page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
).to_have_count(1, timeout=30_000)
# Grounded (the bubble + marker above) — and ZERO citation chips:
# the turn read nothing, so (phase 119, LOCKED A1) the chip row is
# empty (the retired phase-118 A4 suggested-chip is gone).
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
expect(page.locator("#send-btn")).to_be_enabled()
tests/e2e/test_ls_tree_drilldown.py
+2 -2
View File
@@ -678,8 +678,8 @@ def test_drill_down_sources_folders_files_and_read(
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The read document is in the turn's sources (retrieval + agent-read,
# deduped — the grounded-turn record).
# The read document is in the turn's sources (phase 119, LOCKED A1:
# done.sources is the READ docs only — this turn's one read).
assert any(
s["path"] == TWO_A and s["source"] == ALPHA for s in done["sources"]
), done["sources"]
tests/e2e/test_markdown_tables.py
+5 -3
View File
@@ -185,9 +185,11 @@ def test_chat_table_renders(
assert "|---|" not in bubble_text, "the |---| separator leaked into the bubble"
assert "| Service | Port | Host |" not in bubble_text, "the raw header row leaked"
# Grounded retrieval: the table fixture is the top source chip.
chip = page.locator(".msg.brain .source-chip", has_text="homelab/tables.md")
expect(chip).to_have_count(1)
# Grounded (the table answer is the mock's non-deflected table
# branch) — and ZERO citation chips: the turn read nothing, so
# (phase 119, LOCKED A1) the chip row is empty (the retired
# phase-118 A4 suggested-chip is gone).
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# ---------------------------------------------------------------------------
tests/e2e/test_name_signal_read_chips.py
+794
View File
@@ -0,0 +1,794 @@
"""Phase 119 E2E (Playwright, mock-only): the name-signal seeds the
right document — and chips cite READ docs only.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_name_signal_read_chips.py -v --no-cov
MOCK-ONLY suite: the story's gates are the deterministic contracts —
the mock's scripted-turn lenses (the tail echo, the single-read tool
flow, the new ``FOLDER_MAP_TRIGGER`` echo) make "what reached the
prompt / what got cited" assertable byte-exactly.
KB fixture — ``tests/fixtures/namekb`` (tracked; registered as a
local-directory source through the authenticated API + the REAL
in-process ``POST /api/sync``, the ``test_summary_seed_context.py``
pattern), seven markdown documents in one source (``namekb``) whose
bodies are token-controlled so the hybrid gate + the phase-119
name-hit bonus pick the intended tiers deterministically
(``synced_kb`` pins it with the app's REAL retrieval — a
fixture-text regression that breaks the design fails at setup with a
clear message, not at the wire assertions):
* every path carries a DIGITLESS question name token (the four
guides' stems hold ``install``, the two forgejo docs' stems +
folder hold ``forgejo``, ``zz-folder-map.md`` holds ``folder``) —
so the name-hit bonus applies to ALL seven docs (a uniform
``+0.005``) and the tier order reduces to the pure retrieval
signal (the bonus head-start is exercised, the ordering is not
distorted by it);
* the four guides' bodies repeat the question phrase a strength
gradient (×4/×3/×2/×1) so they OUT-RANK the name-hit docs on
vector/FTS; the forgejo docs and the filler doc carry NEUTRAL
bodies (their name is their main signal — the phase-119 owner
scenario: "the file's name is the distinctive part");
* engineered invariant (asserted in ``synced_kb`` for ALL FOUR
scripted questions, measured + probe-verified, stable across
re-imports): the name-hit doc ``forgejo/forgejo-home.md`` is the
LAST of the top-5 suggested (the name signal lifts it into the
tier, the weak body keeps it at the back — its ``Source:`` tail
line is the tail echo), and the related tier is the rank-6+
remainder (``forgejo-nginx`` + ``zz-folder-map``; for the folder
question the two swap order — both stay rank 6+).
Test → contract mapping (task 06 cases a–d; one Playwright file per
story, A16):
1. ``test_name_hit_doc_is_last_suggested`` — (a): the distinctive
question + ``show the end of your notes`` ⇒ the answer quotes the
last 160 chars of the seeded ``<documents>`` block — the
name-hit doc's SUMMARY tail (its ``Source: namekb/forgejo/
forgejo-home.md`` pointer line; no other doc's pointer line, no
doc's tail sentinel — the summary seed, not the full text). The
zero-read grounded turn chips NOTHING (LOCKED A1); the related
row renders rank 6+; the durable record carries suggested +
related (118-A3 untouched).
2. ``test_single_read_chips_only_the_read_doc`` — (b): the
``use your tools`` flow (the mock's single read: ``ls`` → drill
``ls(namekb)`` → read the FIRST file line —
``namekb/zz-folder-map.md``, the root file) ⇒ the bubble carries
EXACTLY ONE ``.source-chip`` — the read doc (LOCKED A1); none of
the five suggested docs chips; the de-emphasized related row
renders (rank 6+ deduped against the cited read doc — the read
doc is a chip, never a "nearby doc"); the durable record carries
suggested + related + read (118-A3 untouched).
3. ``test_zero_read_grounded_turn_chips_nothing`` — (c): a PLAIN
distinctive question (no tool trigger) ⇒ the ``done`` frame's
``sources == []`` and ZERO ``.source-chip`` elements in the
bubble — LOCKED A1's visible consequence; the answer still
renders grounded (deflected: false); the related row renders.
4. ``test_folder_lines_echoed_verbatim`` — (d): the distinctive
question + ``repeat your folder map`` (the new mock trigger) ⇒
the answer ends with the suggested-folder lines VERBATIM — the
name-hit doc's folder line ``namekb/forgejo/: forgejo-nginx.md``
(the line prefix + its known sibling entry) alongside the
guides' folder line.
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import httpx
import pytest
from playwright.sync_api import Locator, Page, expect
from sqlalchemy import select, text
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.models import Document, QueryLog
from app.rag.retriever import TRUNCATION_MARKER, retrieve, select_related, select_suggested
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
from e2e.mock_llm import TOKEN_RE, embed_text
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module app binds its own port instead
# (a same-port second uvicorn dies on bind and would drive the wrong
# server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_NAMESIG", "8145"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
SOURCE = "namekb" # the local directory's basename = the source name
FIXTURES = REPO / "tests" / "fixtures" / "namekb"
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
# --------------------------------------------------------------------------
# Fixture documents (tracked — tests/fixtures/namekb; deterministic,
# token-controlled — see the module docstring for the design)
# --------------------------------------------------------------------------
#: 22 neutral tokens (no question tokens) — the digest + filler
#: material of the neutral docs; the guides' strength phrase is
#: question-token-only.
PRELUDE = (
"fixture preamble block one two three four five six seven eight nine ten "
"eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen"
)
assert len(TOKEN_RE.findall(PRELUDE.lower())) == 22
PHRASE = "install forgejo with the deployment steps"
#: (path, title, phrase repeats, filler line, tail sentinel) — the
#: guides carry the strength gradient (×4/×3/×2/×1); the forgejo docs
#: (name hits — distinctive digitless ``forgejo`` component) and the
#: root filler carry NEUTRAL bodies (0 repeats).
DOCS: list[tuple[str, str, int, str, str]] = [
("deploy/install-guide-a.md", "Install Forgejo Guide A", 4,
"alpha1 alpha2 alpha3 alpha4 alpha5 alpha6", "NAMEKB-TAIL-c5d6"),
("deploy/install-guide-b.md", "Install Forgejo Guide B", 3,
"beta1 beta2 beta3 beta4 beta5 beta6", "NAMEKB-TAIL-d7e8"),
("deploy/install-guide-c.md", "Install Forgejo Guide C", 2,
"gamma1 gamma2 gamma3 gamma4 gamma5 gamma6", "NAMEKB-TAIL-f9a0"),
("deploy/install-guide-d.md", "Install Forgejo Guide D", 1,
"delta1 delta2 delta3 delta4 delta5 delta6", "NAMEKB-TAIL-b1c2"),
("forgejo/forgejo-home.md", "Zeta Forge One", 0,
"kilo1 kilo2 kilo3 kilo4 kilo5 kilo6", "NAMEKB-TAIL-a1b2"),
("forgejo/forgejo-nginx.md", "Zeta Forge Two", 0,
"lambda1 lambda2 lambda3 lambda4 lambda5 lambda6", "NAMEKB-TAIL-b3c4"),
("zz-folder-map.md", "Zeta Misc Three", 0,
"micro1 micro2 micro3 micro4 micro5 micro6", "NAMEKB-TAIL-e3f4"),
]
#: The engineered invariant (measured, probe-verified, stable across
#: re-imports — pinned for ALL FOUR scripted questions): the name-hit
#: doc is the LAST of the top-5 suggested; the guides lead in
#: retrieval-strength order (the ×2/×1 docs swap under the md5
#: collision noise — pinned as measured, not as the gradient order).
SUGGESTED = [
"deploy/install-guide-a.md",
"deploy/install-guide-b.md",
"deploy/install-guide-d.md",
"deploy/install-guide-c.md",
"forgejo/forgejo-home.md",
]
NAME_HIT = "forgejo/forgejo-home.md" # the (a) tail-echo target
RELATED = ["forgejo/forgejo-nginx.md", "zz-folder-map.md"]
#: The folder question's rank-6+ order (the filler's name hit —
#: ``folder`` — lifts it above the sibling; both stay rank 6+).
RELATED_FOLDER = ["zz-folder-map.md", "forgejo/forgejo-nginx.md"]
READ_TARGET = "zz-folder-map.md" # the single-read's first file line
SENTINELS = [sentinel for _p, _t, _i, _f, sentinel in DOCS]
ALL_SOURCE_LINES = [f"Source: {SOURCE}/{p}" for p, _t, _i, _f, _s in DOCS]
#: The turn's questions (the mock's trigger phrases — see the module
#: docstring). The base question's name tokens (``install``,
#: ``forgejo``) ground every turn (best cosine ≈ 0.35–0.41 ≥ the E2E
#: 0.30 threshold) and fire the D1 name hits.
BASE_QUESTION = "How do I install forgejo?"
TAIL_QUESTION = BASE_QUESTION + " show the end of your notes"
TOOLS_QUESTION = BASE_QUESTION + " use your tools"
FOLDER_QUESTION = BASE_QUESTION + " repeat your folder map"
assert "show the end of your notes" in TAIL_QUESTION.lower()
assert "use your tools" in TOOLS_QUESTION.lower()
assert "repeat your folder map" in FOLDER_QUESTION.lower()
def _expected_summary(content: str, path: str) -> str:
"""The mock lite model's byte-stable digest + the code pointer line
(mirrors ``mock_llm.compose_answer``'s ``SUMMARY_MODE`` branch —
first 24 tokens of the document content — plus the summarizer's
deterministic ``Source:`` line; no model output is ever trusted)."""
digest = " ".join(TOKEN_RE.findall(content.lower())[:24])
return f"This document covers {digest}.\nSource: {SOURCE}/{path}"
# --------------------------------------------------------------------------
# Fixtures
# --------------------------------------------------------------------------
@pytest.fixture(scope="module")
def seed_dirs() -> Path:
"""The story's local-directory source: the tracked fixture dir
(the app server runs on the same host, so the path is visible to
it). The directory's basename is the source name (``kind=local``,
phase 38)."""
assert FIXTURES.is_dir(), "tests/fixtures/namekb is missing"
for path, _t, _i, _f, _s in DOCS:
assert (FIXTURES / path).is_file(), f"fixture doc {path} is missing"
return FIXTURES
@pytest.fixture(scope="module")
def app_server(mock_llm: int, seed_dirs: Path) -> Iterator[str]:
"""The real app under test — per-module app (the conftest pattern,
cf. ``test_summary_seed_context.py``): NO ``BOR_GIT_SOURCES`` (the
env fallback is git-only — the source here is a DB-registered local
directory), the mock LLM, the mock-calibrated threshold, and the
leak-guarded code defaults (the suggested/related tier settings
ride their code defaults — 5 / 2 — exactly like the production
``.env``-free defaults). The session app is never started in this
isolated run, so no port clash."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated gate (conftest pattern): the strength gradient
# keeps every scripted turn grounded (best cosine ≈ 0.35–0.41).
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
# Phase 67: instant retry waits + the code-default budget (the
# conftest leak-guard pattern).
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
# reads it from cwd) — override it with an EMPTY value (the env var
# beats the .env file): the registry must hold EXACTLY the local
# directory this suite registers (a leftover env git list would
# pollute the KB the scripted turns run against).
env["BOR_GIT_SOURCES"] = ""
# Leak guards (conftest pattern): an operator's local (gitignored)
# .env cannot leak corpus-specific settings into the app under test.
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
_Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
share one Postgres, so a leftover git_sources row or document would
pollute the retrieval the scripted turns run against (the
name-signal margins are pinned against EXACTLY these seven
documents)."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources, folder_summaries"
)
)
db.commit()
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
"""Poll the (cookie-authenticated) status endpoint until the run
reaches a terminal state (the test_summary_seed_context pattern,
over plain httpx — this fixture has no browser page yet)."""
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
body = r.json()
if body["state"] in ("success", "failed"):
return body
time.sleep(0.5)
raise AssertionError(f"sync did not reach a terminal state: {body}")
def _assert_tiers(
question: str, related_paths: list[str]
) -> None:
"""Pin the name-signal design with the app's REAL hybrid retrieval
over the mock's embeddings (deterministic): the suggested tier is
exactly the four guides (retrieval-strength order) + the name-hit
doc LAST (LOCKED A3 — top-5, NO floor — the name-hit bonus, D2,
lifts it into the tier) and the related tier is the rank-6+
remainder (``related_max_docs`` = 2). A fixture-text regression
that breaks the design fails here, at setup, with a clear
message."""
with SessionLocal() as db:
chunks = retrieve(db, question, embed_text(question))
suggested = [f"{d.source}/{d.path}" for d in select_suggested(chunks)]
related = [
f"{d.source}/{d.path}"
for d in select_related(
chunks,
{d.id for d in select_suggested(chunks)},
_Settings.model_fields["related_max_docs"].default,
)
]
expected = [f"{SOURCE}/{p}" for p in SUGGESTED]
assert suggested == expected, (
f"suggested tier drifted for {question!r}: {suggested} "
f"(expected {expected})"
)
assert related == [f"{SOURCE}/{p}" for p in related_paths], (
f"related tier drifted for {question!r}: {related}"
)
@pytest.fixture(scope="module")
def synced_kb(app_server: str, seed_dirs: Path) -> None:
"""The story's precondition: the one-source KB synced under the
deterministic mock. Registers the fixture directory through the
authenticated API (the ``test_local_directory_sources.py``
pattern), runs the REAL in-process sync (``POST /api/sync`` — walk
→ chunk → embed → summaries → overview → folder summaries →
version bump), pins the stored content byte-identical to the
tracked fixture files, pins LOCKED A2 end-to-end (every doc stores
the mock's byte-stable digest + exactly one embedded
``is_summary`` chunk), and pins the engineered invariant (the
name-hit doc LAST of the top-5) for ALL FOUR scripted questions."""
_truncate_all()
with httpx.Client(base_url=app_server, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.post(
"/api/git-sources", json={"kind": "local", "path": str(seed_dirs)}
)
assert r.status_code == 201, r.text
r = client.post("/api/sync")
assert r.status_code == 202, r.text
body = _wait_sync_done_http(client)
assert body["state"] == "success", body
detail = body["detail"]
assert detail["added"] == len(DOCS), detail
assert detail["pruned"] == 0, detail
# The import stored the tracked fixture strings BYTE-IDENTICALLY
# and, for EVERY doc, the mock's byte-stable digest: the
# deterministic assertion surface of the whole suite.
with SessionLocal() as db:
for path, _title, _i, _f, _s in DOCS:
on_disk = (FIXTURES / path).read_text(encoding="utf-8")
stored = db.scalar(
select(Document).where(
Document.source == SOURCE, Document.path == path
)
)
assert stored is not None, f"fixture doc {path} was not imported"
assert stored.content == on_disk, f"stored content drifted for {path}"
assert stored.summary == _expected_summary(on_disk, path), (
f"summary is not the mock digest for {path}: {stored.summary!r}"
)
schunks = [c for c in stored.chunks if c.is_summary]
assert len(schunks) == 1 and schunks[0].position == -1, (
f"expected exactly one is_summary chunk for {path}"
)
assert schunks[0].embedding is not None, (
f"the is_summary chunk of {path} is not embedded"
)
_assert_tiers(TAIL_QUESTION, RELATED)
_assert_tiers(TOOLS_QUESTION, RELATED)
_assert_tiers(FOLDER_QUESTION, RELATED_FOLDER)
_assert_tiers(BASE_QUESTION, RELATED)
@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
"""Per-test query_log isolation (the KB itself is module-scoped —
the scripted turns never change it, so the registry and the KB
persist across the tests of this module)."""
with SessionLocal() as db:
db.execute(text("TRUNCATE query_log"))
db.commit()
yield
with SessionLocal() as db:
db.execute(text("TRUNCATE query_log"))
db.commit()
# --------------------------------------------------------------------------
# Page helpers (the test_summary_seed_context house patterns)
# --------------------------------------------------------------------------
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
#: (a response clone read in the background) — wire-level assertions
#: for the ``tool`` / ``done`` frames, independent of the UI rendering.
SSE_HOOK = """
() => {
if (window.__sseInstalled) return;
window.__sseInstalled = true;
window.__sseFrames = [];
const origFetch = window.fetch;
window.fetch = async function (...args) {
const res = await origFetch.apply(this, args);
try {
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
if (url.includes('/api/chat')) {
res.clone().text().then((bodyText) => {
for (const block of bodyText.split('\\n\\n')) {
const line = block.trim();
if (line.startsWith('data: ')) {
window.__sseFrames.push(line.slice(6));
}
}
});
}
} catch (e) { /* non-clonable responses: ignored */ }
return res;
};
}
"""
def _install_page_hooks(page: Page) -> None:
page.evaluate(SSE_HOOK)
def _frames(page: Page) -> list[dict]:
"""The SSE frames captured since the last submit (``_submit``
clears the buffer), once the hook's background read settles."""
deadline = time.monotonic() + 30.0
while True:
raw = page.evaluate("() => window.__sseFrames || []")
parsed = [json.loads(line) for line in raw if line]
if any(f.get("type") == "done" for f in parsed):
return parsed
if time.monotonic() > deadline:
raise AssertionError(
f"SSE hook captured no `done` frame (frames so far: "
f"{len(parsed)}) — hook install failed?"
)
time.sleep(0.05)
def _tool_frames(frames: list[dict]) -> list[dict]:
return [f for f in frames if f.get("type") == "tool"]
def _submit(page: Page, question: str) -> None:
page.evaluate("window.__sseFrames = []")
page.fill("#message-input", question)
page.click("#send-btn")
# The user bubble lands synchronously with the submit handler.
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
def _wait_settled(page: Page) -> None:
"""The turn is complete: answer text in the bubble, button recovered
(the phase-48 settle wait)."""
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
def _last_brain(page: Page) -> Locator:
return page.locator(".msg.brain").last
def _last_query_log() -> QueryLog:
with SessionLocal() as db:
rows = db.scalars(select(QueryLog)).all()
assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}"
return rows[0]
def _assert_chip_row(page: Page, read_paths: list[str]) -> None:
"""The citation surface of a grounded turn (phase 119, LOCKED A1 —
the retired phase-118 A4 union is gone): the chip row is the
AGENT-READ docs only (a zero-read turn chips NOTHING: the
suggested docs are seed context, not citations)."""
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(len(read_paths))
for path in read_paths:
expect(
chips.filter(has_text=path), message=f"chip for {path}"
).to_have_count(1)
def _assert_related_row(page: Page, related_paths: list[str]) -> None:
"""The de-emphasized ``related-docs`` row (phase-113 UI, untouched
by phase 119 — never a citation chip): the rank-6+ remainder
deduped against the cited read docs (a read related doc is a chip,
never a "nearby doc")."""
row = page.locator(".msg.brain .related-docs")
expect(row).to_have_count(1)
expect(row.first).to_have_attribute("aria-label", "Nearby docs, in case")
links = page.locator(".msg.brain .related-docs .related-doc")
expect(links).to_have_count(len(related_paths))
for i, path in enumerate(related_paths):
expect(links.nth(i)).to_contain_text(f"{SOURCE}/{path}")
# The related links are never citation chips.
expect(page.locator(".msg.brain .related-docs .source-chip")).to_have_count(0)
# --------------------------------------------------------------------------
# 1. (a) the name-signal doc is the LAST of the top-5 suggested — the
# tail echo proves it reached the prompt as the last seed block
# --------------------------------------------------------------------------
def test_name_hit_doc_is_last_suggested(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
"""The tail-echo question quotes the last 160 chars of the seeded
``<documents>`` block: they end in the LAST suggested doc's SUMMARY
— the name-hit doc's byte-stable digest tail + pointer line (the
digitless-component rule + the D2 bonus put ``forgejo-home`` in
the tier, its neutral body keeps it last). No other doc's pointer
line and no doc's tail sentinel are in the echoed context — the
summary seed, not the full text. The zero-read grounded turn
chips nothing (LOCKED A1); the related row renders rank 6+; the
durable record carries suggested + related (118-A3 untouched)."""
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, TAIL_QUESTION)
_wait_settled(page)
# No tools on this turn — the answer is the mock's direct tail echo
# of the seeded context (the summary-seed lens).
frames = _frames(page)
assert _tool_frames(frames) == [], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# Phase 119 (LOCKED A1): the grounded done frame cites the READ
# docs only — this turn read NOTHING (the summary-only fast path),
# so the citation surface is empty; the suggested set (name-hit
# doc included) is seed context, not citations.
assert done["sources"] == [], done["sources"]
bubble = _last_brain(page).locator(".bubble")
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
# The echoed tail ends in the name-hit doc's summary: its
# deterministic ``Source:`` pointer line (the digest is pinned in
# ``synced_kb`` — a content preview would carry the neutral-body
# text instead, and the pointer line only ever exists on a stored
# summary). The bubble renders the answer as markdown, which
# collapses the summary's newline — pin the pointer line alone
# (it sits inside the echoed 160 chars, byte-exact).
expect(bubble).to_contain_text(f"Source: {SOURCE}/{NAME_HIT}")
# No OTHER doc's pointer line is in the echoed tail — the
# name-hit doc was the LAST suggested block (the (a) invariant's
# visible proof).
for path, _t, _i, _f, _s in DOCS:
if path != NAME_HIT:
expect(bubble).not_to_contain_text(f"Source: {SOURCE}/{path}")
# And no document's tail sentinel: the full content of no
# suggested doc reached the model (summary seed only).
for sentinel in SENTINELS:
expect(bubble).not_to_contain_text(sentinel)
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
# The zero-read grounded turn chips NOTHING (LOCKED A1); the
# related row renders rank 6+ (the de-emphasized row, untouched).
_assert_chip_row(page, [])
_assert_related_row(page, RELATED)
# Durable record: grounded; suggested + related (LOCKED A3 — the
# log records retrieval, not citations; phase 119 A1 retires the
# chip surface only, not the record).
row = _last_query_log()
assert row.question == TAIL_QUESTION
assert row.deflected is False
assert row.sources == ", ".join(f"{SOURCE}/{p}" for p in SUGGESTED + RELATED)
# --------------------------------------------------------------------------
# 2. (b) chips = READ docs only — the single-read flow chips exactly
# the one read doc; the related row renders (deduped)
# --------------------------------------------------------------------------
def test_single_read_chips_only_the_read_doc(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
"""The ``use your tools`` flow drives the mock's single read:
``ls`` (the top level) → drill ``ls(namekb)`` → ``read`` the FIRST
file line (the root file ``zz-folder-map.md`` — subfolders list
first, files after). The bubble carries EXACTLY ONE
``.source-chip`` — the read doc (LOCKED A1); none of the five
suggested docs chips (the name-hit doc is seed context, not a
citation); the de-emphasized related row renders the rank-6+
remainder deduped against the cited read doc (the read doc is a
chip, never a "nearby doc")."""
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, TOOLS_QUESTION)
_wait_settled(page)
# Wire level: exactly three `tool` frames — ``ls`` (the top
# level), the drill ``ls`` scoped to the first (only) source
# (phase 94), then ``read`` the first file line's combined
# source/path — and all three ahead of the first `delta` frame.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "ls", "argument": None},
{"type": "tool", "name": "ls", "argument": SOURCE},
{"type": "tool", "name": "read", "argument": f"{SOURCE}/{READ_TARGET}"},
], _tool_frames(frames)
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
assert all(
i < first_delta for i, f in enumerate(frames) if f.get("type") == "tool"
)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# Phase 119 (LOCKED A1): done.sources = the READ docs only —
# exactly the one read; the five suggested docs (name-hit doc
# included) never chip.
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(SOURCE, READ_TARGET)
], done["sources"]
# done.related = the rank-6+ remainder deduped against the cited
# read doc — the read doc is rank 7 (IN the remainder) and drops
# out: only the sibling name-hit doc remains.
assert [(s["source"], s["path"]) for s in done["related"]] == [
(SOURCE, "forgejo/forgejo-nginx.md")
], done["related"]
# The final answer quotes the read document (the mock's
# deterministic quote: "Read <source/path>. <first 80 chars of its
# content>").
bubble = _last_brain(page).locator(".bubble")
expect(
bubble,
).to_contain_text(f"Read {SOURCE}/{READ_TARGET}.", timeout=30_000)
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
# The UI chip row: EXACTLY the one read doc (LOCKED A1) — and
# explicitly NONE of the suggested docs (the name-hit doc first).
_assert_chip_row(page, [READ_TARGET])
expect(
page.locator(".msg.brain .source-chip", has_text="install-guide")
).to_have_count(0)
expect(
page.locator(".msg.brain .source-chip", has_text=NAME_HIT)
).to_have_count(0)
# The related row renders rank 6+ deduped against the cited read
# doc (phase-113 behavior untouched).
_assert_related_row(page, ["forgejo/forgejo-nginx.md"])
# Durable record: grounded; suggested + related + read (deduped —
# the read doc sits in the related slot, LOCKED A3 — phase 119 A1
# retires the chip surface only, not the record).
row = _last_query_log()
assert row.question == TOOLS_QUESTION
assert row.deflected is False
assert row.sources == ", ".join(
f"{SOURCE}/{p}" for p in SUGGESTED + RELATED
)
# --------------------------------------------------------------------------
# 3. (c) a zero-read grounded turn chips nothing (LOCKED A1's visible
# consequence)
# --------------------------------------------------------------------------
def test_zero_read_grounded_turn_chips_nothing(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
"""A PLAIN distinctive question (no tool trigger) grounds on the
name tokens: the ``done`` frame's ``sources == []`` and ZERO
``.source-chip`` elements in the bubble — LOCKED A1's visible
consequence (the suggested docs, name-hit doc included, are seed
context, never citations); the answer still renders grounded
(deflected: false); the related row renders rank 6+."""
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, BASE_QUESTION)
_wait_settled(page)
frames = _frames(page)
assert _tool_frames(frames) == [], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# LOCKED A1's wire-level consequence: a grounded turn that read
# nothing cites nothing.
assert done["sources"] == [], done["sources"]
bubble = _last_brain(page).locator(".bubble")
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
# Zero chips — not just "no suggested chips": the citation
# surface is empty.
_assert_chip_row(page, [])
_assert_related_row(page, RELATED)
# Durable record: grounded; suggested + related (LOCKED A3).
row = _last_query_log()
assert row.question == BASE_QUESTION
assert row.deflected is False
assert row.sources == ", ".join(f"{SOURCE}/{p}" for p in SUGGESTED + RELATED)
# --------------------------------------------------------------------------
# 4. (d) the suggested-folder lines ride the prompt — echoed verbatim
# by the new mock trigger
# --------------------------------------------------------------------------
def test_folder_lines_echoed_verbatim(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
"""The distinctive question + ``repeat your folder map`` (the new
mock trigger — the prompt-injection-echo convention of
``<tuning>`` / ``<knowledge_base>``) ⇒ the answer ends with the
suggested-folder lines VERBATIM (joined with ``; ``): the guides'
folder line (its owner excluded from the entries) and the
name-hit doc's folder line — ``namekb/forgejo/: forgejo-nginx.md``
(the line prefix + its known sibling entry; the owner
``forgejo-home`` is excluded — its identity is already in its
``<document>`` block)."""
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, FOLDER_QUESTION)
_wait_settled(page)
frames = _frames(page)
assert _tool_frames(frames) == [], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
assert done["sources"] == [], done["sources"]
bubble = _last_brain(page).locator(".bubble")
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
# The folder lines VERBATIM — both lines, suggested-folder order
# (the guides' folder first — its first suggested doc leads; the
# name-hit doc's folder second), entries owner-excluded.
expect(bubble).to_contain_text(
"(folders: "
"namekb/deploy/: install-guide-b.md, install-guide-c.md, install-guide-d.md; "
"namekb/forgejo/: forgejo-nginx.md)",
)
# The task's pin: the name-hit doc's folder line prefix + one
# known sibling entry.
expect(bubble).to_contain_text(f"{SOURCE}/forgejo/: forgejo-nginx.md")
# Zero-read grounded turn — chips nothing (LOCKED A1).
_assert_chip_row(page, [])
# Durable record: grounded; suggested + related (LOCKED A3 — the
# folder question's rank-6+ order).
row = _last_query_log()
assert row.question == FOLDER_QUESTION
assert row.deflected is False
assert row.sources == ", ".join(
f"{SOURCE}/{p}" for p in SUGGESTED + RELATED_FOLDER
)
tests/e2e/test_no_reply_autoscroll.py
+13 -13
View File
@@ -472,13 +472,13 @@ def test_answer_content_intact(page: Page, app_url: str, seeded_kb: None) -> Non
page.set_default_timeout(30_000)
login(page, app_url, next="/")
# The long answer streams to completion with its sources ...
# The long answer streams to completion (phase 119, LOCKED A1: a
# zero-read turn chips nothing — the suggested kubernetes doc never
# chips; the retired phase-118 A4 union is gone) ...
submit(page, LONG_QUESTION)
wait_settled(page)
expect(page.locator(".msg.brain .bubble").last).to_contain_text(LONG_ANSWER_END)
expect(
page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
).to_have_count(1)
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# ... and a thinking turn completes with its block auto-collapsed
# (phase 17: open while streaming, closed from the first delta on).
@@ -489,25 +489,25 @@ def test_answer_content_intact(page: Page, app_url: str, seeded_kb: None) -> Non
expect(details.locator(".thinking-text")).to_contain_text(THINKING_FRAGMENT)
expect(page.locator(".msg.brain .bubble").last).to_contain_text(MOCK_ANSWER_MARKER)
# Persistence: four messages, the thinking text + sources stored raw.
# Persistence: four messages, the thinking text stored raw — and
# the thinking turn's sources EMPTY (phase 119, LOCKED A1: it read
# nothing, so the done frame — and the record — carry no sources;
# the retired phase-118 A4 union is gone).
raw = page.evaluate(f"() => localStorage.getItem('{STORAGE_KEY}')")
stored = json.loads(raw)
assert [m["who"] for m in stored["messages"]] == ["user", "brain", "user", "brain"]
assert LONG_ANSWER_END in stored["messages"][1]["text"]
assert THINKING_FRAGMENT in stored["messages"][3]["thinking"]
assert any(
s["path"] == "homelab/kubernetes.md" for s in stored["messages"][3]["sources"]
)
assert stored["messages"][3]["sources"] == []
# Restore: the long answer (with its chip) and the COLLAPSED thinking
# block come back intact.
# Restore: the long answer and the COLLAPSED thinking block come
# back intact — with ZERO citation chips (both turns read nothing;
# phase 119, LOCKED A1).
page.reload()
expect(page.locator(".msg.user .bubble")).to_have_count(2)
expect(page.locator(".msg.brain .bubble")).to_have_count(2)
expect(page.locator(".msg.brain .bubble").first).to_contain_text(LONG_ANSWER_END)
expect(
page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
).to_have_count(2)
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
restored = page.locator(".msg.brain").last.locator("details.thinking")
expect(restored).not_to_have_attribute("open")
expect(restored.locator(".thinking-text")).to_contain_text(THINKING_FRAGMENT)
tests/e2e/test_quadlet_jinja_import.py
+14 -6
View File
@@ -24,8 +24,10 @@ Test → story mapping (Playwright Mapping Rule):
4. ``test_jinja_retrievable_not_deflected`` — a question carrying the
``.j2`` sentinel FTS-matches the chunk (A8: LOW requires best cosine
below threshold **and** zero FTS hits) → honest-positive: the answer
bubble is not ``.is-deflected`` and a source chip names
``templates/deploy.j2``.
bubble is not ``.is-deflected`` — and (phase 119, LOCKED A1) the
zero-read turn chips NOTHING (the ``templates/deploy.j2`` retrieval
doc is suggested context, not a chip; the retired phase-118 A4
union is gone).
Phase 97 adaptation: the Sources table is the DRILL-DOWN TREE — the
rows live at their folder levels (``docs`` → ``homelab`` → ``quadlet``
@@ -218,10 +220,16 @@ def test_jinja_retrievable_not_deflected(
page.fill("#message-input", JINJA_QUESTION)
page.click("#send-btn")
# The done event appends source chips — waiting on the .j2 chip means
# the turn is finished and the retrieval doc reached the UI.
chip = page.locator(".msg.brain .source-chip", has_text="templates/deploy.j2")
expect(chip).to_have_count(1, timeout=30_000)
# Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation
# chips — the .j2 retrieval doc is suggested context, not a chip
# (the retired phase-118 A4 union is gone). The turn is finished
# (and the grounded, non-deflected state is reached) when the
# answer settles with the send button recovered.
expect(page.locator(".msg.brain .bubble").last).to_contain_text(
"Deterministic mock answer for E2E", timeout=30_000
)
expect(page.locator("#send-label")).to_have_text("Send")
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# A8: LOW requires best cosine < threshold AND zero FTS hits — the
# question's sentinel tokens FTS-match the .j2 chunk, so the gate is
tests/e2e/test_read_truncation_cap.py
+7 -4
View File
@@ -658,10 +658,11 @@ def test_truncated_read_frame_order_live_marker_and_llm_notice(
assert i_tool < i_result < i_delta, (i_tool, i_result, i_delta)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The read document is the turn's cited source (retrieval +
# agent-read, deduped — the grounded-turn record; phase 118: the
# target may be a suggested (summary-seeded) document — its seed
# status never changes the cap's wire contract).
# The read document is in the turn's cited sources (phase 119,
# LOCKED A1: done.sources is the READ docs only — this turn's one
# read; phase 118: the target may be a suggested (summary-seeded)
# document — its seed status never changes the cap's wire
# contract).
assert any(
s["path"] == CAPPED_REL and s["source"] == SOURCE for s in done["sources"]
), done["sources"]
@@ -797,6 +798,8 @@ def test_short_read_control_no_frame_no_marker(
assert _result_frames(frames) == [], _result_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The read document is the turn's cited source (phase 119 A1 — the
# READ docs only; the target may be a suggested/summary-seeded doc).
assert any(
s["path"] == SHORT_REL and s["source"] == SOURCE for s in done["sources"]
), done["sources"]
tests/e2e/test_retrieval_quality.py
+25 -10
View File
@@ -10,11 +10,15 @@ deterministic mock embeddings (same pattern as the earlier story suites).
The four tests map the story's acceptance criteria:
1. multi-format fixture import — hidden doc excluded, ``/api/docs`` counts
2. "How did I install gitlab?" — grounded (not deflected), gitlab chip,
``query_log`` row with the gitlab doc in ``sources``
2. "How did I install gitlab?" — grounded (not deflected), ZERO citation
chips (phase 119, LOCKED A1 — the turn reads nothing: the suggested
gitlab doc is context, not a chip), ``query_log`` row with the gitlab
doc in ``sources`` (LOCKED A3, untouched)
3. keyword-only question ("kafkabridge") beats the vector ranking — the
corroborated-lexical gate (A8 revised 2026-09-14) grounds it end to
end: weak cosine, but an FTS hit AND cosine >= lexical_support_floor
end: weak cosine, but an FTS hit AND cosine >= lexical_support_floor;
the FTS-matched doc tops the durable record (the chip pin is retired
with the phase-118 A4 union — a zero-read turn chips nothing)
4. "sourdough" — deflected bubble + ≥2 "Maybe try" chips
"""
from __future__ import annotations
@@ -156,9 +160,15 @@ def test_gitlab_question_is_grounded_with_gitlab_chip(
# Grounded: no deflected bubble at all.
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
# The gitlab document is cited (a chip carrying its path).
chip = page.locator(".msg.brain .source-chip", has_text="container_gitlab/gitlab.md")
expect(chip).to_have_count(1, timeout=30_000)
# Phase 119 (LOCKED A1): the turn read nothing, so ZERO citation
# chips — the suggested gitlab doc is seed context, not a citation
# (the retired phase-118 A4 union is gone); the ranking assertion
# lives in the durable record below (LOCKED A3, untouched). The
# button label is the settle sync: the client re-labels Send on the
# done frame, and the server writes the query_log row just before
# yielding it.
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
# Durable record: not deflected, and the gitlab doc is in sources.
with SessionLocal() as db:
@@ -186,10 +196,15 @@ def test_keyword_only_question_beats_vector_ranking(
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
# The FTS-matched doc is the TOP source chip (it beats the vector rank).
first_chip = page.locator(".msg.brain .source-chip").first
first_chip.wait_for(state="visible", timeout=30_000)
expect(first_chip).to_contain_text("static-dns.json")
# Phase 119 (LOCKED A1): zero-read turn → ZERO chips (the
# FTS-matched doc beats the vector rank in the DURABLE record below
# — LOCKED A3, untouched; the retired phase-118 A4 chip pin is
# gone). Wait for the settle via the Send label instead of a chip
# (the in-flight button is the enabled Stop control — only the
# label proves the done frame landed and the query_log row was
# written).
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
with SessionLocal() as db:
row = db.scalars(select(QueryLog)).one()
tests/e2e/test_search_tool.py
+15 -14
View File
@@ -50,9 +50,11 @@ Test → phase mapping:
(``Found …`` — the match reached the model), and the turn settles to
idle with no error banner.
2. ``test_search_adds_no_source_by_itself`` — context accounting
(locked A5): the search-only flow (no read) leaves
``done.sources`` / the source chips / ``query_log.sources`` at the
retrieval baseline — the search adds no source by itself.
(locked A5, phase-119 A1 shape): the search-only flow (no read)
leaves ``done.sources`` / the source chips EMPTY (chips cite read
docs only — the search adds no source by itself, and the retrieval
doc was never read) while ``query_log.sources`` keeps the retrieval
baseline (LOCKED A3, untouched).
3. ``test_search_tool_line_re_renders_after_reload`` — the persisted
record (phase 14 convention: the generic ``{name, argument}``
toolAcc) re-renders the search line through the same helper.
@@ -423,25 +425,24 @@ def test_search_adds_no_source_by_itself(
_wait_settled(page)
# The search really ran (its wire frame is present) — yet the
# search-only flow (no read) leaves done.sources at the RETRIEVAL
# baseline: the one fixture doc, nothing added by the search.
# search-only flow (no read) adds NO cited source: phase 119,
# LOCKED A1 — done.sources is the READ docs only, and nothing was
# read (the retrieval doc was never read, so it never chips — the
# retired phase-118 A4 suggested+read union is gone).
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "grep", "argument": SEARCH_PATTERN}
]
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(SEED_SOURCE, SEED_PATH)
]
assert done["sources"] == [], done["sources"]
# UI: exactly one source chip — the retrieval doc (the search
# renders no chip of its own).
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(1)
expect(chips.nth(0)).to_contain_text(SEED_SP)
# UI: ZERO source chips — the search renders no chip of its own, and
# the never-read retrieval doc chips nothing (phase 119 A1).
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# Durable record: the sources row is unchanged by the search alone.
# Durable record: the sources row keeps the retrieval baseline (the
# search adds nothing to it either — LOCKED A3, untouched).
row = _last_query_log()
assert row.deflected is False
assert row.sources == SEED_SP
tests/e2e/test_share_chat.py
+15 -17
View File
@@ -25,12 +25,14 @@ click is the idempotent share of the linked row):
cookies) opening ``/shared/<token>`` sees the full conversation
read-only through the same record shape: title = the auto-title,
user + brain bubbles (the same deterministic answer text the admin
session saw), the thinking block RESTORED COLLAPSED, the source
chips as PLAIN TEXT (zero ``a.source-chip`` — guests cannot open
documents, the documents API is admin-only), and ZERO interactive
controls anywhere (no composer, no Save/Share pills, no Tune/Retry,
no button chips); the nav's admin-only links stay hidden for a
guest;
session saw), the thinking block RESTORED COLLAPSED, ZERO source
chips of any kind — the shared turn read nothing, so (phase 119,
LOCKED A1) its ``done`` sources are empty and nothing re-renders as a
chip; where chips exist they are always PLAIN TEXT (zero
``a.source-chip`` — guests cannot open documents, the documents API
is admin-only) — and ZERO interactive controls anywhere (no
composer, no Save/Share pills, no Tune/Retry, no button chips); the
nav's admin-only links stay hidden for a guest;
* **Share from History + unshare** — the History row's Share column:
"Create link" → Copy + Unshare; Unshare is the inline two-step
(no native dialog); after Yes the cell returns to "Create link",
@@ -345,17 +347,13 @@ def test_anonymous_shared_view(
expect(think).to_have_count(1)
expect(think.first).not_to_have_attribute("open")
# Source chips are PLAIN TEXT: the on-topic turn carries its
# source chips (the suggested docs — the hybrid retrieval's
# summary-seed tier, phase 118), but every one as a <span>:
# zero <a.source-chip> anywhere (a guest cannot open documents;
# the documents API is admin-only).
assert (
anon.locator(".msg.brain .source-chip").count() >= 1
), "the grounded turn must carry its source chips"
expect(
anon.locator(".msg.brain .source-chip", has_text="kubernetes.md")
).to_have_count(1)
# Source chips: ZERO — the shared turn read nothing, so
# (phase 119, LOCKED A1) its done sources are empty and the
# shared page re-renders nothing (the retired phase-118 A4
# suggested+read union is gone); and where chips DO exist they
# are always plain text: zero <a.source-chip> anywhere (a
# guest cannot open documents; the documents API is admin-only).
expect(anon.locator(".msg.brain .source-chip")).to_have_count(0)
expect(anon.locator("a.source-chip")).to_have_count(0)
# ZERO interactive controls anywhere in the conversation: no
tests/e2e/test_source_chip_quality.py
+58 -42
View File
@@ -1,22 +1,27 @@
"""Phase 113 E2E (Playwright) — phase 118 re-targeted: the citation-surface
contract (LOCKED A4) as VISIBLE chip counts — the summary-seed contract
replaced the phase-112/113 usefulness bar: the chip row is the
SUGGESTED tier (top-5 distinct docs, NO floor) + agent reads (none on
these turns), the related row is rank 6+ (capped at
``related_max_docs`` = 2), and a deflected turn still cites nothing
(done.sources = [] — its weak hits are suggested for the durable record
but are never citation chips).
"""Phase 113 E2E (Playwright) — phase 118 re-targeted, phase 119
re-targeted again: the citation-surface contract (phase 119, LOCKED
A1) as VISIBLE chip counts — chips are the AGENT-READ docs only (the
retired phase-118 A4 suggested+read union is gone): neither turn on
this suite reads anything, so a grounded turn shows ZERO citation
chips (the suggested tier is seed context, not citations — the
phase-112/113 usefulness bar was retired with the full-text seeds, and
the suggested chips were retired with the phase-118 A4 union), the
related row is rank 6+ (capped at ``related_max_docs`` = 2, untouched),
and a deflected turn still cites nothing (done.sources = [] — its weak
hits are suggested for the durable record but are never citation
chips).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov
Acceptance (TODO L144–146, phase-118 shape): a grounded turn's chip row
is the suggested tier — the 5 docs the model was seeded with (its
"start here" set — the old "one chip per bar-clearing doc" is retired
with the full-text seeds; the LLM decides what the summaries earn);
a deflected turn shows zero citation chips (its weak hits, if any, live
in the de-emphasized related row — ``.related-doc`` links, NEVER
Acceptance (TODO L144–146, phase-119 shape): a grounded turn's chip row
is the READ set — empty on these no-read turns (the visible consequence
of the owner decision 2026-09-16: the suggested docs the model was
seeded with are context, not citations — the LLM decides what the
summaries earn, and a zero-read turn chips nothing); a deflected turn
shows zero citation chips (its weak hits, if any, live in the
de-emphasized related row — ``.related-doc`` links, NEVER
``.source-chip``; L2c: never render weak hits as answer citations).
The fixture KB's tier shapes are deterministic under the E2E mock's
@@ -25,14 +30,20 @@ here against the wire):
* **grounded question** — "What SSH aliases do I have?": best cosine
0.352 ≥ 0.30 → grounded; the suggested tier (top-5, NO floor) is
ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > kubernetes.md
> backups.md (fused rank — the lexical-only cosine-0.0 docs rank when
they rank: no floor filters them, LOCKED A3) and the done frame
carries EXACTLY those five cited refs; the rank-6+ remainder
(compose.container, static-dns.json) rides the related row, capped at
two. (The four OBSERVED live shapes are unit-pinned at plan level in
``tests/unit/test_source_chip_quality.py`` — the fixture KB
reproduces the tiered shape live, so no docstring caveat is needed.)
ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > backups.md
> kubernetes.md (fused rank — the lexical-only cosine-0.0 docs rank
when they rank: no floor filters them, LOCKED A3; phase 119, D1 —
``aliases`` is a stem sub-component of ``ssh_aliases.txt``, so the
doc name-hits, its name-hit row LEADS the lexical list, and the FTS
rank shift re-orders the fused scores — the pre-phase
kubernetes-before-backups order is retired) and the done frame
carries NO cited refs (phase 119 A1 — nothing was read: the
suggested tier seeds the prompt but never chips); the rank-6+
remainder (compose.container, static-dns.json) rides the related
row, capped at two. (The four OBSERVED live shapes are unit-pinned at
plan level in ``tests/unit/test_source_chip_quality.py`` — the
fixture KB reproduces the tiered shape live, so no docstring caveat
is needed.)
* **deflected question** — "How do I bake sourdough bread?": best cosine
0.109 < 0.30 and zero FTS hits → honest deflection → zero cited refs
(done.sources = []); the weak hits ARE suggested (no floor) for the
@@ -106,14 +117,15 @@ def _ask(page: Page, message: str) -> None:
def test_single_source_question_shows_exactly_one_citation_chip(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""The acceptance pin (phase 118, A4): a grounded question → the done
bubble carries EXACTLY the suggested tier as ``.source-chip``s (the
five docs the model was seeded with — top-5, NO floor: the old
usefulness-bar "one chip" is retired with the full-text seeds) and
the rank-6+ remainder renders only in the de-emphasized
``.related-docs`` row (``.related-doc`` links — never
``.source-chip``), labeled "Nearby docs, in case:". The durable
record keeps the FULL retrieval (LOCKED A3)."""
"""The acceptance pin (phase 119, A1): a grounded, NO-READ question →
the done bubble carries ZERO ``.source-chip``s (the chip row is the
agent-read docs only — nothing was read here: the five suggested
docs the model was seeded with are context, not citations — the
retired phase-118 A4 union is gone) and the rank-6+ remainder
renders only in the de-emphasized ``.related-docs`` row
(``.related-doc`` links — never ``.source-chip``), labeled "Nearby
docs, in case:" (phase-113 behavior, untouched). The durable record
keeps the FULL retrieval (LOCKED A3, untouched)."""
_reset_db(mock_llm)
page.set_default_timeout(30_000)
login(page, app_url, next="/") # phase 79: chat is require_user-gated
@@ -127,16 +139,14 @@ def test_single_source_question_shows_exactly_one_citation_chip(
# Grounded: no deflected bubble at all.
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
# The citation surface IS the suggested tier (LOCKED A4 — top-5,
# NO floor): five chips, in fused rank order — the model was
# seeded with exactly these five docs' summaries.
chip = page.locator(".msg.brain .source-chip")
expect(chip).to_have_count(5, timeout=30_000)
expect(chip.nth(0)).to_contain_text("ssh_aliases.txt")
expect(chip.nth(1)).to_contain_text("gitlab-compose.yaml")
expect(chip.nth(2)).to_contain_text("uptime_probe.py")
expect(chip.nth(3)).to_contain_text("kubernetes.md")
expect(chip.nth(4)).to_contain_text("backups.md")
# Phase 119 (LOCKED A1): the citation surface is the READ docs only
# — this turn read nothing, so ZERO chips; the five suggested docs
# (ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py >
# backups.md > kubernetes.md — the phase-119 D1 re-rank, pinned in
# the durable record below) seeded the prompt but never chip —
# the retired phase-118 A4 union is gone. The suggested tier itself
# is pinned below by the durable record (LOCKED A3, untouched).
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# The rank-6+ remainder rides the related row: a labeled,
# de-emphasized list — one .related-doc link per doc (rank order,
@@ -162,13 +172,19 @@ def test_single_source_question_shows_exactly_one_citation_chip(
row_log = db.scalars(select(QueryLog)).one()
assert row_log.question == SINGLE_SOURCE_QUESTION
assert row_log.deflected is False
# Suggested tier + related remainder, in the logged order.
# Suggested tier + related remainder, in the logged order. Phase
# 119 (D1): the stem sub-component ``aliases`` name-hits
# ``ssh_aliases.txt`` — its name-hit row leads the lexical list,
# the FTS rank shift re-orders the fused scores, and ``backups.md``
# (best eff 0.016277) now out-ranks ``kubernetes.md`` (0.016036);
# re-pinned against the post-phase walk (the record's CONTENT —
# suggested + related + read, deduped — is LOCKED A3, untouched).
assert row_log.sources == (
"docs/homelab/ssh/ssh_aliases.txt, "
"docs/homelab/container_gitlab/gitlab-compose.yaml, "
"docs/homelab/scripts/uptime_probe.py, "
"docs/homelab/kubernetes.md, "
"docs/homelab/backups.md, "
"docs/homelab/kubernetes.md, "
"docs/homelab/quadlet/compose.container, "
"docs/homelab/networking/static-dns.json"
), row_log.sources
tests/e2e/test_sources_midstream_bug.py
+17 -11
View File
@@ -89,10 +89,6 @@ HESITATE_QUESTION = (
#: pre-content pause (SLOW_PRETOKEN_TRIGGER) is running.
THINKING_TAIL = "nothing is invented"
#: Phase-10 viewer URL + phase-13 back=/ (byte-identical to the chip the
#: persistence suite pins — grounded-turn sources are unchanged by 20).
CHIP_HREF = "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F"
async def _import_fixtures(mock_port: int) -> ImportSummary:
kwargs: dict[str, Any] = {"_env_file": None, "llm_base_url": f"http://127.0.0.1:{mock_port}/v1"}
@@ -334,7 +330,10 @@ def test_full_answer_completes_after_rag_nav_midstream(
brain = msgs[1]
assert brain["text"] == FULL_LONG
assert brain["deflected"] is False
assert any(s["path"] == "homelab/kubernetes.md" for s in brain["sources"])
# Phase 119 (LOCKED A1): the turn read nothing, so its done sources
# — and the persisted record — are EMPTY (the retired phase-118 A4
# union is gone).
assert brain["sources"] == []
# The turn SETTLED — the phase-48 query_log row exists (a
# cancelled turn would leave no row at all).
@@ -408,7 +407,11 @@ def test_nav_switch_before_first_token_completes(
assert [m["who"] for m in msgs] == ["user", "brain"]
assert MOCK_ANSWER_MARKER in msgs[1]["text"]
assert msgs[1]["deflected"] is False
assert any(s["path"] == "homelab/kubernetes.md" for s in msgs[1]["sources"])
# Phase 119 (LOCKED A1): the turn read nothing, so its done sources
# — the stored record's sources list — are EMPTY (the retired
# phase-118 A4 suggested-citation is gone; the full retrieval
# stays in query_log, pinned below).
assert msgs[1]["sources"] == []
# The turn settled — one finalized row (a cancelled turn would
# leave no row at all).
@@ -495,22 +498,25 @@ def test_completed_turn_unaffected(
brain = before["messages"][1]
assert MOCK_ANSWER_MARKER in brain["text"]
assert brain["deflected"] is False
assert any(s["path"] == "homelab/kubernetes.md" for s in brain["sources"])
# Phase 119 (LOCKED A1): the turn read nothing, so its done sources
# — and the persisted record — are EMPTY (the retired phase-118 A4
# union is gone).
assert brain["sources"] == []
# A trip to Sources and back (the turn finished long ago — uiState is
# idle, so the pagehide save point must be a no-op).
page.goto(app_url + "/sources.html")
page.goto(app_url + "/")
# Full answer + source chip rendered; no error banner.
# Full answer rendered with ZERO citation chips (the zero-read turn
# persisted an empty sources list — phase 119, LOCKED A1; the
# retired phase-118 A4 chip pin is gone); no error banner.
expect(page.locator("#empty-state")).to_be_hidden()
expect(page.locator(".msg.user .bubble")).to_contain_text(QUESTION)
bubble = page.locator(".msg.brain .bubble")
expect(bubble).to_have_count(1)
expect(bubble.first).to_contain_text(MOCK_ANSWER_MARKER)
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
expect(chip).to_have_count(1)
expect(chip.first).to_have_attribute("href", CHIP_HREF)
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
_no_error_banner(page)
# Storage is byte-identical to the pre-navigation payload — the
tests/e2e/test_summary_seed_context.py
+44 -36
View File
@@ -59,19 +59,21 @@ TODO item — one Playwright file per story, A16):
The inverse of the retired phase-24 pin: NO document's tail
sentinel is in the echoed context — the full content of no
suggested doc ever reached the model (case b). The grounded turn
cites the 5 suggested docs (case e's chip surface, no read yet) and
cites NOTHING (phase 119, LOCKED A1 — a zero-read turn chips
nothing; the retired phase-118 A4 suggested+read union is gone) and
renders the de-emphasized related row (rank 6+); the durable
record carries suggested + related.
record carries suggested + related (118-A3 untouched).
2. ``test_read_suggested_doc_adds_full_text_and_cites`` — cases (c) +
(e): the scripted flow ``read``s the rank-1 suggested doc — the
read SUCCEEDS (phase 118: the seeds are summaries, not full text —
the retired top-2 seed-read refusal is gone) and the mock's
verbatim echo of the read result lands the tail sentinel in the
answer (case c: the full text now arrives through the ``read``
tool, not the seed); the citation chips = suggested + read,
deduped — the read doc is among the suggested, so the chip row is
exactly the 5 suggested docs, and the related row renders rank 6+
(case e); the durable record carries suggested + related + read.
tool, not the seed); the citation chips = the READ docs only
(phase 119, LOCKED A1 — exactly the one read doc; none of the
four other suggested docs chips), and the related row renders
rank 6+ (case e); the durable record carries suggested + related +
read (118-A3 untouched).
"""
from __future__ import annotations
@@ -503,14 +505,16 @@ def _last_query_log() -> QueryLog:
return rows[0]
def _assert_suggested_chips_and_related_row(page: Page, suggested_paths: list[str]) -> None:
"""The citation surface of a grounded turn (LOCKED A4): the chip row
is the suggested set (+ agent reads, deduped — asserted per test)
and the de-emphasized ``related-docs`` row carries the rank-6+
remainder (phase-113 UI, never a citation chip)."""
def _assert_read_chips_and_related_row(page: Page, read_paths: list[str]) -> None:
"""The citation surface of a grounded turn (phase 119, LOCKED A1 —
the retired phase-118 A4 union is gone): the chip row is the
AGENT-READ docs only (deduped, read order — a zero-read turn chips
NOTHING: the suggested docs are seed context, not citations), and
the de-emphasized ``related-docs`` row carries the rank-6+ remainder
(phase-113 UI, untouched — never a citation chip)."""
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(len(suggested_paths))
for path in suggested_paths:
expect(chips).to_have_count(len(read_paths))
for path in read_paths:
expect(
chips.filter(has_text=path), message=f"chip for {path}"
).to_have_count(1)
@@ -553,11 +557,12 @@ def test_summaries_seed_the_prompt_not_the_full_text(
assert _tool_frames(frames) == [], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The grounded turn cites the 5 suggested docs (LOCKED A4 — no read
# yet: the chip row IS the suggested set, in rank order).
assert [
(s["source"], s["path"]) for s in done["sources"]
] == [(SOURCE, p) for p in SUGGESTED_TAIL], done["sources"]
# Phase 119 (LOCKED A1): the grounded done frame cites the READ docs
# only — this turn read NOTHING (the summary-only fast path), so the
# citation surface is empty; the suggested set is seed context, not
# citations (the retired phase-118 A4 union is gone). The durable
# record below still carries suggested + related (118-A3 untouched).
assert done["sources"] == [], done["sources"]
bubble = _last_brain(page).locator(".bubble")
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
@@ -583,12 +588,13 @@ def test_summaries_seed_the_prompt_not_the_full_text(
expect(bubble).not_to_contain_text(sentinel)
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
# Case (e)'s chip surface (no read yet): chips = the 5 suggested
# docs; the related row renders rank 6+ (the de-emphasized row).
_assert_suggested_chips_and_related_row(page, SUGGESTED_TAIL)
# Case (e)'s chip surface (phase 119 A1 — no read yet): ZERO chips;
# the related row renders rank 6+ (the de-emphasized row, untouched).
_assert_read_chips_and_related_row(page, [])
# Durable record: grounded; suggested + related (LOCKED A3 — the log
# records retrieval, not citations).
# records retrieval, not citations; phase 119 A1 retires the chip
# union only, not the record).
row = _last_query_log()
assert row.question == TAIL_QUESTION
assert row.deflected is False
@@ -609,10 +615,10 @@ def test_read_suggested_doc_adds_full_text_and_cites(
retired top-2 seed-read refusal is gone) and the mock's verbatim
echo of the read result lands the document's tail sentinel in the
answer (case c: the full text reached the model through the
``read`` tool, not the seed). The citation chips = suggested +
read, deduped — the read doc is among the suggested, so the chip
row is exactly the 5 suggested docs; the related row renders rank
6+ (case e)."""
``read`` tool, not the seed). The citation chips = the READ docs
only (phase 119, LOCKED A1 — exactly the one read doc; none of the
other four suggested docs chips; the retired phase-118 A4 union is
gone); the related row renders rank 6+ (case e, untouched)."""
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
@@ -628,13 +634,15 @@ def test_read_suggested_doc_adds_full_text_and_cites(
], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# LOCKED A4: done.sources = suggested + read, deduped — the read doc
# is already among the suggested, so the citation surface is
# exactly the 5 suggested docs (in rank order).
# Phase 119 (LOCKED A1): done.sources = the READ docs only —
# exactly the one scripted read; the other four suggested docs
# never chip (the retired phase-118 A4 suggested+read union is
# gone).
assert [
(s["source"], s["path"]) for s in done["sources"]
] == [(SOURCE, p) for p in SUGGESTED_READ], done["sources"]
# done.related = the rank-6+ remainder (deduped against the cited).
] == [(SOURCE, READ_TARGET)], done["sources"]
# done.related = the rank-6+ remainder (deduped against the cited —
# the read doc is rank 1, not in the remainder).
assert [
(s["source"], s["path"]) for s in done["related"]
] == [(SOURCE, p) for p in RELATED_PATHS], done["related"]
@@ -658,13 +666,13 @@ def test_read_suggested_doc_adds_full_text_and_cites(
if path != READ_TARGET:
expect(bubble).not_to_contain_text(sentinel)
# Case (e): the UI chip row = suggested + read (deduped — the read
# doc is among the suggested, so chips = the 5 suggested docs), and
# the related row renders rank 6+.
_assert_suggested_chips_and_related_row(page, SUGGESTED_READ)
# Case (e): the UI chip row = the READ docs only (phase 119 A1 —
# exactly the one read doc), and the related row renders rank 6+.
_assert_read_chips_and_related_row(page, [READ_TARGET])
# Durable record: grounded; suggested + related + read (deduped,
# LOCKED A3).
# LOCKED A3 — phase 119 A1 retires the chip union only, not the
# record).
row = _last_query_log()
assert row.question == READ_QUESTION
assert row.deflected is False
tests/e2e/test_thinking_display.py
+14 -15
View File
@@ -55,9 +55,6 @@ DEFLECT_PHRASE = r"haven't done anything like that"
#: suite keys off it (mock_llm.compose_thinking).
THINKING_FRAGMENT = "Step 2: Check my notes"
STORAGE_KEY = "bor.chat.v1"
#: Phase-10 viewer URL + phase-13 back=/ (byte-identical to the chip the
#: persistence suite pins — grounded-turn sources are unchanged by 17).
CHIP_HREF = "/document.html?source=docs&path=homelab%2Fkubernetes.md&back=%2F"
async def _import_fixtures(mock_port: int) -> ImportSummary:
@@ -154,13 +151,13 @@ def test_thinking_block_streams_open_then_collapses(
expect(bubble).not_to_have_text("", timeout=30_000)
expect(details).not_to_have_attribute("open")
# Settled: full scratchpad, grounded mock answer, source chip(s),
# and the re-enabled send button.
# Settled: full scratchpad, grounded mock answer, ZERO citation
# chips (phase 119, LOCKED A1 — the zero-read turn chips nothing;
# the retired phase-118 A4 union is gone), and the re-enabled send
# button.
expect(details.locator(".thinking-text")).to_contain_text(THINKING_FRAGMENT)
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
expect(chip.first).to_be_visible()
expect(chip.first).to_have_attribute("href", CHIP_HREF)
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
expect(page.locator("#send-btn")).to_be_enabled()
expect(page.locator("#send-label")).to_have_text("Send")
@@ -232,10 +229,12 @@ def test_thinking_restored_after_reload(page: Page, app_url: str, seeded_kb: Non
expect(restored).not_to_have_attribute("open") # restored COLLAPSED
expect(restored.locator(".thinking-text")).to_have_text(captured)
# Answer bubble + source chip are intact (phase-14 restore path).
# Answer bubble is intact after the restore (phase-14 restore path)
# — with ZERO citation chips (phase 119, LOCKED A1: the zero-read
# turn persisted an empty sources list; the retired phase-118 A4
# union is gone).
expect(page.locator(".msg.brain .bubble").last).to_contain_text(MOCK_ANSWER_MARKER)
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
expect(chip.first).to_have_attribute("href", CHIP_HREF)
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# ---------------------------------------------------------------------------
@@ -252,10 +251,10 @@ def test_no_thinking_block_without_trigger(page: Page, app_url: str, seeded_kb:
# No trigger → no thinking events → no block anywhere on the page.
expect(page.locator("details.thinking")).to_have_count(0)
# The turn itself is complete and grounded, exactly as before phase 17.
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
expect(chip.first).to_be_visible()
expect(chip.first).to_have_attribute("href", CHIP_HREF)
# The turn itself is complete and grounded, exactly as before phase
# 17 — with ZERO citation chips (phase 119, LOCKED A1: the zero-read
# turn chips nothing; the retired phase-118 A4 union is gone).
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
# ---------------------------------------------------------------------------
tests/fixtures/namekb/deploy/install-guide-a.md
Vendored
+9
View File
@@ -0,0 +1,9 @@
# Install Forgejo Guide A
fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen
install forgejo with the deployment steps install forgejo with the deployment steps install forgejo with the deployment steps install forgejo with the deployment steps
alpha1 alpha2 alpha3 alpha4 alpha5 alpha6
NAMEKB-TAIL-c5d6
tests/fixtures/namekb/deploy/install-guide-b.md
Vendored
+9
View File
@@ -0,0 +1,9 @@
# Install Forgejo Guide B
fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen
install forgejo with the deployment steps install forgejo with the deployment steps install forgejo with the deployment steps
beta1 beta2 beta3 beta4 beta5 beta6
NAMEKB-TAIL-d7e8
tests/fixtures/namekb/deploy/install-guide-c.md
Vendored
+9
View File
@@ -0,0 +1,9 @@
# Install Forgejo Guide C
fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen
install forgejo with the deployment steps install forgejo with the deployment steps
gamma1 gamma2 gamma3 gamma4 gamma5 gamma6
NAMEKB-TAIL-f9a0
tests/fixtures/namekb/deploy/install-guide-d.md
Vendored
+9
View File
@@ -0,0 +1,9 @@
# Install Forgejo Guide D
fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen
install forgejo with the deployment steps
delta1 delta2 delta3 delta4 delta5 delta6
NAMEKB-TAIL-b1c2
tests/fixtures/namekb/forgejo/forgejo-home.md
Vendored
+9
View File
@@ -0,0 +1,9 @@
# Zeta Forge One
fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen
kilo1 kilo2 kilo3 kilo4 kilo5 kilo6
NAMEKB-TAIL-a1b2
tests/fixtures/namekb/forgejo/forgejo-nginx.md
Vendored
+9
View File
@@ -0,0 +1,9 @@
# Zeta Forge Two
fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen
lambda1 lambda2 lambda3 lambda4 lambda5 lambda6
NAMEKB-TAIL-b3c4
tests/fixtures/namekb/zz-folder-map.md
Vendored
+9
View File
@@ -0,0 +1,9 @@
# Zeta Misc Three
fixture preamble block one two three four five six seven eight nine ten eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen
micro1 micro2 micro3 micro4 micro5 micro6
NAMEKB-TAIL-e3f4
tests/fixtures/retrieval_battery.txt
Vendored
+17
View File
@@ -0,0 +1,17 @@
# Phase 119 retrieval battery (task 06, D6) — the 1,000-doc live replica.
#
# Gate (per the phase overview, post-phase measured numbers):
# 1-3. each name question's suggested top-5 contains its canonical doc:
# gitea -> deploy/Deployments/reeseapps/gitea/README.md (rank 4)
# forgejo-> 5/5 forgejo-tree docs, README #1
# gateway-> deploy/Deployments/reeseapps/gateway/README.md (rank 3)
# 4. the 2026-09-05 incident regression: a qwen3.8-27b quadlet stays in
# the suggested top-5 (the digit-bearing prefix rule, unchanged).
# 5. deflection sanity: the out-of-KB question's gate verdict + fts
# count match the pre-phase replica run (HIGH, fts_hits=5) — the
# name-hit change must not move it.
How do I deploy gitea?
How do I deploy forgejo?
How do I set up the gateway?
Do you have notes on the qwen 3.8 27b model?
What is the capital of Mongolia?
tests/integration/test_auth_api.py
+6 -2
View File
@@ -288,7 +288,10 @@ def test_token_user_surface_matrix(client: TestClient, db, seeded_kb: FakeRagLLM
assert content_type.startswith("text/event-stream")
assert frames[-1]["type"] == "done"
assert frames[-1]["deflected"] is False
assert frames[-1]["sources"][0]["path"] == "homelab/kubernetes.md"
# Phase 119 (A1): the citation surface is the agent's READ docs
# only — the canned turn reads nothing ⇒ no chips (the cited
# document's content pin below is the surface's other half).
assert frames[-1]["sources"] == []
# Suggestion chips + the cited document's content (the viewer).
assert client.get("/api/suggestions").status_code == 200
@@ -455,7 +458,8 @@ def test_admin_chat_still_streams(client: TestClient, db, seeded_kb: FakeRagLLM)
assert len(deltas) >= 2 # genuinely streamed
assert frames[-1]["type"] == "done"
assert frames[-1]["deflected"] is False
assert frames[-1]["sources"][0]["path"] == "homelab/kubernetes.md"
# Phase 119 (A1): read docs only — nothing was read ⇒ no chips.
assert frames[-1]["sources"] == []
# ---------- phase 81: rate-limited failed sign-ins (audit SEC-03) ----------
tests/integration/test_chat_api.py
+65 -58
View File
@@ -280,14 +280,14 @@ def test_chat_streams_deltas_then_done_with_sources(client, db, seeded_kb: FakeR
assert frames[-1]["type"] == "done" # done is the final event
assert done[0]["deflected"] is False
assert done[0]["suggestions"] == []
sources = done[0]["sources"]
assert sources, "done must carry the cited sources"
# Phase 118 (A4): the citation surface is the suggested tier (top-5,
# no floor) + the agent's reads (none on this turn) — deduped.
assert len(sources) == get_settings().suggested_docs
assert sources[0]["path"] == "homelab/kubernetes.md"
assert sources[0]["source"] == "docs"
assert sources[0]["title"] == "Kubernetes Homelab Cluster"
# Phase 119 (LOCKED A1 — the phase-118 A4 union retired): the
# citation surface is the agent's READ docs only — this turn's
# canned LLM never emits a tool call, so nothing was read and the
# grounded done frame chips nothing (an accepted, owner-directed
# consequence — the answer prose names the doc it used). The
# retrieval stays durably recorded (118-A3; pinned in the
# query_log test below this one).
assert done[0]["sources"] == []
# The LLM received the locked HIGH prompt — the ``<documents>`` block
# seeds the document's stored SUMMARY (phase 118, LOCKED A6: summary
@@ -334,13 +334,13 @@ def test_chat_streams_thinking_before_deltas(client, db, seeded_kb: FakeRagLLM)
assert "".join(f["text"] for f in thinking) == thinker.thinking
assert "".join(d["text"] for d in deltas) == thinker.answer
# Done still last; sources unchanged by the thinking extension.
# Done still last; the citation surface is unchanged by the thinking
# extension — phase 119 (A1): read docs only, nothing read ⇒ none.
done = frames[-1]
assert done["type"] == "done"
assert done["deflected"] is False
assert done["suggestions"] == []
assert done["sources"][0]["path"] == "homelab/kubernetes.md"
assert done["sources"][0]["source"] == "docs"
assert done["sources"] == []
assert not any(f.get("type") == "error" for f in frames)
@@ -469,12 +469,13 @@ def test_off_topic_question_deflects_honestly(client, db, seeded_kb: FakeRagLLM)
def test_done_frame_carries_related_tier_on_grounded_turn(
client, db, seeded_kb: FakeRagLLM
) -> None:
"""Phase 118 (LOCKED A3/A4): a grounded turn's done frame carries the
suggested tier in ``sources`` (top-5, no floor) and the related
tier — the ranked docs from rank 6+ after the suggested set, capped
at ``related_max_docs`` (2) — in ``related``, disjoint from the
citation surface. The durable record keeps the FULL retrieval
(suggested + related + read, LOCKED A3)."""
"""Phase 118 (A3) × phase 119 (A1): a grounded turn's done frame
carries the related tier — the ranked docs from rank 6+ after the
suggested set, capped at ``related_max_docs`` (2) — in ``related``,
disjoint from the read-only ``sources`` (nothing was read on this
turn ⇒ no chips; the never-read suggested tier is not on the wire).
The durable record keeps the FULL retrieval (suggested + related +
read, 118-A3 stands)."""
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: seeded_kb
try:
_, _, frames = _stream_chat(client, QUESTION)
@@ -483,11 +484,11 @@ def test_done_frame_carries_related_tier_on_grounded_turn(
done = frames[-1]
assert done["deflected"] is False
# A4: the citation surface is the suggested tier (5, no read on this
# turn) — ranks 1–5 for the Kubernetes question.
# Phase 119 (A1): the citation surface is the agent's READ docs
# only — no read on this turn ⇒ no chips (the retired A4 union
# would have carried the suggested ranks 1–5 here).
sources = [(s["source"], s["path"]) for s in done["sources"]]
assert len(sources) == get_settings().suggested_docs
assert sources[0] == ("docs", "homelab/kubernetes.md")
assert sources == []
related = done["related"]
# Rank 6–7 for the Kubernetes question (after the top-5 suggested
# set), capped at related_max_docs.
@@ -546,12 +547,13 @@ def test_deflected_done_frame_carries_weak_hits_in_related(
def test_related_doc_read_by_agent_is_cited_not_related(
client, db, seeded_kb: FakeRagLLM
) -> None:
"""Phase 118 × phase 37: an agent-read doc is a citation by definition
(LOCKED A4) — when the agent ``read``s a rank-6+ doc (the related
tier, "nearby docs"), it joins done.sources (deduped, last — it was
not suggested, so the read appends it) and is EXCLUDED from
done.related (a "nearby doc" that was actually used must not read as
nearby)."""
"""Phase 119 (A1) × phase 37: an agent-read doc is a chip by
definition — when the agent ``read``s a rank-6+ doc (the related
tier, "nearby docs"), it is the done frame's ONLY chip (the read
docs ARE the citation surface since phase 119; the never-read
suggested tier is not on the wire) and is EXCLUDED from
done.related (a "nearby doc" that was actually used must not read
as nearby — unchanged intent, the dedupe keyed on read docs)."""
scripted = FakeRagLLM(
tool_script=[
[
@@ -572,9 +574,9 @@ def test_related_doc_read_by_agent_is_cited_not_related(
done = frames[-1]
assert done["deflected"] is False
sources = [(s["source"], s["path"]) for s in done["sources"]]
# A4: suggested (5) + the read doc (last).
assert len(sources) == get_settings().suggested_docs + 1
assert sources[-1] == ("docs", "homelab/quadlet/cache.volume") # read ⇒ cited
# A1: read docs only — the read related doc is the ONLY chip (the
# retired A4 union would have carried 6 here).
assert sources == [("docs", "homelab/quadlet/cache.volume")] # read ⇒ chip
related = [(s["source"], s["path"]) for s in done["related"]]
assert ("docs", "homelab/quadlet/cache.volume") not in related
assert set(sources).isdisjoint(set(related))
@@ -605,8 +607,9 @@ def test_keyword_question_grounded_by_lexical_hit_despite_weak_cosine(
assert done["type"] == "done"
assert done["deflected"] is False # weak cosine, but a lexical hit
assert done["suggestions"] == []
sources = done["sources"]
assert sources and sources[0]["path"] == "homelab/networking/static-dns.json"
# Phase 119 (A1): read docs only — nothing was read ⇒ no chips;
# the lexical hit stays in the durable record (asserted below).
assert done["sources"] == []
(system, _user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1]
assert "<relevance>HIGH</relevance>" in system["content"] # grounded prompt
@@ -1085,12 +1088,13 @@ def test_truncated_read_streams_tool_result_frame_after_tool_frame(
assert body.startswith(f"Document docs/big-read.md:\ndate: 2024-06-15\n{content[:cap]}")
assert TRUNCATION_MARKER in body
assert READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) in body
# The truncated read is still a SUCCESSFUL call — cited in done.
# The truncated read is still a SUCCESSFUL call — it is the frame's
# only chip (phase 119, A1: read docs only; the suggested kubernetes
# doc was never read, so it is not on the wire).
done = frames[-1]
assert done["type"] == "done" and done["deflected"] is False
assert ("docs", "big-read.md") in [(s["source"], s["path"]) for s in done["sources"]]
assert ("docs", "homelab/kubernetes.md") in [
(s["source"], s["path"]) for s in done["sources"]
assert [(s["source"], s["path"]) for s in done["sources"]] == [
("docs", "big-read.md")
]
@@ -1159,11 +1163,12 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
) -> None:
"""(a) Grounded turn with tool calls: the event sequence is
``thinking?/tool/tool/delta…/done``; ``done.sources`` and the
``query_log`` row include the read document (deduped, order
preserved); the per-turn log line carries ``tool_calls=2``.
Phase 45: the agent loop keeps offering the tools for the whole
turn — the round cap (not per-tool budgets) is the bound."""
``thinking?/tool/tool/delta…/done``; ``done.sources`` is the read
document (phase 119, A1 — the read docs are the citation surface)
and the ``query_log`` row includes it (deduped, order preserved);
the per-turn log line carries ``tool_calls=2``. Phase 45: the
agent loop keeps offering the tools for the whole turn — the round
cap (not per-tool budgets) is the bound."""
scripted = FakeRagLLM(
tool_script=[
[
@@ -1212,12 +1217,12 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
done = frames[-1]
assert done["deflected"] is False
# done.sources = the retrieval docs + the read doc, deduped, order kept.
# Phase 119 (A1): done.sources = the agent's READ docs only — the
# read doc is the frame's only chip (the suggested retrieval docs
# are not on the wire; they stay in the durable record below).
sources = [(s["source"], s["path"]) for s in done["sources"]]
assert sources[-1] == ("docs", "homelab/backups.md") # the read doc is cited
assert ("docs", "homelab/kubernetes.md") in sources # …after the retrieval docs
assert len(sources) == len(set(sources)) # deduped by (source, path)
assert done["sources"][-1]["title"] == "Backup Strategy"
assert sources == [("docs", "homelab/backups.md")] # the read doc is cited
assert done["sources"][0]["title"] == "Backup Strategy"
# Phase 45: the tools stay offered on every request — the round cap
# (not spent budgets) bounds the loop, and the model answered while
@@ -1250,7 +1255,8 @@ def test_grounded_turn_streams_grep_tool_frames(
the raw pattern is the frame's ``argument`` (the UI renders the
"searching for" line from it). A non-string pattern — a model error
the backend refuses — yields ``argument: null``. A grep adds no
source: ``done.sources`` stays the retrieval docs (locked A5)."""
source (locked A5): with no read on the turn, ``done.sources`` is
empty (phase 119, A1)."""
scripted = FakeRagLLM(
tool_script=[
[
@@ -1292,9 +1298,8 @@ def test_grounded_turn_streams_grep_tool_frames(
assert [f for f in frames if f["type"] == "delta"]
done = frames[-1]
assert done["type"] == "done" and done["deflected"] is False
paths = [s["path"] for s in done["sources"]]
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
assert "homelab/backups.md" not in paths # a grep adds no source
# A grep adds no source (A5) and nothing was read (A1) ⇒ no chips.
assert done["sources"] == []
def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
@@ -1306,7 +1311,8 @@ def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
and frame emission is execution-independent: a rejected call (an
unknown ``read`` path) still streams its frame with the model's
argument as-is. The rejected read adds no source (``done.sources``
stays the retrieval docs), and rejected calls count nothing
carries no chip — the read failed and nothing else was read,
phase 119 A1), and rejected calls count nothing
(``tool_calls=1`` — only the executed scoped ``ls``)."""
# The scoped ``ls`` source-name check reads the registry — insert a
# row resolving to ``docs`` (the fixture's source name) and delete
@@ -1350,12 +1356,12 @@ def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
# passed — frame emission is execution-independent.
assert read_frame["argument"] == "docs/homelab/nope.md"
# The rejected read adds no source — done.sources stays retrieval.
# The rejected read adds no source — and the read-only surface is
# empty on this turn (nothing was read; the refused read cites
# nothing, phase 119 A1).
done = frames[-1]
assert done["type"] == "done" and done["deflected"] is False
paths = [s["path"] for s in done["sources"]]
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
assert "homelab/nope.md" not in paths # the refused read cites nothing
assert done["sources"] == []
# The rejected call counts nothing — only the executed scoped ls.
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
@@ -1462,9 +1468,10 @@ def test_zero_max_rounds_reproduce_pre_phase_single_request(
done = frames[-1]
assert done["type"] == "done"
assert done["deflected"] is False
paths = [s["path"] for s in done["sources"]]
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
assert "homelab/backups.md" not in paths # nothing was read
# Phase 119 (A1): nothing was read ⇒ no chips (the kill-switch
# turn's citation surface is empty; the retrieval stays in the
# durable record below).
assert done["sources"] == []
# Exactly one request, and it carried no ``tools`` key at all — the
# scripted tool calls were never even offered a chance.
tests/integration/test_kb_overview_api.py
+40 -13
View File
@@ -25,15 +25,17 @@ from pathlib import Path
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import select, text
from sqlalchemy import text
from test_chat_api import FakeRagLLM, _stream_chat, _token_vec
from app.api import chat as chat_api
from app.config import get_settings
from app.main import app as fastapi_app
from app.models import Document, KbOverview
from app.rag.agent import suggested_folder_lines
from app.rag.importer import import_sources
from app.rag.prompts import build_deflect_prompt, build_high_prompt
from app.rag.retriever import retrieve, weak_hit_titles
from app.rag.retriever import retrieve, select_suggested, weak_hit_titles
from tests.conftest import ADMIN_PASSWORD
FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "docs"
@@ -90,15 +92,27 @@ def _seed_overview(db) -> None:
db.commit()
def _cited_docs(db, frames: list[dict]) -> list[Document]:
"""The documents the done event cited, in citation order — the same
list ``plan_turn`` passed to the prompt builder."""
docs = []
for s in frames[-1]["sources"]:
doc = db.scalar(select(Document).where(Document.path == s["path"]))
assert doc is not None, f"done source {s['path']!r} missing from the KB"
docs.append(doc)
return docs
def _suggested_docs(db) -> list[Document]:
"""The documents the grounded turn's HIGH prompt seeded, in the same
order ``plan_turn`` walked them — the SAME deterministic suggested
walk over the retrieval. Phase 119 (A1) moved the citation surface
to the agent's READ docs, so the seeded tier can no longer be read
off the done frame (the canned LLM here reads nothing ⇒
``done.sources`` is empty) — reconstructing from the retrieval is
the faithful source now."""
chunks = retrieve(db, QUESTION, _token_vec(QUESTION))
return list(select_suggested(chunks, n=get_settings().suggested_docs))
def _suggested_folder_lines(db) -> list[str]:
"""The suggested-folder context lines the endpoint computed (phase
119, D3, LOCKED A4) — the SAME deterministic suggested walk
``plan_turn`` performs internally (one extra walk, no shared
state), over the real seeded catalog."""
chunks = retrieve(db, QUESTION, _token_vec(QUESTION))
return suggested_folder_lines(
db, select_suggested(chunks, n=get_settings().suggested_docs)
)
def _turn_log_lines(caplog: pytest.LogCaptureFixture) -> list[str]:
@@ -124,7 +138,14 @@ def test_no_row_high_prompt_byte_identical_to_pre_phase(
assert frames[-1]["deflected"] is False
(system, user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1]
assert user["content"] == QUESTION
expected = build_high_prompt(_cited_docs(db, frames), notes=[], kb_overview=None)
# Phase 119 (D3): the endpoint's suggested-folder lines ride the
# HIGH prompt — reconstructed the same deterministic way.
expected = build_high_prompt(
_suggested_docs(db),
notes=[],
kb_overview=None,
folder_lines=_suggested_folder_lines(db),
)
assert system["content"] == expected
assert "<knowledge_base>" not in system["content"]
@@ -179,8 +200,14 @@ def test_row_high_prompt_carries_kb_section_exactly(
assert frames[-1]["deflected"] is False
(system, _user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1]
# Phase 119 (D3): the suggested-folder lines ride the HIGH prompt
# alongside the <knowledge_base> section — reconstructed the same
# deterministic way.
expected = build_high_prompt(
_cited_docs(db, frames), notes=[], kb_overview=OVERVIEW
_suggested_docs(db),
notes=[],
kb_overview=OVERVIEW,
folder_lines=_suggested_folder_lines(db),
)
assert system["content"] == expected
tests/integration/test_name_hit_lexical.py
+54 -15
View File
@@ -6,10 +6,12 @@ mapping with fake rows; this suite covers the SQL side on real
Postgres: the document-projection scan, the LATERAL representative-
chunk fetch (the ``is_summary`` chunk wins, chunk 0 otherwise, and a
chunk-less name match is EXCLUDED — the ``c.id IS NOT NULL`` guard),
the (count, length, catalog) ranking, the name-hits-lead-the-lexical-
list union with the FTS rows (chunk-id dedup), and the full
``retrieve()`` → ``select_documents()`` path putting the versioned-
name document into the seeded top-N.
the (matched-token count, catalog) ranking (phase 119, LOCKED A2 —
the two-class component rule: digit-bearing prefix, digitless exact,
titles never matched), the name-hits-lead-the-lexical-list union with
the FTS rows (chunk-id dedup), and the full ``retrieve()`` →
``select_documents()`` path putting the versioned-name document into
the seeded top-N.
Requires: ``podman compose up -d db``.
"""
@@ -137,19 +139,29 @@ def kb(db) -> Iterator[None]:
def test_name_hit_chunks_real_sql(kb, db) -> None:
"""Real Postgres: the projection scan finds exactly the qwen3.8
quadlet (the qwen3.6 sibling and the chunk-less name match are
excluded), and the LATERAL fetch hands back the SUMMARY chunk as
the representative (position −1, is_summary)."""
"""Real Postgres: the projection scan finds the qwen3.8 quadlet
(the ``qwen38`` stem prefix — the incident's original case) AND
the ``notes/llama.cpp.md`` doc (the question names "llama.cpp" —
the dotted token ``llamacpp`` exact-matches the file stem, the
phase-119 two-class rule). The qwen3.6 sibling (``qwen38`` is not
a prefix of ``qwen36…``) and the chunk-less name match are
excluded; the LATERAL fetch hands back the SUMMARY chunk as the
representative for the quadlet (position −1, is_summary) and chunk
0 for the single-chunk notes doc."""
out = _name_hit_chunks(db, INCIDENT_QUESTION)
assert [rc.document.path for rc in out] == [
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container"
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container",
"notes/llama.cpp.md", # the question names it — stem exact (A2)
]
rc = out[0]
assert rc.position == -1 # the summary chunk wins the LATERAL order
assert rc.is_summary is True
assert rc.fts_hit is True # the lexical signal — the A8 gate answers
assert rc.cosine == 0.0 # no vector rank on the name-hit row
rc_notes = out[1]
assert rc_notes.position == 0 # chunk 0 (no summary chunk)
assert rc_notes.is_summary is False
assert all(rc.name_hit is True for rc in out) # phase 119 — D2 bonus input
assert "qwen3.8-empty.container" not in [r.document.path for r in out] # chunk-less guard
@@ -163,13 +175,19 @@ def test_lexical_candidates_name_hit_leads_real_sql(kb, db) -> None:
assert paths[0] == (
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container"
)
assert paths[1] == "notes/llama.cpp.md" # the second name hit (stem exact)
# The FTS pollution is still present (the incident's shape) — but
# behind the name hit, no longer ahead of it.
# behind the name hits, no longer ahead of them.
assert (
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.6-27b-juggernaut-vulkan.container"
in paths
)
assert all(rc.fts_hit is True for rc in out)
# Phase 119: the two name-hit representative rows are flagged, the
# plain FTS rows are not.
assert out[0].name_hit is True
assert out[1].name_hit is True
assert all(not rc.name_hit for rc in out[2:])
def test_retrieve_selects_name_hit_doc_into_top_n(kb, db) -> None:
@@ -177,7 +195,12 @@ def test_retrieve_selects_name_hit_doc_into_top_n(kb, db) -> None:
fused) → ``select_documents`` puts the qwen3.8 quadlet in the
seeded top-N — the incident's seed miss (the two overview docs
only) is fixed. The question vector is parallel to the q38 chunk
embeddings (cosine 1.0), orthogonal to q36 (0.0)."""
embeddings (cosine 1.0), orthogonal to q36 (0.0).
Phase 119: the ``name_hit`` flag survives the fusion — the two
name-hit representative chunks are double hits (vector ∪ lexical),
and the double-hit merge ORs the flag into the surviving row;
the plain q36 vector+FTS row stays False."""
question_vec = _vec(1)
chunks = retrieve(db, INCIDENT_QUESTION, question_vec)
docs = select_documents(chunks, n=2)
@@ -185,16 +208,31 @@ def test_retrieve_selects_name_hit_doc_into_top_n(kb, db) -> None:
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container",
"notes/llama.cpp.md",
]
by_path: dict[str, list[bool]] = {}
for rc in chunks:
by_path.setdefault(rc.document.path, []).append(rc.name_hit)
# The q38 doc's summary chunk is the double-hit name hit (flagged);
# its plain chunk 0 (vector + FTS) is not — the doc has a flagged row.
assert any(by_path[
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container"
])
assert any(by_path["notes/llama.cpp.md"]) # its only chunk is the name hit
assert not any(by_path[
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.6-27b-juggernaut-vulkan.container"
]) # plain vector+FTS — never a name hit
def test_name_hit_limit_real_sql(db) -> None:
"""Twelve identical (1, 6) name hits — the LATERAL fetch (and the
output) carries exactly ``NAME_HIT_LIMIT`` winners, catalog order."""
"""Twelve tied name hits (one matched token each — the ``qwen38``
stem PREFIX; the token must lead the stem, the old mid-stem
containment no longer matches) — the LATERAL fetch (and the
output) carries exactly ``NAME_HIT_LIMIT`` winners, catalog
order."""
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
for i in range(12):
doc = _doc(
db, "S", f"quadlets/m{i:02d}-qwen38.container", f"m{i:02d}-qwen38", "llama cpp qwen38\n"
db, "S", f"quadlets/qwen38-m{i:02d}.container", f"qwen38-m{i:02d}", "llama cpp qwen38\n"
)
db.flush()
c = _chunk(db, doc, 0, f"llama cpp qwen38 doc {i}")
@@ -203,5 +241,6 @@ def test_name_hit_limit_real_sql(db) -> None:
out = _name_hit_chunks(db, "what are the llama.cpp arguments for qwen 3.8")
assert len(out) == NAME_HIT_LIMIT
assert [rc.document.path for rc in out] == [
f"quadlets/m{i:02d}-qwen38.container" for i in range(NAME_HIT_LIMIT)
f"quadlets/qwen38-m{i:02d}.container" for i in range(NAME_HIT_LIMIT)
]
assert all(rc.name_hit is True for rc in out)
tests/integration/test_recency_boost.py
+36 -15
View File
@@ -12,13 +12,21 @@ margins below are exact floats, not flaky measurements.
**Measured geometry (recorded per task step 4/5):**
* Owner scenario — A (``backups/retention.md``, created 2020-01-01,
the exact answer, cosine 1.0) lands at vector rank 1 + FTS rank 1
(fused 0.03278689); B (``backups/retention-draft.md``, created
the exact answer, cosine 1.0) lands at vector rank 1 + lexical rank
3 (fused 0.03226646); B (``backups/retention-draft.md``, created
yesterday, the "under review, no decision yet" draft, cosine
0.707107) lands at vector rank 10 + FTS rank 3 (fused 0.03015873 —
a solid FTS hit at rank 3, as the task describes). Pre-boost fused
margin **A−B = 0.00262816** (asserted ≥ 3× the zero-age boost =
0.002100 at the default → ratio 1.25, the "comfortable margin").
0.707107) lands at vector rank 10 + lexical rank 1 (fused
0.03067915). Pre-boost fused margin **A−B = 1/63 − 1/70 =
0.00158730** — re-pinned by phase 119 (task 01): the two-class
name-hit rule (LOCKED A2) makes the digitless question token
"retention" exact-match the stem sub-component of all three
``retention*`` docs, so the lexical list now LEADS with those name
hits in catalog order (draft, review, retention) and B takes the
rank-1 lexical slot (pre-119 it was a plain FTS rank-1 / rank-3
hit). The flip-safety bar the margin is pinned against — margin >
the zero-age boost, so a zero-age B's full boost cannot close it
(A's own boost being ≈ 0 at its 2020 age) — still holds, at 2.27×
the default.
* Twin near-tie — C (``twin/c-older.md``, 2019) and D
(``twin/d-newer.md``, yesterday) with IDENTICAL chunk text and
near-identical vectors (cosine 1.0 vs 0.9999 — a literal identical
@@ -29,12 +37,16 @@ margins below are exact floats, not flaky measurements.
scale.
* The DEFAULT was tuned from the design starting point (0.001) down to
**0.0007** (task step 5: "tune the DEFAULTS … until old-correct wins
comfortably"): on the k=60 scale the owner scenario's margin is
comfortably"): on the k=60 scale the owner scenario's margin was
0.00262816 < 3×0.001, and a 0.001 zero-age boost (+0.000997 for a
yesterday doc) would have FLIPPED the pinned scenario. 0.0007 keeps
the flip margin comfortable (0.000698 > 0.00052882, lead
+0.000169) while staying 1.25× under the 3×-boost margin bar. The
owner re-tunes live via ``BOR_RECENCY_BOOST``.
+0.000169). Phase 119 (task 01) re-pinned the owner margin to
0.00158730 (above): still above the zero-age boost (2.27×), so the
worst-case flip — a zero-age B against A's ≈ 0 boost — keeps A
first; the twin-flip figures (gap 0.00052882, boost 0.000698) are
untouched (the twin KB has no name hits). The owner re-tunes live
via ``BOR_RECENCY_BOOST``.
Requires: ``podman compose up -d db``.
"""
@@ -239,11 +251,12 @@ def test_owner_scenario_old_correct_beats_new_similar(
owner_kb, db, monkeypatch: pytest.MonkeyPatch
) -> None:
"""THE owner scenario, pinned at the DEFAULTS: the older doc that
answers ranks above the newer similar one — AND the pre-boost fused
margin is ≥ 3× the zero-age boost (the "comfortable margin"; the
measured 0.00262816 vs the 0.0021 bar is recorded in the module
docstring). Re-pinned with the boost OFF: relevance alone already
ordered them (no regression — the boost is not what makes A win)."""
answers ranks above the newer similar one — AND the pre-boost
fused margin exceeds the zero-age boost (the flip-safety bar; the
phase-119-re-pinned measured 0.00158730 vs the 0.0007 default is
recorded in the module docstring). Re-pinned with the boost OFF:
relevance alone already ordered them (no regression — the boost is
not what makes A win)."""
chunks = retrieve(db, QUESTION, QUESTION_VEC)
assert select_documents(chunks, n=2)[0].path == "backups/retention.md"
@@ -254,7 +267,15 @@ def test_owner_scenario_old_correct_beats_new_similar(
fused = fuse(vector, lexical, s.rrf_k)
by_path = {rc.document.path: rc.score for rc in fused}
margin = by_path["backups/retention.md"] - by_path["backups/retention-draft.md"]
assert margin >= 3 * s.recency_boost
# Phase 119 (task 01) re-pinned geometry: the three ``retention*``
# docs are name hits (the digitless token "retention" exact-matches
# the stem sub-component), ranked in catalog order (draft, review,
# retention) — so A sits at vector rank 1 + lexical rank 3 and B
# at vector rank 10 + lexical rank 1: the margin is exactly
# 1/63 − 1/70. The flip-safety bar: a zero-age B's full boost
# cannot close it (A's own boost is ≈ 0 at its 2020 age).
assert margin == pytest.approx(1 / 63 - 1 / 70)
assert margin > s.recency_boost # 2.27× at the default
# The kill switch: A still first (relevance alone), and the
# weight-0 scores are the pre-phase fused scores byte-identical.
tests/unit/test_agent.py
+223
View File
@@ -1091,6 +1091,229 @@ def test_group_folder_listing_caps_files_at_fifty_keeps_the_total() -> None:
assert total50 == 50 and len(files50) == 50
# ---------- phase 119 (D3): the suggested-folder context lines ----------
def _patch_catalog(
monkeypatch: pytest.MonkeyPatch, catalog: dict[str, list[tuple[str, str, str]]]
) -> None:
"""Monkeypatch the ``ls`` fetchers (house style) with one source
catalog of ``(path, title, date)`` rows per source name."""
monkeypatch.setattr(
agent, "_source_document_rows", lambda db, source: catalog.get(source, [])
)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
def test_suggested_folder_lines_source_root_doc_lists_the_top_level(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A suggested doc at the source root (``""`` prefix) renders the
``<source>/: …`` line with the source's top level — subfolders with
recursive counts, the suggested doc itself excluded."""
_patch_catalog(
monkeypatch,
{
"Homelab": [
("a/b/one.md", "One", "2024-06-15"),
("a/b/two.md", "Two", "2024-06-15"),
("a/c.md", "C", "2024-06-15"),
("z.md", "Z", "2024-06-15"),
]
},
)
assert agent.suggested_folder_lines(
cast("Session", object()), [_doc("Homelab", "z.md")]
) == ["Homelab/: a/ (3 docs)"]
def test_suggested_folder_lines_nested_doc_lists_the_parent_folder(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A nested suggested doc renders the PARENT folder's line — the
direct children only: subfolder names (source-relative, recursive
counts) + the files by relative filename, the suggested doc itself
out of the entries."""
_patch_catalog(
monkeypatch,
{
"Homelab": [
("a/b/one.md", "One", "2024-06-15"),
("a/b/two.md", "Two", "2024-06-15"),
("a/c/d.md", "D", "2024-06-15"),
("a/e.md", "E", "2024-06-15"),
]
},
)
# The parent of a/b/one.md is a/b: one.md excluded, two.md stays.
assert agent.suggested_folder_lines(
cast("Session", object()), [_doc("Homelab", "a/b/one.md")]
) == ["Homelab/a/b/: two.md"]
# The parent of a/e.md is a: the subfolders (recursive counts — a/b
# carries two, a/c one) first, then the files; e.md excluded, so no
# file entries remain.
assert agent.suggested_folder_lines(
cast("Session", object()), [_doc("Homelab", "a/e.md")]
) == ["Homelab/a/: a/b/ (2 docs), a/c/ (1 doc)"]
def test_suggested_folder_lines_excludes_only_the_owning_doc(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The doc that OWNS the line (the first suggested doc in that
folder) is the only one excluded — a folder whose sole entry was
the suggested doc renders its header alone (the empty-level
precedent)."""
_patch_catalog(
monkeypatch,
{
"Homelab": [("a/only.md", "Only", "2024-06-15")]
},
)
assert agent.suggested_folder_lines(
cast("Session", object()), [_doc("Homelab", "a/only.md")]
) == ["Homelab/a/:"]
def test_suggested_folder_lines_dedupes_by_source_and_prefix(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Dedupe: two suggested docs in the same folder ⇒ ONE line (the
first doc's position, the first doc excluded, the second listed);
the key is ``(source, prefix)`` — the same prefix under a DIFFERENT
source is a distinct line; suggested-doc order is the line order."""
_patch_catalog(
monkeypatch,
{
"Homelab": [
("a/b/one.md", "One", "2024-06-15"),
("a/b/two.md", "Two", "2024-06-15"),
("a/c.md", "C", "2024-06-15"),
("a/d/e.md", "E", "2024-06-15"),
("z.md", "Z", "2024-06-15"),
],
"Other": [("a/b/x.md", "X", "2024-06-15")],
},
)
docs = [
_doc("Homelab", "z.md"),
_doc("Homelab", "a/b/one.md"),
_doc("Homelab", "a/b/two.md"), # deduped (a/b already seen)
_doc("Homelab", "a/c.md"),
_doc("Homelab", "a/d/e.md"),
_doc("Other", "a/b/x.md"), # same prefix, different source
]
assert agent.suggested_folder_lines(cast("Session", object()), docs) == [
"Homelab/: a/ (4 docs)",
"Homelab/a/b/: two.md",
"Homelab/a/: a/b/ (2 docs), a/d/ (1 doc)",
"Homelab/a/d/:",
"Other/a/b/:",
]
def test_suggested_folder_lines_max_lines_caps_the_list(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""At most *max_lines* lines — the walk stops, suggested-doc order
kept (the first distinct folders win)."""
_patch_catalog(
monkeypatch,
{
"Homelab": [
("f1/x.md", "1", "2024-06-15"),
("f2/x.md", "2", "2024-06-15"),
("f3/x.md", "3", "2024-06-15"),
]
},
)
docs = [_doc("Homelab", p) for p in ("f1/x.md", "f2/x.md", "f3/x.md")]
lines = agent.suggested_folder_lines(cast("Session", object()), docs, max_lines=2)
assert lines == ["Homelab/f1/:", "Homelab/f2/:"]
assert agent.suggested_folder_lines(
cast("Session", object()), docs, max_lines=1
) == ["Homelab/f1/:"]
def test_suggested_folder_lines_max_entries_cap_and_remainder(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""At most *max_entries* entries, then `` +N more`` with the EXACT
remainder (the true pre-cap folder total — the owning doc leaves
it); the ``(1 doc)`` singular form; no `` +N more`` when every
entry fits."""
rows = [(f"big/s{i}/x.md", f"S{i}", "2024-06-15") for i in range(4)] + [
(f"big/f{i:02d}.md", f"F{i}", "2024-06-15") for i in range(10)
]
_patch_catalog(monkeypatch, {"S": rows})
db = cast("Session", object())
# 4 subfolders (one doc each) + 10 files − the owning f00.md = 13
# entries; 8 shown (the 4 subfolders + f01..f04) + 5 more. The
# subfolder entries are source-relative (``big/s0/``) — exactly as
# the model's own ls output names them.
assert agent.suggested_folder_lines(db, [_doc("S", "big/f00.md")]) == [
"S/big/: big/s0/ (1 doc), big/s1/ (1 doc), big/s2/ (1 doc), big/s3/ (1 doc), "
"f01.md, f02.md, f03.md, f04.md +5 more"
]
# A wide enough cap shows every entry — no remainder marker.
assert agent.suggested_folder_lines(db, [_doc("S", "big/f00.md")], max_entries=20) == [
"S/big/: big/s0/ (1 doc), big/s1/ (1 doc), big/s2/ (1 doc), big/s3/ (1 doc), "
"f01.md, f02.md, f03.md, f04.md, f05.md, f06.md, f07.md, f08.md, f09.md"
]
# The remainder is exact at a tighter cap too (3 subfolders shown —
# files only follow ALL subfolders, the ls order — 10 left).
assert agent.suggested_folder_lines(db, [_doc("S", "big/f00.md")], max_entries=3) == [
"S/big/: big/s0/ (1 doc), big/s1/ (1 doc), big/s2/ (1 doc) +10 more"
]
def test_suggested_folder_lines_entry_order_matches_the_ls_listing(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The line's entry order IS the ``ls`` folder-level rendering order
(pinned against :func:`group_folder_listing` itself): the direct
subfolders first (path order), then the files (path order) — a
mixed folder where the file names and the subfolder names interleave
in path order still lists all subfolders before any file."""
rows = [
("a/a.md", "A", "2024-06-15"),
("a/b.md", "B", "2024-06-15"),
("a/c/d.md", "D", "2024-06-15"),
]
_patch_catalog(monkeypatch, {"Homelab": rows})
db = cast("Session", object())
suggested = _doc("Homelab", "a/b.md")
# The expected entry order, derived from group_folder_listing itself
# (subfolders → files, minus the owning doc) — the line must read
# the same as the model's own ls output.
sub, files, total = agent.group_folder_listing("Homelab", "a", rows, {})
assert total == 2
expected_entries = [
f"{g}/ ({c} {'doc' if c == 1 else 'docs'})" for g, c, _s in sub
] + [path.rsplit("/", 1)[-1] for _src, path, _t, _d in files if path != suggested.path]
# Subfolder source-relative (``a/c/``), files by relative filename.
assert expected_entries == ["a/c/ (1 doc)", "a.md"]
expected_line = f"Homelab/a/: {', '.join(expected_entries)}"
assert agent.suggested_folder_lines(db, [suggested]) == [expected_line]
def test_suggested_folder_lines_empty_suggested_yields_no_lines(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""No suggested docs → no lines (the caller then builds the
byte-identical phase-118 prompt) — and no fetcher call at all."""
calls: list[str] = []
def _rows(db: Any, source: str) -> list[tuple[str, str, str]]:
calls.append(source)
return []
monkeypatch.setattr(agent, "_source_document_rows", _rows)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
assert agent.suggested_folder_lines(cast("Session", object()), []) == []
assert calls == []
# ---------- the pinned drill-down templates (byte-for-byte) ----------
tests/unit/test_chat_cancel.py
+14 -1
View File
@@ -183,6 +183,13 @@ class _FakeSession:
return KbOverview(id=1, content="")
return None
def execute(self, *args: Any, **kwargs: Any) -> list[Any]:
# Phase 119 (D3): the endpoint's suggested-folder lines run the
# ls catalog fetchers on this session — the fake catalog is
# empty (header-only lines), keeping the prompt builds
# deterministic here.
return []
@pytest.fixture()
def env(monkeypatch: pytest.MonkeyPatch) -> Iterator[_FakeSession]:
@@ -411,10 +418,16 @@ def test_completed_turn_still_emits_done_and_writes_query_log(
frames = _frames(chunks)
assert [f["type"] for f in frames] == ["delta", "delta", "delta", "done"]
assert frames[-1]["deflected"] is False
assert frames[-1]["sources"][0]["title"] == "Kubernetes Homelab Cluster"
# Phase 119 (LOCKED A1): the citation surface is the agent's READ
# docs only — this turn's model stream never emits a tool call, so
# nothing was read and the grounded done frame chips nothing (the
# phase-118 "suggested + read" union is retired). The retrieval
# stays durably recorded (118-A3 stands).
assert frames[-1]["sources"] == []
(row,) = env.added
assert isinstance(row, QueryLog)
assert row.question == "How is my Kubernetes cluster set up?"
assert "kubernetes-homelab-cluster.md" in row.sources # durable record kept
assert env.commits == 1
# The per-turn line still goes out; no cancel line for a settled turn.
assert any("question=" in r.getMessage() for r in caplog.records)
tests/unit/test_chat_gate.py
+237 -14
View File
@@ -11,7 +11,7 @@ from __future__ import annotations
import hashlib
import json
import uuid
from collections.abc import Iterator
from collections.abc import Iterator, Sequence
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any
@@ -22,9 +22,10 @@ from app.api import chat as chat_api
from app.config import Settings
from app.main import app as fastapi_app
from app.models import Document, KbOverview, QueryLog
from app.rag import agent as rag_agent
from app.rag.agent import AGENT_TOOLS
from app.rag.llm import StreamPiece
from app.rag.prompts import build_deflect_prompt
from app.rag.llm import StreamPiece, ToolCallPiece
from app.rag.prompts import SUGGEST_INTRO, build_deflect_prompt
from app.rag.retriever import RetrievedChunk, weak_hit_titles
from app.rag.suggestions import MAX_SUGGESTIONS, derive_suggestions
from tests.conftest import ADMIN_PASSWORD
@@ -443,6 +444,51 @@ def test_plan_turn_high_seeds_top5_suggested_related_is_rank6plus() -> None:
assert f"FULL_CONTENT_SENTINEL_{i}" not in plan.system_prompt
def test_plan_turn_folder_lines_ride_the_high_prompt() -> None:
"""Phase 119 (D3, LOCKED A4): plan_turn passes *folder_lines* through
to the HIGH prompt — after ``SUGGEST_INTRO``, before the first
``<document>`` block (each on its own line)."""
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_CONTENT")
lines = ["Homelab/: kubernetes-homelab-cluster.md (2 docs)"]
plan = chat_api.plan_turn(
[_chunk(doc, 0.90)], _settings(threshold=0.30), folder_lines=lines
)
assert plan.deflected is False
prompt = plan.system_prompt
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n{lines[0]}\n\n"
def test_plan_turn_folder_lines_default_keeps_phase_118_high_prompt() -> None:
"""Omitted *folder_lines* (the default ``()``) ⇒ the HIGH prompt is
the phase-118 shape, byte-identical."""
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_CONTENT")
plan = chat_api.plan_turn([_chunk(doc, 0.90)], _settings(threshold=0.30))
assert plan.deflected is False
prompt = plan.system_prompt
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n\n"
def test_plan_turn_folder_lines_ignored_on_the_low_branch() -> None:
"""The LOW (deflected) branch IGNORES *folder_lines* — the
deflected prompt stays byte-identical (LOCKED A4)."""
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_CONTENT")
low_with = chat_api.plan_turn(
[_chunk(doc, 0.05, cosine=0.05)],
_settings(threshold=0.30),
folder_lines=["Homelab/: x.md"],
)
low_without = chat_api.plan_turn(
[_chunk(doc, 0.05, cosine=0.05)], _settings(threshold=0.30)
)
assert low_with.deflected is True
assert low_with.system_prompt == low_without.system_prompt
assert "Homelab/: x.md" not in low_with.system_prompt
def test_plan_turn_high_single_strong_doc_yields_one_suggested() -> None:
"""The suggested cap is a CEILING, not a quota: one doc ⇒ one
suggested doc, an empty related tier (nothing beyond rank 1)."""
@@ -458,10 +504,11 @@ def test_plan_turn_high_single_strong_doc_yields_one_suggested() -> None:
def test_plan_turn_high_suggests_strong_and_weak_no_floor() -> None:
"""The recurring incident under phase 118 (A3): the weak 2nd doc no
longer loses a citation slot to a bar — the floor never filters, so
BOTH docs are suggested (rank order) and ride the citation surface
(A4); the HIGH prompt seeds both summaries (the A5 fallback carries
the short fixture content whole)."""
longer loses a seeding slot to a bar — the floor never filters, so
BOTH docs are suggested (rank order) and ride the durable record
(118-A3 — the phase-119 A1 citation surface is the read docs only);
the HIGH prompt seeds both summaries (the A5 fallback carries the
short fixture content whole)."""
strong = _doc("Kubernetes Homelab Cluster", "STRONG_DOC_CONTENT")
weak = _doc("Backup Strategy", "WEAK_DOC_CONTENT")
chunks = [_chunk(strong, 0.90, cosine=0.80), _chunk(weak, 0.80, cosine=0.20)]
@@ -845,15 +892,21 @@ def test_suggestions_empty_input_yields_fallback_only() -> None:
class _CannedLLM:
"""Records the messages it is given; streams a canned answer.
Never emits tool calls, so a grounded turn through the phase-37 agent
loop ends after the single (tools-offered) request; *seen_tools*
records each request's ``tools`` value for the phase-37 wiring pins.
Without *read_paths* it never emits tool calls, so a grounded turn
through the phase-37 agent loop ends after the single (tools-offered)
request. With *read_paths*, the request whose conversation carries
*i* tool results (i < len(read_paths)) emits ``read(read_paths[i])``
(phase 119: the read-only done-sources pins drive multi-read turns),
and once the list is exhausted the request streams the answer.
*seen_tools* records each request's ``tools`` value for the phase-37
wiring pins.
"""
def __init__(self, answer: str = ANSWER) -> None:
def __init__(self, answer: str = ANSWER, read_paths: Sequence[str] = ()) -> None:
self.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
self.embed_batches = 0
self.answer = answer
self.read_paths = list(read_paths)
self.seen: list[list[dict[str, str]]] = []
self.seen_tools: list[list[dict[str, Any]] | None] = []
@@ -868,6 +921,15 @@ class _CannedLLM:
):
self.seen.append(messages)
self.seen_tools.append(tools)
if tools is not None and self.read_paths:
tool_results = sum(1 for m in messages if m.get("role") == "tool")
if tool_results < len(self.read_paths):
yield ToolCallPiece(
id=f"call_{tool_results + 1}",
name="read",
arguments={"path": self.read_paths[tool_results]},
)
return
for i in range(0, len(self.answer), 12):
yield StreamPiece("content", self.answer[i : i + 12])
@@ -913,6 +975,15 @@ class _FakeSession:
return KbOverview(id=1, content=self.kb_overview)
return None
def execute(self, *args: Any, **kwargs: Any) -> list[Any]:
"""Phase 119 (D3): the endpoint's suggested-folder lines run the
``ls`` catalog fetchers (``_source_document_rows`` /
``_source_folder_summaries``) on this session — the fake catalog
is empty, so each suggested doc's line is its header alone
(``Homelab/:``), keeping the prompt builds deterministic in
these gate tests."""
return []
@pytest.fixture(autouse=True)
def _admin_signed_in(client: TestClient) -> None:
@@ -972,6 +1043,16 @@ def _fake_retriever(chunks: list[RetrievedChunk]) -> Any:
return retrieve
def _find_docs(docs: dict[tuple[str, str], Document]) -> Any:
"""A fake ``find_document`` resolving *docs* by ``(source, path)``
(the agent module's hook — the fake sessions here have no catalog)."""
def find_document(_db: Any, source: str, path: str) -> Document | None:
return docs.get((source, path))
return find_document
def test_endpoint_just_below_threshold_deflects(
client: TestClient,
gate_env: tuple[_FakeSession, _CannedLLM],
@@ -1073,11 +1154,18 @@ def test_endpoint_deflected_turn_never_offers_tools(
assert "You may extend your context with three tools" not in system["content"]
def test_endpoint_score_at_threshold_answers(
def test_endpoint_score_at_threshold_answers_zero_read_grounded_turn_chips_nothing(
client: TestClient,
gate_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 119 (LOCKED A1 — the phase-118 "always chips" pin
re-targeted): a grounded turn on which the agent reads NOTHING (the
summary-only fast path — the canned LLM never emits a tool call)
chips nothing: ``done.sources`` is the READ DOCS only, so the
never-read suggested doc appears NOWHERE in it (the explicit contrast
against the retired A4 union). The durable record keeps the retrieval
(118-A3 stands)."""
session, llm = gate_env
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT")
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(doc, 0.30)]))
@@ -1088,17 +1176,152 @@ def test_endpoint_score_at_threshold_answers(
assert done["type"] == "done"
assert done["deflected"] is False
assert done["suggestions"] == []
assert done["sources"] and done["sources"][0]["title"] == "Kubernetes Homelab Cluster"
# A1: read docs only — nothing was read ⇒ no chips. The suggested
# doc (rank 1) appears NOWHERE in done.sources (the retired A4 union
# would have carried it).
assert done["sources"] == []
(system, _user) = llm.seen[0][0], llm.seen[0][1]
assert "<relevance>HIGH</relevance>" in system["content"]
assert "DEFLECT_MODE" not in system["content"]
assert "TALOS_DOC_SENT" in system["content"]
# The durable record still records the retrieval (118-A3 stands).
(row,) = session.added
assert isinstance(row, QueryLog)
assert row.deflected is False
assert row.top_score == pytest.approx(0.30)
assert "kubernetes-homelab-cluster.md" in row.sources
def test_endpoint_done_sources_are_the_read_docs_in_read_order(
client: TestClient,
gate_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 119 (LOCKED A1): the done frame's ``sources`` is exactly
the agent's READ documents — deduped by (source, path), READ order
(not the suggested tier's rank order). Suggested docs that were
never read appear NOWHERE in the frame (the contrast pin against
the retired A4 union); the durable record keeps the full retrieval
(118-A3 stands)."""
session, _llm = gate_env
a = _doc("Alpha", "ALPHA_DOC_CONTENT")
b = _doc("Beta", "BETA_DOC_CONTENT")
c = _doc("Gamma", "GAMMA_DOC_CONTENT")
chunks = [_chunk(a, 0.90), _chunk(b, 0.80), _chunk(c, 0.70)]
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever(chunks))
llm = _CannedLLM(read_paths=[f"Homelab/{b.path}", f"Homelab/{a.path}"])
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: llm
monkeypatch.setattr(
rag_agent,
"find_document",
_find_docs({(a.source, a.path): a, (b.source, b.path): b}),
)
frames = _ask(client, "Which of these documents do I have?")
done = frames[-1]
assert done["deflected"] is False
# Both reads streamed as tool frames (the agent loop ran).
reads = [f for f in frames if f["type"] == "tool"]
assert [f["argument"] for f in reads] == [
f"Homelab/{b.path}",
f"Homelab/{a.path}",
]
# A1: read docs only, read order — B first (it was read first), A
# second; the never-read suggested doc C is NOWHERE in the frame, and
# the suggested tier's rank order (A, B, C) is NOT the chip order.
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(b.source, b.path),
(a.source, a.path),
]
assert not any(s["path"] == c.path for s in done["sources"])
assert len(done["sources"]) == len({(s["source"], s["path"]) for s in done["sources"]})
assert done["related"] == [] # nothing beyond rank 3 for 3 docs
# The durable record keeps the full retrieval (118-A3 stands).
(row,) = session.added
assert isinstance(row, QueryLog)
for doc in (a, b, c):
assert f"Homelab/{doc.path}" in row.sources
def test_endpoint_read_related_doc_is_cited_not_related(
client: TestClient,
gate_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 119 (A1) × phase 118 (A3): a related-tier doc (rank 6+) that
the agent ``read``s is a CHIP (in ``done.sources``) and is EXCLUDED
from ``done.related`` — a "nearby doc" that was actually used must
not read as nearby (unchanged intent, keyed on read docs since
phase 119). The suggested docs, never read, are absent from the
frame; the other related doc stays in the tier."""
session, _llm = gate_env
docs = [_doc(f"Doc {i}", f"CONTENT_{i}") for i in range(7)]
chunks = [
_chunk(d, 0.9 - 0.1 * i, cosine=0.8 - 0.05 * i) for i, d in enumerate(docs)
]
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever(chunks))
read_doc = docs[5] # rank 6 — the related tier (rank 7 = docs[6])
llm = _CannedLLM(read_paths=[f"Homelab/{read_doc.path}"])
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: llm
monkeypatch.setattr(
rag_agent,
"find_document",
_find_docs({(read_doc.source, read_doc.path): read_doc}),
)
frames = _ask(client, "What does the sixth ranked document say?")
done = frames[-1]
assert done["deflected"] is False
# A1: read docs only — the read related doc is the ONLY chip (the
# never-read suggested docs are nowhere in the frame).
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(read_doc.source, read_doc.path)
]
# The read related doc is NOT "nearby"; the other related doc is.
related = [(s["source"], s["path"]) for s in done["related"]]
assert related == [(docs[6].source, docs[6].path)]
assert not any((d.source, d.path) in related for d in docs[:5])
# The durable record keeps the full retrieval (118-A3 stands).
(row,) = session.added
assert isinstance(row, QueryLog)
assert row.deflected is False
for d in docs:
assert f"Homelab/{d.path}" in row.sources
def test_endpoint_suggested_folder_lines_reach_the_high_prompt(
client: TestClient,
gate_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 119 (D3, LOCKED A4): the endpoint computes the
suggested-folder lines BEFORE plan_turn (the same deterministic
suggested walk, over the short-lived step session) and they ride
the grounded HIGH prompt — after ``SUGGEST_INTRO``, before the
first ``<document>`` block."""
_session, llm = gate_env
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT")
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(doc, 0.30)]))
line = "Homelab/: kubernetes-homelab-cluster.md (2 docs)"
monkeypatch.setattr(
chat_api,
"suggested_folder_lines",
lambda db, suggested: [line],
)
frames = _ask(client, "How is my Kubernetes cluster set up?")
assert frames[-1]["deflected"] is False
(system, _user) = llm.seen[0][0], llm.seen[0][1]
i_open = system["content"].index("<documents>")
i_block = system["content"].index("<document ")
assert (
system["content"][i_open:i_block]
== f"<documents>\n{SUGGEST_INTRO}\n{line}\n\n"
)
# ---------- endpoint: KB overview row (phase 31) ----------
tests/unit/test_config.py
+31
View File
@@ -41,6 +41,10 @@ def test_defaults_match_locked_decisions(monkeypatch: pytest.MonkeyPatch) -> Non
# fine-line-tuned value, task 07) with a 365-day decay timescale.
assert s.recency_boost == 0.0007
assert s.recency_half_life_days == 365
# Phase 119, D2 (LOCKED A3): the bounded name-hit bonus on the
# selection-time document score is ON by default (0.005 — the
# owner-tunable starting point; ``0`` is the kill switch).
assert s.name_hit_bonus == 0.005
# Owner instruction 2026-08-22: answers may run up to 32 768 tokens.
assert s.max_output_tokens == 32_768
# Phase 17: the model's thinking streams by default (kill-switch off).
@@ -391,6 +395,33 @@ def test_recency_boost_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> None
_settings()
def test_name_hit_bonus_default_and_env_override(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 119, D2 (LOCKED A3): the bounded name-hit bonus on the
SELECTION-time document score is ON by default (0.005 — the
owner-tunable starting point, not a calibrated constant) and
env-tunable so the owner re-tunes live (the phase-106 recency-boost
precedent). ``0`` is legal — the byte-identical kill switch."""
monkeypatch.delenv("BOR_NAME_HIT_BONUS", raising=False)
s = _settings()
assert s.name_hit_bonus == 0.005
monkeypatch.setenv("BOR_NAME_HIT_BONUS", "0")
assert _settings().name_hit_bonus == 0.0
monkeypatch.setenv("BOR_NAME_HIT_BONUS", "0.01")
assert _settings().name_hit_bonus == 0.01
def test_name_hit_bonus_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> None:
"""``0`` is the byte-identical kill switch (the pre-phase selection
order) — a NEGATIVE bonus would demote name-hit documents (the exact
opposite of D2), so the validator fails loudly at startup naming the
field (the ``agent_max_rounds`` pattern, phase 119)."""
monkeypatch.setenv("BOR_NAME_HIT_BONUS", "-0.001")
with pytest.raises(ValidationError, match="name_hit_bonus"):
_settings()
def test_recency_half_life_rejects_non_positive(
monkeypatch: pytest.MonkeyPatch,
) -> None:
tests/unit/test_embed_question_length.py
+7
View File
@@ -173,6 +173,13 @@ class _FakeSession:
return None
return None
def execute(self, *args: Any, **kwargs: Any) -> list[Any]:
# Phase 119 (D3): the endpoint's suggested-folder lines run the
# ls catalog fetchers on this session — the fake catalog is
# empty (header-only lines), keeping the prompt builds
# deterministic here.
return []
def _doc(title: str, content: str) -> Document:
return Document(
tests/unit/test_prompt_lock.py
+35 -15
View File
@@ -148,16 +148,27 @@ def test_low_prompt_build_byte_identical_to_pre_phase() -> None:
assert "SUGGEST_INTRO" not in prompt and "<documents>" not in prompt
# ---------- SUGGEST_INTRO (phase 118, task 03 — the start-here framing) ----------
# ---------- SUGGEST_INTRO (phase 118 framing, re-cut for phase 119, LOCKED A5) ----------
#: Phase-118 anchors for ``SUGGEST_INTRO`` — the ``<documents>``
#: section's intro line (the owner's "start here if these summaries seem
#: right to you" framing, TODO L3). The E2E mock's ``_document_block``
#: parser is regex-based over the block markup (which stays byte-stable
#: around the intro), so this constant is a prompt-copy lock, pinned the
#: way ``TOOLS_SECTION`` is: sha256 + prefix + total length.
SUGGEST_INTRO_SHA256 = "7b14d2dedc6ebc4e440d32dd1edb979a7461c94034b9541c9f37b9042f394d3a"
SUGGEST_INTRO_LEN = 323
#: Anchors for ``SUGGEST_INTRO`` — the ``<documents>`` section's intro
#: line (the owner's "start here if these summaries seem right to you"
#: framing, TODO L3, phase 118). RE-CUT for phase 119 (task 04, LOCKED
#: A5, owner directive 2026-09-16): the final sentence was replaced
#: with the cite-discipline sentence (cite only the documents read — or
#: the suggested document answered from without reading — never a
#: document neither read nor used), closing the live confabulation in
#: which the model cited a file it never read; the phase-118 "cite the
#: document(s) you used, by path" sentence is retired. Only that
#: sentence moved — the start-here framing, the ``read`` pointer, and
#: the full-text clause survive byte-identical, so the prefix anchor is
#: the same as pre-phase-119. The E2E mock's ``_document_block`` parser
#: is regex-based over the block markup (which stays byte-stable around
#: the intro), so this constant is a prompt-copy lock, pinned the way
#: ``TOOLS_SECTION`` is: sha256 + prefix + total length. ``PERSONA``,
#: ``TOOLS_SECTION``, and the LOW body anchors above stay byte-identical
#: — this change touches no other constant.
SUGGEST_INTRO_SHA256 = "c6ffbb19975cbf184910a9a8e8fdea101c16f5d3eecb872fb07afeabd2763302"
SUGGEST_INTRO_LEN = 460
SUGGEST_INTRO_PREFIX = (
"The blocks below are the summaries of the top-ranked documents for "
"your question — start here if one seems right to you: "
@@ -165,14 +176,23 @@ SUGGEST_INTRO_PREFIX = (
def test_suggest_intro_byte_locked() -> None:
"""The start-here framing is LOCKED copy (phase 118): sha256 + exact
prefix + total length; the three contracts it must carry (summaries
are the starting points; ``read`` adds the full text, which is NOT
in the prompt until read; cite by path) are pinned as substrings."""
"""The start-here framing is LOCKED copy (phase 118; the final
sentence re-cut by phase 119, LOCKED A5): sha256 + exact prefix +
total length; the four contracts it must carry (summaries are the
starting points; ``read`` adds the full text, which is NOT in the
prompt until read; the phase-119 cite discipline — cite only what
was read, or the suggested document answered from without reading,
never a document neither read nor used) are pinned as substrings."""
assert len(SUGGEST_INTRO) == SUGGEST_INTRO_LEN
assert _sha256(SUGGEST_INTRO) == SUGGEST_INTRO_SHA256
assert SUGGEST_INTRO.startswith(SUGGEST_INTRO_PREFIX)
assert "call `read`" in SUGGEST_INTRO
assert "call `read" in SUGGEST_INTRO
assert "combined `source/path`" in SUGGEST_INTRO
assert "its full text is not in the prompt until you read it" in SUGGEST_INTRO
assert "Cite the document(s) you used, by path." in SUGGEST_INTRO
assert (
"Cite only the document(s) you read — or, if you answered from a "
"suggested summary without reading it, cite that suggested "
"document — never a document you neither read nor used."
) in SUGGEST_INTRO
# The retired phase-118 final sentence is gone from the constant.
assert "Cite the document(s) you used, by path." not in SUGGEST_INTRO
tests/unit/test_prompts.py
+156
View File
@@ -24,6 +24,13 @@ document is refused — while the ``ls``/``grep`` clauses and the
discipline rules stay byte-identical): the teaching refusals in
:mod:`app.rag.agent` re-state the same contract; the ``<tools>``
marker keying (HIGH only) is unchanged.
And the phase-119 cite discipline (task 04, LOCKED A5): the intro's
final sentence ("cite the document(s) you used, by path", phase 118)
is REPLACED — the HIGH prompt carries the discipline sentence exactly
once, inside ``<documents>`` after the intro's start-here framing;
the LOW prompt and ``PERSONA`` / ``TOOLS_SECTION`` stay byte-identical
(the full sha pins live in :mod:`tests.unit.test_prompt_lock`).
"""
from __future__ import annotations
@@ -447,6 +454,155 @@ def test_documents_section_without_blocks_has_no_intro() -> None:
)
# ---------- phase 119 (D3, LOCKED A4): the suggested-folder lines ----------
def test_high_prompt_folder_lines_after_intro_before_first_block() -> None:
"""The folder lines ride the ``<documents>`` section immediately
AFTER the ``SUGGEST_INTRO`` line — each on its own line — then a
blank line, then the first ``<document>`` block. Plain lines: no
new markup/tag anywhere (the E2E mock keys off the ``<documents>``
marker and the LAST block's tail)."""
doc = _doc(
"deploy/Deployments/reeseapps/gitea/app/gitea-web.env.j2",
"FULL_CONTENT_SENTINEL_119",
"Gitea Web Env",
summary="Gitea web env file.",
)
lines = [
"Homelab/deploy/Deployments/reeseapps/gitea/: app/ (5 docs), README.md",
"Homelab/deploy/Deployments/reeseapps/: gateway/ (3 docs), gitea/ (8 docs)",
]
prompt = build_high_prompt([doc], folder_lines=lines)
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n{lines[0]}\n{lines[1]}\n\n"
# Each line occurs exactly once, inside the section, before the
# first block (never after a summary — the E2E tail echo is safe).
for line in lines:
assert prompt.count(line) == 1
assert prompt.index("<documents>") < prompt.index(line) < prompt.index("<document ")
assert prompt.index(line) < prompt.index("</documents>")
# The <document> block markup AND body stay byte-identical: from the
# first block on, the prompt equals the no-folder-line build.
plain = build_high_prompt([doc])
assert prompt[prompt.index("<document ") :] == plain[plain.index("<document ") :]
# No new markup: the section still opens/closes exactly once (the
# ``<documents>`` MENTION in the TOOLS_SECTION copy is plain text —
# count the tag + newline, not the bare substring).
assert prompt.count("<documents>\n") == 1
assert prompt.count("</documents>") == 1
assert prompt.count("<document ") == 1
def test_high_prompt_single_folder_line_exact_shape() -> None:
"""One folder line: the exact slice between the section open and the
first block is ``<documents>\n`` + intro + ``\n`` + line +
``\n\n``."""
doc = _doc("a.md", "CONTENT", "Title A", summary="Summary A.")
line = "Homelab/: a.md (2 docs), b/ (1 doc)"
prompt = build_high_prompt([doc], folder_lines=[line])
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n{line}\n\n"
def test_high_prompt_empty_folder_lines_byte_identical_to_phase_118() -> None:
"""LOCKED A4: empty ``folder_lines`` (the default) ⇒ the phase-118
build, byte-identical — the existing pins keep passing and this
pins the default explicitly (both an explicit ``()`` and the
omitted parameter)."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
block = (
'<document source="Homelab" path="kubernetes.md" '
'title="Kubernetes Homelab Cluster" date="2024-06-15">\n'
"Talos Linux on three nodes.\n"
"</document>"
)
expected = (
_base("HIGH")
+ "\n<documents>\n"
+ SUGGEST_INTRO
+ "\n\n"
+ block
+ "\n</documents>"
+ "\n"
+ TOOLS_SECTION
)
# An explicit empty tuple AND the omitted parameter (the default):
assert build_high_prompt([doc], folder_lines=()) == expected
assert build_high_prompt([doc]) == expected
# The LOW (deflected) prompt has no folder_lines parameter at all —
# the deflected build stays byte-identical (the phase-118 pin in
# test_zero_note_prompt_is_byte_identical_to_pre_steering stands).
def test_folder_lines_ride_only_on_present_blocks() -> None:
"""Like the intro, the folder lines ride on present blocks ONLY: an
empty ``<documents>`` section is unchanged (no lines, no blocks)."""
prompt = build_high_prompt([], folder_lines=["X/: y.md"])
assert prompt == build_high_prompt([])
assert "X/: y.md" not in prompt
# ---------- phase 119 (D4, LOCKED A5): the cite-discipline sentence ----------
#: The LOCKED A5 sentence (phase 119, task 04) — the exact replacement
#: for the retired phase-118 final sentence of :data:`SUGGEST_INTRO`
#: ("Cite the document(s) you used, by path."). Closes the live
#: confabulation: the answer's "Docs used:" line cited a file the
#: agent never read.
CITE_DISCIPLINE = (
"Cite only the document(s) you read — or, if you answered from a "
"suggested summary without reading it, cite that suggested "
"document — never a document you neither read nor used."
)
def test_high_prompt_carries_cite_discipline_exactly_once_in_documents() -> None:
"""LOCKED A5: the HIGH prompt carries the discipline sentence
EXACTLY ONCE — it IS the intro's final sentence, so it sits inside
``<documents>``, AFTER the intro's start-here framing and BEFORE
the first block (both the plain build and the folder-line build;
the sentence rides the intro line, which never moves)."""
doc = _doc(
"kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster", summary="K8S"
)
assert CITE_DISCIPLINE in SUGGEST_INTRO
assert SUGGEST_INTRO.endswith(CITE_DISCIPLINE) # it is the final sentence
for high in (build_high_prompt([doc]), build_high_prompt([doc], folder_lines=["X/: y.md"])):
assert high.count(CITE_DISCIPLINE) == 1
i_docs = high.index("<documents>")
i_sentence = high.index(CITE_DISCIPLINE)
i_block = high.index("<document ")
i_close = high.index("</documents>")
assert i_docs < i_sentence < i_close
assert i_sentence > high.index("start here if one seems right")
assert i_sentence < i_block # before the first block, never after a summary
# The retired phase-118 sentence is gone from the prompt (it is
# gone from the constant — pinned in test_prompt_lock too).
high = build_high_prompt([doc])
assert "Cite the document(s) you used, by path." not in high
# The sentence rides on present blocks ONLY: an empty ``<documents>``
# section (no intro) carries none of it.
assert CITE_DISCIPLINE not in build_high_prompt([])
def test_cite_discipline_absent_from_low_prompt() -> None:
"""LOCKED A5: the discipline sentence belongs to ``SUGGEST_INTRO``
(the HIGH path) — it never leaks into the LOW (deflected) prompt,
whose byte-identity is pinned separately (the LOW anchors in
``test_prompt_lock`` pass unchanged)."""
for prompt in (
build_deflect_prompt(["T1", "T2"]),
build_deflect_prompt(["T1"], notes=["be concise"], kb_overview=OVERVIEW),
build_deflect_prompt([]),
):
assert CITE_DISCIPLINE not in prompt
assert "Cite only the document(s) you read" not in prompt
assert "Cite the document(s) you used, by path." not in prompt
def test_tools_section_old_names_and_budget_copy_gone() -> None:
"""The phase-37/68 tool names and the phase-37 per-tool budget line
(phase 45: the round cap is the bound — the prompt does not
tests/unit/test_retriever.py
+449 -45
View File
@@ -721,14 +721,15 @@ def _lexical_row(is_summary: bool, doc_path: str) -> object:
def test_lexical_candidates_carry_is_summary_flag() -> None:
"""The lexical list reads ``c.is_summary`` from the raw row.
The question carries no digit-bearing name token (no bare, no
numeric-join), so the name-hit path issues NO queries at all — the
single FTS rowset answers the only (FTS) call, and the list is the
plain FTS rows: the pre-name-hit behavior, unchanged.
The question's tokens are name candidates (class-agnostic, phase
119), so the name-hit projection runs — but the (empty) catalog
yields no path match, the LATERAL fetch is skipped, and the FTS
rowset answers the second call: the list is the plain FTS rows,
every one ``name_hit=False``.
"""
rows = [_lexical_row(True, "summary-src.yaml"), _lexical_row(False, "other.md")]
out = _lexical_candidates(
_FakeSession(rows), "how do i configure the thing", limit=10 # pyright: ignore[reportArgumentType]
_FakeSession([], rows), "how do i configure the thing", limit=10 # pyright: ignore[reportArgumentType]
)
assert len(out) == 2
by_path = {rc.document.path: rc for rc in out}
@@ -736,6 +737,7 @@ def test_lexical_candidates_carry_is_summary_flag() -> None:
assert by_path["summary-src.yaml"].position == -1
assert by_path["other.md"].is_summary is False
assert all(rc.fts_hit is True for rc in out)
assert all(rc.name_hit is False for rc in out) # ordinary FTS rows
def test_fuse_keeps_is_summary_on_double_hit() -> None:
@@ -783,30 +785,54 @@ def test_normalize_name() -> None:
def test_name_hit_tokens_incident_question() -> None:
"""The incident question yields EXACTLY the versioned join
``qwen38`` — the token the document names actually carry. Plain
prose words (``what``, ``llamacpp``, ``arguments``, ``server`` —
no digit) never name-match (the precision guard); the single
digits ("3", "8") and the bare "38" are < 4 chars; the
digit-leading ``38show`` boundary artifact is dropped."""
"""Phase 119 (LOCKED A2): the candidate list is CLASS-AGNOSTIC —
every normalized token of length >= 4 (dotted kept whole:
``llama.cpp`` → ``llamacpp``) plus the versioned join ``qwen38``.
The digit distinction moved to the match side (:func:`_name_hit_chunks`) —
prose precision now comes from the match class (a digitless token
must EQUAL a whole path component). The single digits ("3", "8")
and the bare "38" are < 4 chars; the digit-leading ``38show``
boundary artifact cannot survive (the join only fires on a purely
numeric SECOND token)."""
tokens = name_hit_tokens(INCIDENT_QUESTION)
assert tokens == ["qwen38"]
for absent in ("what", "qwen", "llamacpp", "arguments", "3", "8", "38", "38show", "server"):
assert tokens == ["what", "correct", "llamacpp", "arguments", "qwen", "qwen38"]
for absent in ("3", "8", "38", "38show", "server"):
assert absent not in tokens
def test_name_hit_tokens_no_digit_question_returns_empty() -> None:
"""A question with no digit-bearing token (bare or joined) yields
no name candidates — prose joins like ``correctllama`` never count."""
assert name_hit_tokens("what is the correct caddy config") == []
def test_name_hit_tokens_digitless_question_yields_long_tokens() -> None:
"""A question with NO digit-bearing token still yields candidates
(every normalized token of length >= 4) — the 2026-09-16 fix:
product names without digits ("gitea", "gateway") must get a name
signal. Prose joins (``correctcaddy``) never count (the second
token is not purely numeric)."""
assert name_hit_tokens("what is the correct caddy config") == [
"what", "correct", "caddy", "config",
]
assert name_hit_tokens("a e i o u 3 8") == []
def test_name_hit_tokens_bare_digit_bearing_token() -> None:
"""A single written token that carries a digit (``1panel``) is a
name candidate on its own — no join needed."""
name candidate on its own — no join needed — alongside the plain
prose tokens of the same question (class-agnostic list)."""
tokens = name_hit_tokens("what is my 1panel dashboard setup")
assert tokens == ["1panel"]
assert tokens == ["what", "1panel", "dashboard", "setup"]
def test_name_hit_tokens_versioned_join_and_short_tokens() -> None:
"""The versioned join survives the class-agnostic change ("Qwen
3.8" → ``qwen38``), and short tokens (< :data:`NAME_TOKEN_MIN_LEN`
normalized — the single digits, "3.8" → ``38``) never become
candidates, with or without a join."""
tokens = name_hit_tokens("help me with Qwen 3.8 please")
# The join is appended at its FIRST token's position (after "qwen").
assert tokens == ["help", "with", "qwen", "qwen38", "please"]
assert name_hit_tokens("3.8 8 16 9") == [] # 38 / 8 / 16 / 9 / 816 / 169 all < 4
# Word-after-version: the word itself is a candidate, but the
# digit-leading join artifact ("38show") cannot survive (the join
# only fires on a purely numeric SECOND token).
assert name_hit_tokens("3.8 show") == ["show"]
def _name_row(doc: Document) -> tuple:
@@ -834,7 +860,8 @@ def _name_hit_lateral_row(doc: Document, is_summary: bool = False) -> SimpleName
def test_name_hit_chunks_no_tokens_skips_all_queries() -> None:
"""A question with no name tokens issues no queries at all."""
"""A question with no name tokens (every normalized token < 4)
issues no queries at all."""
session = _FakeSession([]) # any call would surface a statement
assert _name_hit_chunks(session, "a e i o u 3 8") == [] # pyright: ignore[reportArgumentType]
assert session.statements == []
@@ -850,46 +877,124 @@ def test_name_hit_chunks_no_matching_doc_returns_empty() -> None:
assert len(session.statements) == 1 # projection only — no LATERAL fetch
def test_name_hit_chunks_ranked_by_count_length_catalog() -> None:
"""A two-candidate question (``qwen38`` + ``1panel``): the document
whose name carries BOTH (2 matches, 12 total chars) leads; the two
single-match documents tie on (1, 6) and fall to catalog order
(``dashboards/1panel-notes.md`` before ``quadlets/qwen3.8…``).
Hits carry ``fts_hit=True`` (the A8 gate answers), ``cosine=0.0``,
and the summary flag of their representative chunk."""
both = _doc("dashboards/1panel-qwen3.8.md", "body", title="1Panel Qwen 3.8")
panel = _doc("dashboards/1panel-notes.md", "body")
q38 = _doc("quadlets/qwen3.8-27b-juggernaut-vulkan.container", "body")
name_rows = [_name_row(d) for d in (panel, both, q38)] # catalog order
question = "what are the correct llama.cpp arguments for qwen 3.8 and the 1panel dashboard?"
def test_name_hit_chunks_digitless_exact_part_stem_subcomponent() -> None:
"""A DIGITLESS token EQUALS a normalized path part (the ``gitea/``
folder), the file stem (``gitea.md``), or a stem sub-component
(``kubernetes_gitea``, ``gitea-values``, ``test-gateway`` — the
stem split on non-alphanumeric runs) — the 2026-09-16 product-name
signal (LOCKED A2)."""
question = "how do i set up gitea or the gateway" # tokens: [gitea, gateway]
docs = [
_doc("deploy/reeseapps/gitea/README.md", "body"), # path part
_doc("notes/gitea.md", "body"), # file stem
_doc("deploy/k8s/kubernetes_gitea.md", "body"), # sub-component
_doc("deploy/k8s/gitea-values.yaml", "body"), # sub-component
_doc("deploy/istio/test-gateway.yaml", "body"), # sub-component (gateway)
_doc("notes/gitlab.md", "body"), # NO component matches — excluded
]
name_rows = [_name_row(d) for d in docs]
session = _FakeSession(name_rows, [_name_hit_lateral_row(d) for d in docs[:5]])
out = _name_hit_chunks(session, question) # pyright: ignore[reportArgumentType]
# Five one-token hits, catalog order (source, path):
assert [rc.document.path for rc in out] == [
"deploy/istio/test-gateway.yaml",
"deploy/k8s/gitea-values.yaml",
"deploy/k8s/kubernetes_gitea.md",
"deploy/reeseapps/gitea/README.md",
"notes/gitea.md",
]
assert all(rc.name_hit is True for rc in out)
assert all(rc.fts_hit is True for rc in out) # the lexical signal
assert all(rc.cosine == 0.0 for rc in out) # no vector rank
def test_name_hit_chunks_digitless_title_never_matched() -> None:
"""The owner-verified failure mode of the naive relaxation: a doc
under a ``Deployments/`` folder titled "Deployments" does NOT hit
the common token ``deploy`` (the part normalizes to
``deployments`` ≠ ``deploy``), and a doc titled "Gitea" with no
gitea path component does NOT hit ``gitea`` — TITLES ARE NEVER
MATCHED (LOCKED A2)."""
question = "how do i deploy gitea" # tokens: [deploy, gitea]
docs = [
_doc("Deployments/reeseapps/README.md", "body", title="Deployments"),
_doc("notes/internal-notes.md", "body", title="Gitea"), # title only
]
session = _FakeSession([_name_row(d) for d in docs], [])
assert _name_hit_chunks(session, question) == [] # pyright: ignore[reportArgumentType]
assert len(session.statements) == 1 # projection only — no LATERAL fetch
def test_name_hit_chunks_digit_bearing_prefix_not_midword() -> None:
"""A DIGIT-BEARING token is a PREFIX of a normalized part or stem
(``qwen38`` → ``qwen3.8-27b-epic-vulkan.container``) — a stem that
merely CONTAINS the token mid-word (``xqwen38y…``) does NOT hit;
sub-components are in the exact-match class only (LOCKED A2)."""
question = "what are the arguments for qwen 3.8" # tokens: what, arguments, qwen, qwen38
hit = _doc("quadlets/qwen3.8-27b-epic-vulkan.container", "body")
miss = _doc("quadlets/xqwen38y-test.container", "body") # mid-word containment
name_rows = [_name_row(hit), _name_row(miss)]
session = _FakeSession(name_rows, [_name_hit_lateral_row(hit)])
out = _name_hit_chunks(session, question) # pyright: ignore[reportArgumentType]
assert [rc.document.path for rc in out] == [hit.path]
assert out[0].name_hit is True
def test_name_hit_chunks_ranked_by_count_then_catalog() -> None:
"""A question (``deploy`` + ``gitea`` + ``qwen`` + ``qwen38``):
the document whose path carries BOTH a digitless component and a
digit-bearing prefix (2 matched tokens) leads; the two
single-token documents tie on count and fall to CATALOG ORDER —
the old total-matched-length tie-break is RETIRED (it would have
put the 6-char ``qwen38`` hit, ``quadlets/…``, before the 5-char
``gitea`` hit, ``gitea/notes.md`` — the flip is pinned). The
"Deployments"-titled doc and the title-only "Gitea" doc never
appear (titles are never matched)."""
precision = _doc("Deployments/reeseapps/README.md", "body", title="Deployments")
gitea_notes = _doc("gitea/notes.md", "body", title="Internal notes")
both = _doc("gitea/qwen3.8-model.container", "body", title="The model quadlet")
q38 = _doc("quadlets/qwen3.8-27b-juggernaut-vulkan.container", "body", title="juggernaut")
title_only = _doc("notes/internal-notes.md", "body", title="Gitea")
name_rows = [_name_row(d) for d in (precision, gitea_notes, both, q38, title_only)]
question = "how do i deploy gitea with qwen 3.8"
lateral_rows = [
_name_hit_lateral_row(q38, is_summary=True), # LATERAL may return any order
_name_hit_lateral_row(both),
_name_hit_lateral_row(panel),
_name_hit_lateral_row(gitea_notes),
]
session = _FakeSession(name_rows, lateral_rows)
out = _name_hit_chunks(session, question) # pyright: ignore[reportArgumentType]
assert [rc.document.path for rc in out] == [
"dashboards/1panel-qwen3.8.md", # 2 matched tokens — leads
"dashboards/1panel-notes.md", # (1, 6) — catalog order
"quadlets/qwen3.8-27b-juggernaut-vulkan.container", # (1, 6) — after
"gitea/qwen3.8-model.container", # 2 matched tokens (gitea + qwen38) — leads
"gitea/notes.md", # 1 token (gitea, 5 chars) — catalog order beats quadlets
"quadlets/qwen3.8-27b-juggernaut-vulkan.container", # 1 token (qwen38, 6 chars)
]
assert all(rc.name_hit is True for rc in out)
assert all(rc.fts_hit is True for rc in out) # the lexical signal
assert all(rc.cosine == 0.0 for rc in out) # no vector rank
assert all(rc.score == 0.0 for rc in out) # fuse fills the score
by_path = {rc.document.path: rc for rc in out}
# The representative chunk keeps its summary flag (the LATERAL
# choice: is_summary DESC, position ASC — chunk 0 otherwise).
assert by_path["quadlets/qwen3.8-27b-juggernaut-vulkan.container"].is_summary is True
assert by_path["quadlets/qwen3.8-27b-juggernaut-vulkan.container"].position == -1
assert by_path["dashboards/1panel-notes.md"].is_summary is False
assert by_path["gitea/notes.md"].is_summary is False
def test_name_hit_chunks_short_tokens_never_hit() -> None:
"""Short tokens (< 4 normalized — "3.8" → ``38``, the single
digits) are never candidates, so they can never hit, with or
without the versioned join."""
session = _FakeSession([]) # any call would surface a statement
assert _name_hit_chunks(session, "3.8 8 16 9") == [] # pyright: ignore[reportArgumentType]
assert session.statements == []
def test_name_hit_chunks_capped_at_limit() -> None:
"""Twelve tied name hits (one matched token each) yield exactly
``NAME_HIT_LIMIT`` of them — catalog order (the deterministic
tie-break)."""
docs = [_doc(f"quadlets/m{i:02d}.container", "body") for i in range(12)]
for d in docs: # give every document a name that carries the token
d.title = "qwen38 model i"
"""Twelve tied name hits (one matched token each — the ``qwen38``
stem prefix) yield exactly ``NAME_HIT_LIMIT`` of them — catalog
order (the deterministic tie-break)."""
docs = [_doc(f"quadlets/qwen3.8-m{i:02d}.container", "body") for i in range(12)]
name_rows = [_name_row(d) for d in docs]
# Only the ten winners (catalog order — the deterministic tie-break
# of the twelve identical scores) reach the LATERAL fetch; the fake
@@ -900,7 +1005,10 @@ def test_name_hit_chunks_capped_at_limit() -> None:
session, "tell me about the qwen 3.8 models" # pyright: ignore[reportArgumentType]
)
assert len(out) == NAME_HIT_LIMIT
assert [rc.document.path for rc in out] == [f"quadlets/m{i:02d}.container" for i in range(10)]
assert [rc.document.path for rc in out] == [
f"quadlets/qwen3.8-m{i:02d}.container" for i in range(10)
]
assert all(rc.name_hit is True for rc in out)
def test_lexical_candidates_name_hits_lead_and_dedupe_with_fts() -> None:
@@ -933,7 +1041,7 @@ def test_lexical_candidates_name_hits_lead_and_dedupe_with_fts() -> None:
session = _FakeSession(name_rows, lateral_rows, fts_rows)
out = _lexical_candidates(session, INCIDENT_QUESTION, limit=10) # pyright: ignore[reportArgumentType]
assert len(out) == 3 # q38 (once), other (name hit), other (FTS chunk)
# Both name hits tie on (1, 6) — catalog order: "qwen3." (ASCII 46)
# Both name hits tie on count (1) — catalog order: "qwen3." (ASCII 46)
# sorts before "qwen38" (ASCII 56).
assert out[0].document.path == "quadlets/qwen3.8-27b-juggernaut-vulkan.container"
assert out[1].document.path == "quadlets/qwen38-other.container"
@@ -942,3 +1050,299 @@ def test_lexical_candidates_name_hits_lead_and_dedupe_with_fts() -> None:
rc.chunk_id for rc in out
} == {q38_chunk, fts_rows[1].chunk_id, lateral_rows[1].chunk_id}
assert all(rc.fts_hit is True for rc in out)
# Phase 119: the name-hit representative rows are flagged, the plain
# FTS row is not (the selection tier's bonus input, task 02).
assert out[0].name_hit is True
assert out[1].name_hit is True
assert out[2].name_hit is False
# ---------------------------------------------------------------------------
# Phase 119, D1 — the name_hit flag through fusion
# ---------------------------------------------------------------------------
def test_fuse_keeps_name_hit_on_lexical_only_hit() -> None:
"""A name-hit row with no vector rank keeps ``name_hit=True``
through the fusion (the ``replace()`` copy carries the field)."""
nh = _rc("gitea/README.md")
nh.name_hit = True
out = fuse([], [nh], k=60)
assert len(out) == 1
assert out[0].name_hit is True
assert out[0].fts_hit is True
assert out[0].cosine == 0.0
def test_fuse_or_s_name_hit_on_double_hit() -> None:
"""A vector row that is ALSO the name hit's representative chunk
(the RRF merge dedupes by chunk id) keeps ``name_hit=True`` — the
merge ORs the flag in, so the selection tier (task 02) still sees
the name hit on the fused list."""
v = _rc("gitea/README.md", cosine=0.9)
l1 = _rc("gitea/README.md", cosine=0.1) # the lexical copy of the same chunk
l1.chunk_id = v.chunk_id
l1.name_hit = True
out = fuse([v], [l1], k=60)
assert len(out) == 1
assert out[0].name_hit is True
assert out[0].fts_hit is True
assert out[0].score == pytest.approx(2 / 61)
def test_fuse_default_name_hit_stays_false_for_ordinary_rows() -> None:
"""Neither list flagged ⇒ fusion never invents a name-hit flag —
ordinary vector and FTS rows are ``name_hit=False``."""
out = fuse([_rc("a.md", cosine=0.8)], [_rc("b.md", fts_hit=True)], k=60)
assert len(out) == 2
assert all(rc.name_hit is False for rc in out)
# ---------------------------------------------------------------------------
# Phase 119, D2 — the bounded name-hit bonus (LOCKED A3)
# ---------------------------------------------------------------------------
from app.rag.retriever import _selection_order, weak_hit_titles # noqa: E402
def _bonus_rc(
doc: Document,
score: float,
cosine: float,
position: int = 0,
name_hit: bool = False,
) -> RetrievedChunk:
"""One fused-list candidate (name-hit rows follow the D1 lexical
convention: ``cosine=0.0``, ``fts_hit=True``)."""
return RetrievedChunk(
chunk_id=uuid.uuid4(),
position=position,
content=doc.content[:20],
score=score,
document=doc,
cosine=cosine,
fts_hit=cosine == 0.0,
name_hit=name_hit,
)
def _bonus_chunks() -> list[RetrievedChunk]:
"""A mixed FUSED list — already in the ``fuse()`` key order
(−score, −cosine, path, position) — with one name-hit document
(``gitea/README.md``, the D1 convention: cosine 0.0) and ordinary
vector/FTS documents: ``a.md`` carries two chunks, and ``b.md`` /
``c.md`` tie on the fused score (separated only by cosine).
The pre-phase (bonus-0) document order this list walks — the golden
the kill switch must reproduce — is a (0.0200) → gitea (0.0160) →
b (0.0150, cos 0.7) → c (0.0150, cos 0.6) → d (0.0100).
"""
gitea = _doc("gitea/README.md", "G" * 50)
a = _doc("a.md", "A" * 50)
b = _doc("b.md", "B" * 50)
c = _doc("c.md", "C" * 50)
d = _doc("d.md", "D" * 50)
return [
_bonus_rc(a, 0.0200, 0.9, 0),
_bonus_rc(gitea, 0.0160, 0.0, 0, name_hit=True),
_bonus_rc(b, 0.0150, 0.7, 0),
_bonus_rc(c, 0.0150, 0.6, 0),
_bonus_rc(a, 0.0120, 0.5, 1),
_bonus_rc(d, 0.0100, 0.1, 0),
]
#: The golden document order the OLD pre-phase loop (stable score-
#: descending walk, first-seen-chunk dedupe) produces over
#: :func:`_bonus_chunks` — pinned byte-identical by the kill switch.
GOLDEN_PRE_PHASE_ORDER = ["a.md", "gitea/README.md", "b.md", "c.md", "d.md"]
def test_selection_order_bonus_zero_is_the_pre_phase_golden_walk() -> None:
"""LOCKED A3 kill switch: ``bonus=0`` returns the EXACT pre-phase
document order of the old score-descending first-seen walk — the
golden list pinned from the old loop over the mixed fused list
(incl. the b/c fused-score tie resolved by the input order the
fusion produced — the walk never re-sorts it away)."""
out = _selection_order(_bonus_chunks(), 0.0)
assert [d.path for d, _eff, _cos, _idx in out] == GOLDEN_PRE_PHASE_ORDER
# The re-rank inputs are exposed and exact: effective == best fused
# score (no bonus), best cosine tracked across a doc's chunks (a: 0.9
# from its rank-1 chunk, not 0.5), first-seen index in the
# score-descending walk.
assert [eff for _d, eff, _cos, _idx in out] == [
0.0200, 0.0160, 0.0150, 0.0150, 0.0100
]
assert [cos for _d, _eff, cos, _idx in out] == [
pytest.approx(v) for v in (0.9, 0.0, 0.7, 0.6, 0.1)
]
assert [idx for _d, _eff, _cos, idx in out] == [0, 1, 2, 3, 5]
def test_selection_order_bonus_inert_without_name_hits() -> None:
"""No name-hit chunk present → the bonus cannot fire: the order is
IDENTICAL to the pre-phase walk even with the default bonus on
(LOCKED A3)."""
chunks = _bonus_chunks()
for rc in chunks:
rc.name_hit = False
out = _selection_order(chunks, 0.005)
assert [d.path for d, _eff, _cos, _idx in out] == GOLDEN_PRE_PHASE_ORDER
def test_selection_order_bonus_lifts_name_hit_doc_below_bonus_gap() -> None:
"""The name-hit doc's effective 0.016 + 0.005 = 0.021 EXCEEDS a's
0.020 — a gap of 0.004 < bonus 0.005 — so the bonus lifts it to
rank 1; the rest keep their fused order (the bonus re-ranks, it
does not inflate)."""
out = _selection_order(_bonus_chunks(), 0.005)
assert [d.path for d, _eff, _cos, _idx in out] == [
"gitea/README.md", "a.md", "b.md", "c.md", "d.md"
]
assert out[0][1] == pytest.approx(0.016 + 0.005)
def test_selection_order_bonus_does_not_lift_above_bonus_gap() -> None:
"""A gap LARGER than the bonus is not closed: the name-hit doc's
best 0.010 + 0.005 = 0.015 ties b/c on effective and LOSES to both
on the (−effective, −best_cosine) tie-break (its D1 cosine is 0.0) —
a keeps the lead (0.020). A second name-hit doc (``e.md``, also
0.010/cos 0.0) trails gitea on the ``document.path`` tie-break —
the full re-rank key pinned."""
chunks = _bonus_chunks()
chunks[1].score = 0.010 # the name-hit doc drops to a 0.010 best
e = _doc("e.md", "E" * 50)
chunks.insert(2, _bonus_rc(e, 0.010, 0.0, 0, name_hit=True))
out = _selection_order(chunks, 0.005)
assert [d.path for d, _eff, _cos, _idx in out] == [
"a.md", "b.md", "c.md", "e.md", "gitea/README.md", "d.md"
]
def test_selection_order_first_seen_breaks_equal_path_ties() -> None:
"""Two documents sharing a path across sources (``notes.md`` in two
sources) can tie on (effective, cosine, path) — the pre-bonus
first-seen rank decides (the last key element)."""
s1 = _doc("notes.md", "X" * 50, source="Src1")
s2 = _doc("notes.md", "Y" * 50, source="Src2")
chunks = [
_bonus_rc(s1, 0.016, 0.0, 0, name_hit=True),
_bonus_rc(s2, 0.016, 0.0, 0, name_hit=True),
]
out = _selection_order(chunks, 0.005)
assert [d.source for d, *_ in out] == ["Src1", "Src2"]
def test_selection_order_bonus_applied_once_per_document() -> None:
"""The bonus is per DOCUMENT — applied ONCE no matter how many of
the doc's chunks are name hits (3×bonus would push the name-hit doc
above the 0.030 leader; one bonus cannot)."""
gitea = _doc("gitea/README.md", "G" * 50)
a = _doc("a.md", "A" * 50)
chunks = [
_bonus_rc(a, 0.030, 0.8),
_bonus_rc(gitea, 0.016, 0.0, 0, name_hit=True),
_bonus_rc(gitea, 0.010, 0.0, 1, name_hit=True),
_bonus_rc(gitea, 0.008, 0.0, 2, name_hit=True),
]
out = _selection_order(chunks, 0.005)
assert [d.path for d, _eff, _cos, _idx in out] == ["a.md", "gitea/README.md"]
assert out[1][1] == pytest.approx(0.016 + 0.005) # best + ONE bonus
def test_selection_order_bonus_fires_when_name_hit_is_not_first_chunk() -> None:
"""The bonus fires on ANY name-hit chunk of the document — including
when the doc's first-seen (best) chunk is an ordinary vector row and
only a lower-ranked chunk is the D1 name-hit representative (a
document's name hit and its best chunk can be different chunks).
The bonus still lands on the doc's BEST fused score, and the doc's
best cosine stays tracked across ALL its chunks."""
gitea = _doc("gitea/README.md", "G" * 50)
a = _doc("a.md", "A" * 50)
chunks = [
_bonus_rc(a, 0.024, 0.8),
_bonus_rc(gitea, 0.020, 0.2, 0), # the doc's best — an ordinary chunk
_bonus_rc(gitea, 0.016, 0.55, 2, name_hit=True), # the name-hit rep
]
assert [d.path for d, *_ in _selection_order(chunks, 0.0)] == [
"a.md", "gitea/README.md"
]
out = _selection_order(chunks, 0.005)
assert [d.path for d, *_ in out] == ["gitea/README.md", "a.md"]
assert out[0][1] == pytest.approx(0.020 + 0.005) # bonus on the BEST score
# The doc's best cosine is tracked across ALL its chunks — the
# lower-ranked name-hit chunk (0.55) beats the first-seen chunk's
# 0.2 (the re-rank tie-break input).
assert out[0][2] == pytest.approx(0.55)
def test_suggested_bonus_default_from_settings_and_kill_switch(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The *bonus* parameter defaults to the LIVE ``BOR_NAME_HIT_BONUS``
setting (the ``n`` parameter's settings-read pattern — the default
0.005 is the production value, not a frozen constant); an explicit
``bonus=0`` and a settings kill switch both reproduce the pre-phase
golden walk (LOCKED A3)."""
chunks = _bonus_chunks()
settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
assert settings.name_hit_bonus == 0.005 # the production default
monkeypatch.setattr(retriever, "get_settings", lambda: settings)
assert [d.path for d in select_suggested(chunks, n=5)] == [
"gitea/README.md", "a.md", "b.md", "c.md", "d.md"
]
# Explicit kill switch: the byte-identical pre-phase order.
assert [d.path for d in select_suggested(chunks, n=5, bonus=0.0)] == GOLDEN_PRE_PHASE_ORDER
# Settings kill switch (BOR_NAME_HIT_BONUS=0) — the same golden walk.
off = Settings(_env_file=None, name_hit_bonus=0.0) # pyright: ignore[reportCallIssue]
monkeypatch.setattr(retriever, "get_settings", lambda: off)
assert [d.path for d in select_suggested(chunks, n=5)] == GOLDEN_PRE_PHASE_ORDER
def test_related_skips_excluded_ids_under_the_bonus() -> None:
"""Exclusion is orthogonal to the bonus: excluded ids are skipped
exactly as before, on the bonus-adjusted walk — an excluded doc
never rides the related row even when the bonus would lift it to
the lead."""
chunks = _bonus_chunks()
b_id = chunks[2].document.id
d_id = chunks[5].document.id
out = select_related(chunks, {b_id, d_id}, cap=5, bonus=0.005)
assert [d.path for d in out] == ["gitea/README.md", "a.md", "c.md"]
# Kill switch: the same exclusions on the pre-phase walk.
out0 = select_related(chunks, {b_id, d_id}, cap=5, bonus=0.0)
assert [d.path for d in out0] == ["a.md", "gitea/README.md", "c.md"]
# The name-hit doc itself excluded → the lead goes to the next doc.
g_id = chunks[1].document.id
out2 = select_related(chunks, {g_id}, cap=5, bonus=0.005)
assert [d.path for d in out2][0] == "a.md"
def test_weak_hit_titles_bonus_adjusted_order() -> None:
"""Titles follow the bonus-adjusted selection walk (``_doc`` titles
equal paths here, so the title list mirrors the doc order); the
kill switch returns the pre-phase golden order."""
chunks = _bonus_chunks()
assert weak_hit_titles(chunks, bonus=0.005) == [
"gitea/README.md", "a.md", "b.md", "c.md", "d.md"
]
assert weak_hit_titles(chunks, bonus=0.0) == GOLDEN_PRE_PHASE_ORDER
def test_bonus_lives_in_the_selection_layer_only() -> None:
"""LOCKED A3: the bonus never touches the chunk objects —
``score``/``cosine``/``fts_hit`` (the A8 gate's inputs —
``query_log.top_score`` is the best fused chunk score, the same
values) are unchanged after every selection walk, even with the
bonus lifting a document."""
chunks = _bonus_chunks()
before = {
rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks
}
select_suggested(chunks, n=5, bonus=0.005)
select_related(chunks, set(), cap=5, bonus=0.005)
weak_hit_titles(chunks, bonus=0.005)
after = {
rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks
}
assert before == after
tests/unit/test_source_chip_quality.py
+33 -26
View File
@@ -633,6 +633,13 @@ class _FakeSession:
def get(self, _model: Any, _pk: Any) -> Any:
return None
def execute(self, *args: Any, **kwargs: Any) -> list[Any]:
# Phase 119 (D3): the endpoint's suggested-folder lines run the
# ls catalog fetchers on this session — the fake catalog is
# empty (header-only lines), keeping the prompt builds
# deterministic here.
return []
@pytest.fixture(autouse=True)
def _admin_signed_in(client: TestClient) -> None:
@@ -736,17 +743,19 @@ def test_shape_4_meta_question_deflected_frame_has_no_chips_or_row(
assert "108_history_wire_check/00_phase.md" in row.sources
def test_done_frame_carries_suggested_refs_strong_plus_weak(
def test_done_frame_zero_read_grounded_turn_carries_no_chips(
client: TestClient,
chip_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Shape 2 on the wire under phase 118 (LOCKED A4): the citation
surface is the suggested tier + the agent's reads (deduped) —
with two retrieved docs and no read, BOTH docs are ``sources``
refs (no floor — A3); nothing reaches rank 6+, so ``related`` is
empty; the tiers stay disjoint (the done frame's dedupe). The
durable record keeps the FULL retrieval (LOCKED A3)."""
"""Shape 2 on the wire under phase 119 (LOCKED A1 — the phase-118
A4 union retired): the citation surface is the agent's READ DOCS
only — with two retrieved docs and no read, ``sources`` is empty
(the zero-read grounded turn chips nothing — an accepted,
owner-directed consequence); the never-read suggested docs appear
NOWHERE in the frame (the explicit contrast against the retired
A4 union); nothing reaches rank 6+, so ``related`` is empty. The
durable record keeps the FULL retrieval (118-A3 stands)."""
session, _llm = chip_env
validate = _doc("brain-of-reese", ".agents/validate.sh", "validate.sh",
"VALIDATE_SH_CONTENT")
@@ -766,14 +775,11 @@ def test_done_frame_carries_suggested_refs_strong_plus_weak(
frames = _ask(client, "How does the phase gate decide to validate?")
done = frames[-1]
assert done["deflected"] is False
assert [(s["source"], s["path"]) for s in done["sources"]] == [
("brain-of-reese", ".agents/validate.sh"),
("ServMon", "README.md"), # A4: suggested + read — both suggested (A3)
]
# A1: read docs only — the agent read nothing ⇒ no chips, and the
# never-read suggested docs appear NOWHERE in the frame (the retired
# A4 union would have carried both).
assert done["sources"] == []
assert done["related"] == [] # nothing reaches rank 6+ for 2 docs
cited = {(s["source"], s["path"]) for s in done["sources"]}
related = {(s["source"], s["path"]) for s in done["related"]}
assert cited.isdisjoint(related)
(row,) = session.added
assert isinstance(row, QueryLog)
@@ -788,14 +794,15 @@ def test_agent_read_related_doc_is_cited_not_related(
chip_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The agent-read exemption (LOCKED A4, phase-118 tiering): a
rank-6+ doc — the related tier ("nearby docs") — that the agent
``read`` via the tool is cited by definition: the model read it, so
it was used. It joins ``sources`` (after the suggested docs — it
was not suggested, so the read appends it last, deduped) and is
EXCLUDED from ``related`` (a used doc must never read as "nearby");
the other rank-6+ doc stays in the tier. The read content reached
the model (the tool result in the follow-up request)."""
"""The agent-read exemption (phase-119 A1 × phase-118 tiering):
``sources`` is the READ DOCS only — a rank-6+ doc (the related
tier, "nearby docs") that the agent ``read`` via the tool is the
frame's ONLY chip: the model read it, so it was used. It is EXCLUDED
from ``related`` (a used doc must never read as "nearby" — unchanged
intent, the dedupe now keyed on read docs); the other rank-6+ doc
stays in the tier; the never-read suggested docs appear NOWHERE in
the frame (contrast against the retired A4 union). The read content
reached the model (the tool result in the follow-up request)."""
session, _default_llm = chip_env
strong = _doc("docs", "strong.md", "Strong", "STRONG_DOC_CONTENT")
fillers = [
@@ -838,10 +845,10 @@ def test_agent_read_related_doc_is_cited_not_related(
assert "WEAK_B_READ_BY_AGENT" in tool_msgs[0]["content"]
sources = [(s["source"], s["path"]) for s in done["sources"]]
# A4: suggested (5) + the read doc (last — it was not suggested).
assert sources[-1] == ("docs", "weak-b.md") # read ⇒ cited, last
assert len(sources) == 6
assert ("docs", "weak-c.md") not in sources # never suggested, never read
# A1: read docs only — the read related doc is the ONLY chip (the
# never-read suggested docs are nowhere in the frame).
assert sources == [("docs", "weak-b.md")] # read ⇒ chip
assert ("docs", "weak-c.md") not in sources # never read
related = [(s["source"], s["path"]) for s in done["related"]]
assert related == [("docs", "weak-c.md")] # the read doc is not "nearby"
assert set(sources).isdisjoint(set(related))