Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 119_name_signal_read_chips
Build and Push Containers / build-and-push-app (push) Successful in 2m1s
Details
Build and Push Containers / build-and-push-db (push) Successful in 18s
Details

Browse Source 
All verification complete. Final report:

**Phase 119 final verification pass — all criteria verified, one stale pin fixed.**
- Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry.
- Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged.
- New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2.
- Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed).
- Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met).
- Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
ducoterra
2026-09-16 15:50:48 -04:00
parent 795fb56425
commit a5b63f83ad
89 changed files with 4377 additions and 655 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/unit/test_agent.py
+223
View File
@@ -1091,6 +1091,229 @@ def test_group_folder_listing_caps_files_at_fifty_keeps_the_total() -> None:
assert total50 == 50 and len(files50) == 50
# ---------- phase 119 (D3): the suggested-folder context lines ----------
def _patch_catalog(
monkeypatch: pytest.MonkeyPatch, catalog: dict[str, list[tuple[str, str, str]]]
) -> None:
"""Monkeypatch the ``ls`` fetchers (house style) with one source
catalog of ``(path, title, date)`` rows per source name."""
monkeypatch.setattr(
agent, "_source_document_rows", lambda db, source: catalog.get(source, [])
)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
def test_suggested_folder_lines_source_root_doc_lists_the_top_level(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A suggested doc at the source root (``""`` prefix) renders the
``<source>/: …`` line with the source's top level — subfolders with
recursive counts, the suggested doc itself excluded."""
_patch_catalog(
monkeypatch,
{
"Homelab": [
("a/b/one.md", "One", "2024-06-15"),
("a/b/two.md", "Two", "2024-06-15"),
("a/c.md", "C", "2024-06-15"),
("z.md", "Z", "2024-06-15"),
]
},
)
assert agent.suggested_folder_lines(
cast("Session", object()), [_doc("Homelab", "z.md")]
) == ["Homelab/: a/ (3 docs)"]
def test_suggested_folder_lines_nested_doc_lists_the_parent_folder(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A nested suggested doc renders the PARENT folder's line — the
direct children only: subfolder names (source-relative, recursive
counts) + the files by relative filename, the suggested doc itself
out of the entries."""
_patch_catalog(
monkeypatch,
{
"Homelab": [
("a/b/one.md", "One", "2024-06-15"),
("a/b/two.md", "Two", "2024-06-15"),
("a/c/d.md", "D", "2024-06-15"),
("a/e.md", "E", "2024-06-15"),
]
},
)
# The parent of a/b/one.md is a/b: one.md excluded, two.md stays.
assert agent.suggested_folder_lines(
cast("Session", object()), [_doc("Homelab", "a/b/one.md")]
) == ["Homelab/a/b/: two.md"]
# The parent of a/e.md is a: the subfolders (recursive counts — a/b
# carries two, a/c one) first, then the files; e.md excluded, so no
# file entries remain.
assert agent.suggested_folder_lines(
cast("Session", object()), [_doc("Homelab", "a/e.md")]
) == ["Homelab/a/: a/b/ (2 docs), a/c/ (1 doc)"]
def test_suggested_folder_lines_excludes_only_the_owning_doc(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The doc that OWNS the line (the first suggested doc in that
folder) is the only one excluded — a folder whose sole entry was
the suggested doc renders its header alone (the empty-level
precedent)."""
_patch_catalog(
monkeypatch,
{
"Homelab": [("a/only.md", "Only", "2024-06-15")]
},
)
assert agent.suggested_folder_lines(
cast("Session", object()), [_doc("Homelab", "a/only.md")]
) == ["Homelab/a/:"]
def test_suggested_folder_lines_dedupes_by_source_and_prefix(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Dedupe: two suggested docs in the same folder ⇒ ONE line (the
first doc's position, the first doc excluded, the second listed);
the key is ``(source, prefix)`` — the same prefix under a DIFFERENT
source is a distinct line; suggested-doc order is the line order."""
_patch_catalog(
monkeypatch,
{
"Homelab": [
("a/b/one.md", "One", "2024-06-15"),
("a/b/two.md", "Two", "2024-06-15"),
("a/c.md", "C", "2024-06-15"),
("a/d/e.md", "E", "2024-06-15"),
("z.md", "Z", "2024-06-15"),
],
"Other": [("a/b/x.md", "X", "2024-06-15")],
},
)
docs = [
_doc("Homelab", "z.md"),
_doc("Homelab", "a/b/one.md"),
_doc("Homelab", "a/b/two.md"), # deduped (a/b already seen)
_doc("Homelab", "a/c.md"),
_doc("Homelab", "a/d/e.md"),
_doc("Other", "a/b/x.md"), # same prefix, different source
]
assert agent.suggested_folder_lines(cast("Session", object()), docs) == [
"Homelab/: a/ (4 docs)",
"Homelab/a/b/: two.md",
"Homelab/a/: a/b/ (2 docs), a/d/ (1 doc)",
"Homelab/a/d/:",
"Other/a/b/:",
]
def test_suggested_folder_lines_max_lines_caps_the_list(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""At most *max_lines* lines — the walk stops, suggested-doc order
kept (the first distinct folders win)."""
_patch_catalog(
monkeypatch,
{
"Homelab": [
("f1/x.md", "1", "2024-06-15"),
("f2/x.md", "2", "2024-06-15"),
("f3/x.md", "3", "2024-06-15"),
]
},
)
docs = [_doc("Homelab", p) for p in ("f1/x.md", "f2/x.md", "f3/x.md")]
lines = agent.suggested_folder_lines(cast("Session", object()), docs, max_lines=2)
assert lines == ["Homelab/f1/:", "Homelab/f2/:"]
assert agent.suggested_folder_lines(
cast("Session", object()), docs, max_lines=1
) == ["Homelab/f1/:"]
def test_suggested_folder_lines_max_entries_cap_and_remainder(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""At most *max_entries* entries, then `` +N more`` with the EXACT
remainder (the true pre-cap folder total — the owning doc leaves
it); the ``(1 doc)`` singular form; no `` +N more`` when every
entry fits."""
rows = [(f"big/s{i}/x.md", f"S{i}", "2024-06-15") for i in range(4)] + [
(f"big/f{i:02d}.md", f"F{i}", "2024-06-15") for i in range(10)
]
_patch_catalog(monkeypatch, {"S": rows})
db = cast("Session", object())
# 4 subfolders (one doc each) + 10 files − the owning f00.md = 13
# entries; 8 shown (the 4 subfolders + f01..f04) + 5 more. The
# subfolder entries are source-relative (``big/s0/``) — exactly as
# the model's own ls output names them.
assert agent.suggested_folder_lines(db, [_doc("S", "big/f00.md")]) == [
"S/big/: big/s0/ (1 doc), big/s1/ (1 doc), big/s2/ (1 doc), big/s3/ (1 doc), "
"f01.md, f02.md, f03.md, f04.md +5 more"
]
# A wide enough cap shows every entry — no remainder marker.
assert agent.suggested_folder_lines(db, [_doc("S", "big/f00.md")], max_entries=20) == [
"S/big/: big/s0/ (1 doc), big/s1/ (1 doc), big/s2/ (1 doc), big/s3/ (1 doc), "
"f01.md, f02.md, f03.md, f04.md, f05.md, f06.md, f07.md, f08.md, f09.md"
]
# The remainder is exact at a tighter cap too (3 subfolders shown —
# files only follow ALL subfolders, the ls order — 10 left).
assert agent.suggested_folder_lines(db, [_doc("S", "big/f00.md")], max_entries=3) == [
"S/big/: big/s0/ (1 doc), big/s1/ (1 doc), big/s2/ (1 doc) +10 more"
]
def test_suggested_folder_lines_entry_order_matches_the_ls_listing(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The line's entry order IS the ``ls`` folder-level rendering order
(pinned against :func:`group_folder_listing` itself): the direct
subfolders first (path order), then the files (path order) — a
mixed folder where the file names and the subfolder names interleave
in path order still lists all subfolders before any file."""
rows = [
("a/a.md", "A", "2024-06-15"),
("a/b.md", "B", "2024-06-15"),
("a/c/d.md", "D", "2024-06-15"),
]
_patch_catalog(monkeypatch, {"Homelab": rows})
db = cast("Session", object())
suggested = _doc("Homelab", "a/b.md")
# The expected entry order, derived from group_folder_listing itself
# (subfolders → files, minus the owning doc) — the line must read
# the same as the model's own ls output.
sub, files, total = agent.group_folder_listing("Homelab", "a", rows, {})
assert total == 2
expected_entries = [
f"{g}/ ({c} {'doc' if c == 1 else 'docs'})" for g, c, _s in sub
] + [path.rsplit("/", 1)[-1] for _src, path, _t, _d in files if path != suggested.path]
# Subfolder source-relative (``a/c/``), files by relative filename.
assert expected_entries == ["a/c/ (1 doc)", "a.md"]
expected_line = f"Homelab/a/: {', '.join(expected_entries)}"
assert agent.suggested_folder_lines(db, [suggested]) == [expected_line]
def test_suggested_folder_lines_empty_suggested_yields_no_lines(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""No suggested docs → no lines (the caller then builds the
byte-identical phase-118 prompt) — and no fetcher call at all."""
calls: list[str] = []
def _rows(db: Any, source: str) -> list[tuple[str, str, str]]:
calls.append(source)
return []
monkeypatch.setattr(agent, "_source_document_rows", _rows)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
assert agent.suggested_folder_lines(cast("Session", object()), []) == []
assert calls == []
# ---------- the pinned drill-down templates (byte-for-byte) ----------
tests/unit/test_chat_cancel.py
+14 -1
View File
@@ -183,6 +183,13 @@ class _FakeSession:
return KbOverview(id=1, content="")
return None
def execute(self, *args: Any, **kwargs: Any) -> list[Any]:
# Phase 119 (D3): the endpoint's suggested-folder lines run the
# ls catalog fetchers on this session — the fake catalog is
# empty (header-only lines), keeping the prompt builds
# deterministic here.
return []
@pytest.fixture()
def env(monkeypatch: pytest.MonkeyPatch) -> Iterator[_FakeSession]:
@@ -411,10 +418,16 @@ def test_completed_turn_still_emits_done_and_writes_query_log(
frames = _frames(chunks)
assert [f["type"] for f in frames] == ["delta", "delta", "delta", "done"]
assert frames[-1]["deflected"] is False
assert frames[-1]["sources"][0]["title"] == "Kubernetes Homelab Cluster"
# Phase 119 (LOCKED A1): the citation surface is the agent's READ
# docs only — this turn's model stream never emits a tool call, so
# nothing was read and the grounded done frame chips nothing (the
# phase-118 "suggested + read" union is retired). The retrieval
# stays durably recorded (118-A3 stands).
assert frames[-1]["sources"] == []
(row,) = env.added
assert isinstance(row, QueryLog)
assert row.question == "How is my Kubernetes cluster set up?"
assert "kubernetes-homelab-cluster.md" in row.sources # durable record kept
assert env.commits == 1
# The per-turn line still goes out; no cancel line for a settled turn.
assert any("question=" in r.getMessage() for r in caplog.records)
tests/unit/test_chat_gate.py
+237 -14
View File
@@ -11,7 +11,7 @@ from __future__ import annotations
import hashlib
import json
import uuid
from collections.abc import Iterator
from collections.abc import Iterator, Sequence
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any
@@ -22,9 +22,10 @@ from app.api import chat as chat_api
from app.config import Settings
from app.main import app as fastapi_app
from app.models import Document, KbOverview, QueryLog
from app.rag import agent as rag_agent
from app.rag.agent import AGENT_TOOLS
from app.rag.llm import StreamPiece
from app.rag.prompts import build_deflect_prompt
from app.rag.llm import StreamPiece, ToolCallPiece
from app.rag.prompts import SUGGEST_INTRO, build_deflect_prompt
from app.rag.retriever import RetrievedChunk, weak_hit_titles
from app.rag.suggestions import MAX_SUGGESTIONS, derive_suggestions
from tests.conftest import ADMIN_PASSWORD
@@ -443,6 +444,51 @@ def test_plan_turn_high_seeds_top5_suggested_related_is_rank6plus() -> None:
assert f"FULL_CONTENT_SENTINEL_{i}" not in plan.system_prompt
def test_plan_turn_folder_lines_ride_the_high_prompt() -> None:
"""Phase 119 (D3, LOCKED A4): plan_turn passes *folder_lines* through
to the HIGH prompt — after ``SUGGEST_INTRO``, before the first
``<document>`` block (each on its own line)."""
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_CONTENT")
lines = ["Homelab/: kubernetes-homelab-cluster.md (2 docs)"]
plan = chat_api.plan_turn(
[_chunk(doc, 0.90)], _settings(threshold=0.30), folder_lines=lines
)
assert plan.deflected is False
prompt = plan.system_prompt
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n{lines[0]}\n\n"
def test_plan_turn_folder_lines_default_keeps_phase_118_high_prompt() -> None:
"""Omitted *folder_lines* (the default ``()``) ⇒ the HIGH prompt is
the phase-118 shape, byte-identical."""
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_CONTENT")
plan = chat_api.plan_turn([_chunk(doc, 0.90)], _settings(threshold=0.30))
assert plan.deflected is False
prompt = plan.system_prompt
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n\n"
def test_plan_turn_folder_lines_ignored_on_the_low_branch() -> None:
"""The LOW (deflected) branch IGNORES *folder_lines* — the
deflected prompt stays byte-identical (LOCKED A4)."""
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_CONTENT")
low_with = chat_api.plan_turn(
[_chunk(doc, 0.05, cosine=0.05)],
_settings(threshold=0.30),
folder_lines=["Homelab/: x.md"],
)
low_without = chat_api.plan_turn(
[_chunk(doc, 0.05, cosine=0.05)], _settings(threshold=0.30)
)
assert low_with.deflected is True
assert low_with.system_prompt == low_without.system_prompt
assert "Homelab/: x.md" not in low_with.system_prompt
def test_plan_turn_high_single_strong_doc_yields_one_suggested() -> None:
"""The suggested cap is a CEILING, not a quota: one doc ⇒ one
suggested doc, an empty related tier (nothing beyond rank 1)."""
@@ -458,10 +504,11 @@ def test_plan_turn_high_single_strong_doc_yields_one_suggested() -> None:
def test_plan_turn_high_suggests_strong_and_weak_no_floor() -> None:
"""The recurring incident under phase 118 (A3): the weak 2nd doc no
longer loses a citation slot to a bar — the floor never filters, so
BOTH docs are suggested (rank order) and ride the citation surface
(A4); the HIGH prompt seeds both summaries (the A5 fallback carries
the short fixture content whole)."""
longer loses a seeding slot to a bar — the floor never filters, so
BOTH docs are suggested (rank order) and ride the durable record
(118-A3 — the phase-119 A1 citation surface is the read docs only);
the HIGH prompt seeds both summaries (the A5 fallback carries the
short fixture content whole)."""
strong = _doc("Kubernetes Homelab Cluster", "STRONG_DOC_CONTENT")
weak = _doc("Backup Strategy", "WEAK_DOC_CONTENT")
chunks = [_chunk(strong, 0.90, cosine=0.80), _chunk(weak, 0.80, cosine=0.20)]
@@ -845,15 +892,21 @@ def test_suggestions_empty_input_yields_fallback_only() -> None:
class _CannedLLM:
"""Records the messages it is given; streams a canned answer.
Never emits tool calls, so a grounded turn through the phase-37 agent
loop ends after the single (tools-offered) request; *seen_tools*
records each request's ``tools`` value for the phase-37 wiring pins.
Without *read_paths* it never emits tool calls, so a grounded turn
through the phase-37 agent loop ends after the single (tools-offered)
request. With *read_paths*, the request whose conversation carries
*i* tool results (i < len(read_paths)) emits ``read(read_paths[i])``
(phase 119: the read-only done-sources pins drive multi-read turns),
and once the list is exhausted the request streams the answer.
*seen_tools* records each request's ``tools`` value for the phase-37
wiring pins.
"""
def __init__(self, answer: str = ANSWER) -> None:
def __init__(self, answer: str = ANSWER, read_paths: Sequence[str] = ()) -> None:
self.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
self.embed_batches = 0
self.answer = answer
self.read_paths = list(read_paths)
self.seen: list[list[dict[str, str]]] = []
self.seen_tools: list[list[dict[str, Any]] | None] = []
@@ -868,6 +921,15 @@ class _CannedLLM:
):
self.seen.append(messages)
self.seen_tools.append(tools)
if tools is not None and self.read_paths:
tool_results = sum(1 for m in messages if m.get("role") == "tool")
if tool_results < len(self.read_paths):
yield ToolCallPiece(
id=f"call_{tool_results + 1}",
name="read",
arguments={"path": self.read_paths[tool_results]},
)
return
for i in range(0, len(self.answer), 12):
yield StreamPiece("content", self.answer[i : i + 12])
@@ -913,6 +975,15 @@ class _FakeSession:
return KbOverview(id=1, content=self.kb_overview)
return None
def execute(self, *args: Any, **kwargs: Any) -> list[Any]:
"""Phase 119 (D3): the endpoint's suggested-folder lines run the
``ls`` catalog fetchers (``_source_document_rows`` /
``_source_folder_summaries``) on this session — the fake catalog
is empty, so each suggested doc's line is its header alone
(``Homelab/:``), keeping the prompt builds deterministic in
these gate tests."""
return []
@pytest.fixture(autouse=True)
def _admin_signed_in(client: TestClient) -> None:
@@ -972,6 +1043,16 @@ def _fake_retriever(chunks: list[RetrievedChunk]) -> Any:
return retrieve
def _find_docs(docs: dict[tuple[str, str], Document]) -> Any:
"""A fake ``find_document`` resolving *docs* by ``(source, path)``
(the agent module's hook — the fake sessions here have no catalog)."""
def find_document(_db: Any, source: str, path: str) -> Document | None:
return docs.get((source, path))
return find_document
def test_endpoint_just_below_threshold_deflects(
client: TestClient,
gate_env: tuple[_FakeSession, _CannedLLM],
@@ -1073,11 +1154,18 @@ def test_endpoint_deflected_turn_never_offers_tools(
assert "You may extend your context with three tools" not in system["content"]
def test_endpoint_score_at_threshold_answers(
def test_endpoint_score_at_threshold_answers_zero_read_grounded_turn_chips_nothing(
client: TestClient,
gate_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 119 (LOCKED A1 — the phase-118 "always chips" pin
re-targeted): a grounded turn on which the agent reads NOTHING (the
summary-only fast path — the canned LLM never emits a tool call)
chips nothing: ``done.sources`` is the READ DOCS only, so the
never-read suggested doc appears NOWHERE in it (the explicit contrast
against the retired A4 union). The durable record keeps the retrieval
(118-A3 stands)."""
session, llm = gate_env
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT")
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(doc, 0.30)]))
@@ -1088,17 +1176,152 @@ def test_endpoint_score_at_threshold_answers(
assert done["type"] == "done"
assert done["deflected"] is False
assert done["suggestions"] == []
assert done["sources"] and done["sources"][0]["title"] == "Kubernetes Homelab Cluster"
# A1: read docs only — nothing was read ⇒ no chips. The suggested
# doc (rank 1) appears NOWHERE in done.sources (the retired A4 union
# would have carried it).
assert done["sources"] == []
(system, _user) = llm.seen[0][0], llm.seen[0][1]
assert "<relevance>HIGH</relevance>" in system["content"]
assert "DEFLECT_MODE" not in system["content"]
assert "TALOS_DOC_SENT" in system["content"]
# The durable record still records the retrieval (118-A3 stands).
(row,) = session.added
assert isinstance(row, QueryLog)
assert row.deflected is False
assert row.top_score == pytest.approx(0.30)
assert "kubernetes-homelab-cluster.md" in row.sources
def test_endpoint_done_sources_are_the_read_docs_in_read_order(
client: TestClient,
gate_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 119 (LOCKED A1): the done frame's ``sources`` is exactly
the agent's READ documents — deduped by (source, path), READ order
(not the suggested tier's rank order). Suggested docs that were
never read appear NOWHERE in the frame (the contrast pin against
the retired A4 union); the durable record keeps the full retrieval
(118-A3 stands)."""
session, _llm = gate_env
a = _doc("Alpha", "ALPHA_DOC_CONTENT")
b = _doc("Beta", "BETA_DOC_CONTENT")
c = _doc("Gamma", "GAMMA_DOC_CONTENT")
chunks = [_chunk(a, 0.90), _chunk(b, 0.80), _chunk(c, 0.70)]
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever(chunks))
llm = _CannedLLM(read_paths=[f"Homelab/{b.path}", f"Homelab/{a.path}"])
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: llm
monkeypatch.setattr(
rag_agent,
"find_document",
_find_docs({(a.source, a.path): a, (b.source, b.path): b}),
)
frames = _ask(client, "Which of these documents do I have?")
done = frames[-1]
assert done["deflected"] is False
# Both reads streamed as tool frames (the agent loop ran).
reads = [f for f in frames if f["type"] == "tool"]
assert [f["argument"] for f in reads] == [
f"Homelab/{b.path}",
f"Homelab/{a.path}",
]
# A1: read docs only, read order — B first (it was read first), A
# second; the never-read suggested doc C is NOWHERE in the frame, and
# the suggested tier's rank order (A, B, C) is NOT the chip order.
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(b.source, b.path),
(a.source, a.path),
]
assert not any(s["path"] == c.path for s in done["sources"])
assert len(done["sources"]) == len({(s["source"], s["path"]) for s in done["sources"]})
assert done["related"] == [] # nothing beyond rank 3 for 3 docs
# The durable record keeps the full retrieval (118-A3 stands).
(row,) = session.added
assert isinstance(row, QueryLog)
for doc in (a, b, c):
assert f"Homelab/{doc.path}" in row.sources
def test_endpoint_read_related_doc_is_cited_not_related(
client: TestClient,
gate_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 119 (A1) × phase 118 (A3): a related-tier doc (rank 6+) that
the agent ``read``s is a CHIP (in ``done.sources``) and is EXCLUDED
from ``done.related`` — a "nearby doc" that was actually used must
not read as nearby (unchanged intent, keyed on read docs since
phase 119). The suggested docs, never read, are absent from the
frame; the other related doc stays in the tier."""
session, _llm = gate_env
docs = [_doc(f"Doc {i}", f"CONTENT_{i}") for i in range(7)]
chunks = [
_chunk(d, 0.9 - 0.1 * i, cosine=0.8 - 0.05 * i) for i, d in enumerate(docs)
]
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever(chunks))
read_doc = docs[5] # rank 6 — the related tier (rank 7 = docs[6])
llm = _CannedLLM(read_paths=[f"Homelab/{read_doc.path}"])
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: llm
monkeypatch.setattr(
rag_agent,
"find_document",
_find_docs({(read_doc.source, read_doc.path): read_doc}),
)
frames = _ask(client, "What does the sixth ranked document say?")
done = frames[-1]
assert done["deflected"] is False
# A1: read docs only — the read related doc is the ONLY chip (the
# never-read suggested docs are nowhere in the frame).
assert [(s["source"], s["path"]) for s in done["sources"]] == [
(read_doc.source, read_doc.path)
]
# The read related doc is NOT "nearby"; the other related doc is.
related = [(s["source"], s["path"]) for s in done["related"]]
assert related == [(docs[6].source, docs[6].path)]
assert not any((d.source, d.path) in related for d in docs[:5])
# The durable record keeps the full retrieval (118-A3 stands).
(row,) = session.added
assert isinstance(row, QueryLog)
assert row.deflected is False
for d in docs:
assert f"Homelab/{d.path}" in row.sources
def test_endpoint_suggested_folder_lines_reach_the_high_prompt(
client: TestClient,
gate_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 119 (D3, LOCKED A4): the endpoint computes the
suggested-folder lines BEFORE plan_turn (the same deterministic
suggested walk, over the short-lived step session) and they ride
the grounded HIGH prompt — after ``SUGGEST_INTRO``, before the
first ``<document>`` block."""
_session, llm = gate_env
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT")
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever([_chunk(doc, 0.30)]))
line = "Homelab/: kubernetes-homelab-cluster.md (2 docs)"
monkeypatch.setattr(
chat_api,
"suggested_folder_lines",
lambda db, suggested: [line],
)
frames = _ask(client, "How is my Kubernetes cluster set up?")
assert frames[-1]["deflected"] is False
(system, _user) = llm.seen[0][0], llm.seen[0][1]
i_open = system["content"].index("<documents>")
i_block = system["content"].index("<document ")
assert (
system["content"][i_open:i_block]
== f"<documents>\n{SUGGEST_INTRO}\n{line}\n\n"
)
# ---------- endpoint: KB overview row (phase 31) ----------
tests/unit/test_config.py
+31
View File
@@ -41,6 +41,10 @@ def test_defaults_match_locked_decisions(monkeypatch: pytest.MonkeyPatch) -> Non
# fine-line-tuned value, task 07) with a 365-day decay timescale.
assert s.recency_boost == 0.0007
assert s.recency_half_life_days == 365
# Phase 119, D2 (LOCKED A3): the bounded name-hit bonus on the
# selection-time document score is ON by default (0.005 — the
# owner-tunable starting point; ``0`` is the kill switch).
assert s.name_hit_bonus == 0.005
# Owner instruction 2026-08-22: answers may run up to 32 768 tokens.
assert s.max_output_tokens == 32_768
# Phase 17: the model's thinking streams by default (kill-switch off).
@@ -391,6 +395,33 @@ def test_recency_boost_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> None
_settings()
def test_name_hit_bonus_default_and_env_override(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 119, D2 (LOCKED A3): the bounded name-hit bonus on the
SELECTION-time document score is ON by default (0.005 — the
owner-tunable starting point, not a calibrated constant) and
env-tunable so the owner re-tunes live (the phase-106 recency-boost
precedent). ``0`` is legal — the byte-identical kill switch."""
monkeypatch.delenv("BOR_NAME_HIT_BONUS", raising=False)
s = _settings()
assert s.name_hit_bonus == 0.005
monkeypatch.setenv("BOR_NAME_HIT_BONUS", "0")
assert _settings().name_hit_bonus == 0.0
monkeypatch.setenv("BOR_NAME_HIT_BONUS", "0.01")
assert _settings().name_hit_bonus == 0.01
def test_name_hit_bonus_rejects_negative(monkeypatch: pytest.MonkeyPatch) -> None:
"""``0`` is the byte-identical kill switch (the pre-phase selection
order) — a NEGATIVE bonus would demote name-hit documents (the exact
opposite of D2), so the validator fails loudly at startup naming the
field (the ``agent_max_rounds`` pattern, phase 119)."""
monkeypatch.setenv("BOR_NAME_HIT_BONUS", "-0.001")
with pytest.raises(ValidationError, match="name_hit_bonus"):
_settings()
def test_recency_half_life_rejects_non_positive(
monkeypatch: pytest.MonkeyPatch,
) -> None:
tests/unit/test_embed_question_length.py
+7
View File
@@ -173,6 +173,13 @@ class _FakeSession:
return None
return None
def execute(self, *args: Any, **kwargs: Any) -> list[Any]:
# Phase 119 (D3): the endpoint's suggested-folder lines run the
# ls catalog fetchers on this session — the fake catalog is
# empty (header-only lines), keeping the prompt builds
# deterministic here.
return []
def _doc(title: str, content: str) -> Document:
return Document(
tests/unit/test_prompt_lock.py
+35 -15
View File
@@ -148,16 +148,27 @@ def test_low_prompt_build_byte_identical_to_pre_phase() -> None:
assert "SUGGEST_INTRO" not in prompt and "<documents>" not in prompt
# ---------- SUGGEST_INTRO (phase 118, task 03 — the start-here framing) ----------
# ---------- SUGGEST_INTRO (phase 118 framing, re-cut for phase 119, LOCKED A5) ----------
#: Phase-118 anchors for ``SUGGEST_INTRO`` — the ``<documents>``
#: section's intro line (the owner's "start here if these summaries seem
#: right to you" framing, TODO L3). The E2E mock's ``_document_block``
#: parser is regex-based over the block markup (which stays byte-stable
#: around the intro), so this constant is a prompt-copy lock, pinned the
#: way ``TOOLS_SECTION`` is: sha256 + prefix + total length.
SUGGEST_INTRO_SHA256 = "7b14d2dedc6ebc4e440d32dd1edb979a7461c94034b9541c9f37b9042f394d3a"
SUGGEST_INTRO_LEN = 323
#: Anchors for ``SUGGEST_INTRO`` — the ``<documents>`` section's intro
#: line (the owner's "start here if these summaries seem right to you"
#: framing, TODO L3, phase 118). RE-CUT for phase 119 (task 04, LOCKED
#: A5, owner directive 2026-09-16): the final sentence was replaced
#: with the cite-discipline sentence (cite only the documents read — or
#: the suggested document answered from without reading — never a
#: document neither read nor used), closing the live confabulation in
#: which the model cited a file it never read; the phase-118 "cite the
#: document(s) you used, by path" sentence is retired. Only that
#: sentence moved — the start-here framing, the ``read`` pointer, and
#: the full-text clause survive byte-identical, so the prefix anchor is
#: the same as pre-phase-119. The E2E mock's ``_document_block`` parser
#: is regex-based over the block markup (which stays byte-stable around
#: the intro), so this constant is a prompt-copy lock, pinned the way
#: ``TOOLS_SECTION`` is: sha256 + prefix + total length. ``PERSONA``,
#: ``TOOLS_SECTION``, and the LOW body anchors above stay byte-identical
#: — this change touches no other constant.
SUGGEST_INTRO_SHA256 = "c6ffbb19975cbf184910a9a8e8fdea101c16f5d3eecb872fb07afeabd2763302"
SUGGEST_INTRO_LEN = 460
SUGGEST_INTRO_PREFIX = (
"The blocks below are the summaries of the top-ranked documents for "
"your question — start here if one seems right to you: "
@@ -165,14 +176,23 @@ SUGGEST_INTRO_PREFIX = (
def test_suggest_intro_byte_locked() -> None:
"""The start-here framing is LOCKED copy (phase 118): sha256 + exact
prefix + total length; the three contracts it must carry (summaries
are the starting points; ``read`` adds the full text, which is NOT
in the prompt until read; cite by path) are pinned as substrings."""
"""The start-here framing is LOCKED copy (phase 118; the final
sentence re-cut by phase 119, LOCKED A5): sha256 + exact prefix +
total length; the four contracts it must carry (summaries are the
starting points; ``read`` adds the full text, which is NOT in the
prompt until read; the phase-119 cite discipline — cite only what
was read, or the suggested document answered from without reading,
never a document neither read nor used) are pinned as substrings."""
assert len(SUGGEST_INTRO) == SUGGEST_INTRO_LEN
assert _sha256(SUGGEST_INTRO) == SUGGEST_INTRO_SHA256
assert SUGGEST_INTRO.startswith(SUGGEST_INTRO_PREFIX)
assert "call `read`" in SUGGEST_INTRO
assert "call `read" in SUGGEST_INTRO
assert "combined `source/path`" in SUGGEST_INTRO
assert "its full text is not in the prompt until you read it" in SUGGEST_INTRO
assert "Cite the document(s) you used, by path." in SUGGEST_INTRO
assert (
"Cite only the document(s) you read — or, if you answered from a "
"suggested summary without reading it, cite that suggested "
"document — never a document you neither read nor used."
) in SUGGEST_INTRO
# The retired phase-118 final sentence is gone from the constant.
assert "Cite the document(s) you used, by path." not in SUGGEST_INTRO
tests/unit/test_prompts.py
+156
View File
@@ -24,6 +24,13 @@ document is refused — while the ``ls``/``grep`` clauses and the
discipline rules stay byte-identical): the teaching refusals in
:mod:`app.rag.agent` re-state the same contract; the ``<tools>``
marker keying (HIGH only) is unchanged.
And the phase-119 cite discipline (task 04, LOCKED A5): the intro's
final sentence ("cite the document(s) you used, by path", phase 118)
is REPLACED — the HIGH prompt carries the discipline sentence exactly
once, inside ``<documents>`` after the intro's start-here framing;
the LOW prompt and ``PERSONA`` / ``TOOLS_SECTION`` stay byte-identical
(the full sha pins live in :mod:`tests.unit.test_prompt_lock`).
"""
from __future__ import annotations
@@ -447,6 +454,155 @@ def test_documents_section_without_blocks_has_no_intro() -> None:
)
# ---------- phase 119 (D3, LOCKED A4): the suggested-folder lines ----------
def test_high_prompt_folder_lines_after_intro_before_first_block() -> None:
"""The folder lines ride the ``<documents>`` section immediately
AFTER the ``SUGGEST_INTRO`` line — each on its own line — then a
blank line, then the first ``<document>`` block. Plain lines: no
new markup/tag anywhere (the E2E mock keys off the ``<documents>``
marker and the LAST block's tail)."""
doc = _doc(
"deploy/Deployments/reeseapps/gitea/app/gitea-web.env.j2",
"FULL_CONTENT_SENTINEL_119",
"Gitea Web Env",
summary="Gitea web env file.",
)
lines = [
"Homelab/deploy/Deployments/reeseapps/gitea/: app/ (5 docs), README.md",
"Homelab/deploy/Deployments/reeseapps/: gateway/ (3 docs), gitea/ (8 docs)",
]
prompt = build_high_prompt([doc], folder_lines=lines)
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n{lines[0]}\n{lines[1]}\n\n"
# Each line occurs exactly once, inside the section, before the
# first block (never after a summary — the E2E tail echo is safe).
for line in lines:
assert prompt.count(line) == 1
assert prompt.index("<documents>") < prompt.index(line) < prompt.index("<document ")
assert prompt.index(line) < prompt.index("</documents>")
# The <document> block markup AND body stay byte-identical: from the
# first block on, the prompt equals the no-folder-line build.
plain = build_high_prompt([doc])
assert prompt[prompt.index("<document ") :] == plain[plain.index("<document ") :]
# No new markup: the section still opens/closes exactly once (the
# ``<documents>`` MENTION in the TOOLS_SECTION copy is plain text —
# count the tag + newline, not the bare substring).
assert prompt.count("<documents>\n") == 1
assert prompt.count("</documents>") == 1
assert prompt.count("<document ") == 1
def test_high_prompt_single_folder_line_exact_shape() -> None:
"""One folder line: the exact slice between the section open and the
first block is ``<documents>\n`` + intro + ``\n`` + line +
``\n\n``."""
doc = _doc("a.md", "CONTENT", "Title A", summary="Summary A.")
line = "Homelab/: a.md (2 docs), b/ (1 doc)"
prompt = build_high_prompt([doc], folder_lines=[line])
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n{line}\n\n"
def test_high_prompt_empty_folder_lines_byte_identical_to_phase_118() -> None:
"""LOCKED A4: empty ``folder_lines`` (the default) ⇒ the phase-118
build, byte-identical — the existing pins keep passing and this
pins the default explicitly (both an explicit ``()`` and the
omitted parameter)."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
block = (
'<document source="Homelab" path="kubernetes.md" '
'title="Kubernetes Homelab Cluster" date="2024-06-15">\n'
"Talos Linux on three nodes.\n"
"</document>"
)
expected = (
_base("HIGH")
+ "\n<documents>\n"
+ SUGGEST_INTRO
+ "\n\n"
+ block
+ "\n</documents>"
+ "\n"
+ TOOLS_SECTION
)
# An explicit empty tuple AND the omitted parameter (the default):
assert build_high_prompt([doc], folder_lines=()) == expected
assert build_high_prompt([doc]) == expected
# The LOW (deflected) prompt has no folder_lines parameter at all —
# the deflected build stays byte-identical (the phase-118 pin in
# test_zero_note_prompt_is_byte_identical_to_pre_steering stands).
def test_folder_lines_ride_only_on_present_blocks() -> None:
"""Like the intro, the folder lines ride on present blocks ONLY: an
empty ``<documents>`` section is unchanged (no lines, no blocks)."""
prompt = build_high_prompt([], folder_lines=["X/: y.md"])
assert prompt == build_high_prompt([])
assert "X/: y.md" not in prompt
# ---------- phase 119 (D4, LOCKED A5): the cite-discipline sentence ----------
#: The LOCKED A5 sentence (phase 119, task 04) — the exact replacement
#: for the retired phase-118 final sentence of :data:`SUGGEST_INTRO`
#: ("Cite the document(s) you used, by path."). Closes the live
#: confabulation: the answer's "Docs used:" line cited a file the
#: agent never read.
CITE_DISCIPLINE = (
"Cite only the document(s) you read — or, if you answered from a "
"suggested summary without reading it, cite that suggested "
"document — never a document you neither read nor used."
)
def test_high_prompt_carries_cite_discipline_exactly_once_in_documents() -> None:
"""LOCKED A5: the HIGH prompt carries the discipline sentence
EXACTLY ONCE — it IS the intro's final sentence, so it sits inside
``<documents>``, AFTER the intro's start-here framing and BEFORE
the first block (both the plain build and the folder-line build;
the sentence rides the intro line, which never moves)."""
doc = _doc(
"kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster", summary="K8S"
)
assert CITE_DISCIPLINE in SUGGEST_INTRO
assert SUGGEST_INTRO.endswith(CITE_DISCIPLINE) # it is the final sentence
for high in (build_high_prompt([doc]), build_high_prompt([doc], folder_lines=["X/: y.md"])):
assert high.count(CITE_DISCIPLINE) == 1
i_docs = high.index("<documents>")
i_sentence = high.index(CITE_DISCIPLINE)
i_block = high.index("<document ")
i_close = high.index("</documents>")
assert i_docs < i_sentence < i_close
assert i_sentence > high.index("start here if one seems right")
assert i_sentence < i_block # before the first block, never after a summary
# The retired phase-118 sentence is gone from the prompt (it is
# gone from the constant — pinned in test_prompt_lock too).
high = build_high_prompt([doc])
assert "Cite the document(s) you used, by path." not in high
# The sentence rides on present blocks ONLY: an empty ``<documents>``
# section (no intro) carries none of it.
assert CITE_DISCIPLINE not in build_high_prompt([])
def test_cite_discipline_absent_from_low_prompt() -> None:
"""LOCKED A5: the discipline sentence belongs to ``SUGGEST_INTRO``
(the HIGH path) — it never leaks into the LOW (deflected) prompt,
whose byte-identity is pinned separately (the LOW anchors in
``test_prompt_lock`` pass unchanged)."""
for prompt in (
build_deflect_prompt(["T1", "T2"]),
build_deflect_prompt(["T1"], notes=["be concise"], kb_overview=OVERVIEW),
build_deflect_prompt([]),
):
assert CITE_DISCIPLINE not in prompt
assert "Cite only the document(s) you read" not in prompt
assert "Cite the document(s) you used, by path." not in prompt
def test_tools_section_old_names_and_budget_copy_gone() -> None:
"""The phase-37/68 tool names and the phase-37 per-tool budget line
(phase 45: the round cap is the bound — the prompt does not
tests/unit/test_retriever.py
+449 -45
View File
@@ -721,14 +721,15 @@ def _lexical_row(is_summary: bool, doc_path: str) -> object:
def test_lexical_candidates_carry_is_summary_flag() -> None:
"""The lexical list reads ``c.is_summary`` from the raw row.
The question carries no digit-bearing name token (no bare, no
numeric-join), so the name-hit path issues NO queries at all — the
single FTS rowset answers the only (FTS) call, and the list is the
plain FTS rows: the pre-name-hit behavior, unchanged.
The question's tokens are name candidates (class-agnostic, phase
119), so the name-hit projection runs — but the (empty) catalog
yields no path match, the LATERAL fetch is skipped, and the FTS
rowset answers the second call: the list is the plain FTS rows,
every one ``name_hit=False``.
"""
rows = [_lexical_row(True, "summary-src.yaml"), _lexical_row(False, "other.md")]
out = _lexical_candidates(
_FakeSession(rows), "how do i configure the thing", limit=10 # pyright: ignore[reportArgumentType]
_FakeSession([], rows), "how do i configure the thing", limit=10 # pyright: ignore[reportArgumentType]
)
assert len(out) == 2
by_path = {rc.document.path: rc for rc in out}
@@ -736,6 +737,7 @@ def test_lexical_candidates_carry_is_summary_flag() -> None:
assert by_path["summary-src.yaml"].position == -1
assert by_path["other.md"].is_summary is False
assert all(rc.fts_hit is True for rc in out)
assert all(rc.name_hit is False for rc in out) # ordinary FTS rows
def test_fuse_keeps_is_summary_on_double_hit() -> None:
@@ -783,30 +785,54 @@ def test_normalize_name() -> None:
def test_name_hit_tokens_incident_question() -> None:
"""The incident question yields EXACTLY the versioned join
``qwen38`` — the token the document names actually carry. Plain
prose words (``what``, ``llamacpp``, ``arguments``, ``server`` —
no digit) never name-match (the precision guard); the single
digits ("3", "8") and the bare "38" are < 4 chars; the
digit-leading ``38show`` boundary artifact is dropped."""
"""Phase 119 (LOCKED A2): the candidate list is CLASS-AGNOSTIC —
every normalized token of length >= 4 (dotted kept whole:
``llama.cpp`` → ``llamacpp``) plus the versioned join ``qwen38``.
The digit distinction moved to the match side (:func:`_name_hit_chunks`) —
prose precision now comes from the match class (a digitless token
must EQUAL a whole path component). The single digits ("3", "8")
and the bare "38" are < 4 chars; the digit-leading ``38show``
boundary artifact cannot survive (the join only fires on a purely
numeric SECOND token)."""
tokens = name_hit_tokens(INCIDENT_QUESTION)
assert tokens == ["qwen38"]
for absent in ("what", "qwen", "llamacpp", "arguments", "3", "8", "38", "38show", "server"):
assert tokens == ["what", "correct", "llamacpp", "arguments", "qwen", "qwen38"]
for absent in ("3", "8", "38", "38show", "server"):
assert absent not in tokens
def test_name_hit_tokens_no_digit_question_returns_empty() -> None:
"""A question with no digit-bearing token (bare or joined) yields
no name candidates — prose joins like ``correctllama`` never count."""
assert name_hit_tokens("what is the correct caddy config") == []
def test_name_hit_tokens_digitless_question_yields_long_tokens() -> None:
"""A question with NO digit-bearing token still yields candidates
(every normalized token of length >= 4) — the 2026-09-16 fix:
product names without digits ("gitea", "gateway") must get a name
signal. Prose joins (``correctcaddy``) never count (the second
token is not purely numeric)."""
assert name_hit_tokens("what is the correct caddy config") == [
"what", "correct", "caddy", "config",
]
assert name_hit_tokens("a e i o u 3 8") == []
def test_name_hit_tokens_bare_digit_bearing_token() -> None:
"""A single written token that carries a digit (``1panel``) is a
name candidate on its own — no join needed."""
name candidate on its own — no join needed — alongside the plain
prose tokens of the same question (class-agnostic list)."""
tokens = name_hit_tokens("what is my 1panel dashboard setup")
assert tokens == ["1panel"]
assert tokens == ["what", "1panel", "dashboard", "setup"]
def test_name_hit_tokens_versioned_join_and_short_tokens() -> None:
"""The versioned join survives the class-agnostic change ("Qwen
3.8" → ``qwen38``), and short tokens (< :data:`NAME_TOKEN_MIN_LEN`
normalized — the single digits, "3.8" → ``38``) never become
candidates, with or without a join."""
tokens = name_hit_tokens("help me with Qwen 3.8 please")
# The join is appended at its FIRST token's position (after "qwen").
assert tokens == ["help", "with", "qwen", "qwen38", "please"]
assert name_hit_tokens("3.8 8 16 9") == [] # 38 / 8 / 16 / 9 / 816 / 169 all < 4
# Word-after-version: the word itself is a candidate, but the
# digit-leading join artifact ("38show") cannot survive (the join
# only fires on a purely numeric SECOND token).
assert name_hit_tokens("3.8 show") == ["show"]
def _name_row(doc: Document) -> tuple:
@@ -834,7 +860,8 @@ def _name_hit_lateral_row(doc: Document, is_summary: bool = False) -> SimpleName
def test_name_hit_chunks_no_tokens_skips_all_queries() -> None:
"""A question with no name tokens issues no queries at all."""
"""A question with no name tokens (every normalized token < 4)
issues no queries at all."""
session = _FakeSession([]) # any call would surface a statement
assert _name_hit_chunks(session, "a e i o u 3 8") == [] # pyright: ignore[reportArgumentType]
assert session.statements == []
@@ -850,46 +877,124 @@ def test_name_hit_chunks_no_matching_doc_returns_empty() -> None:
assert len(session.statements) == 1 # projection only — no LATERAL fetch
def test_name_hit_chunks_ranked_by_count_length_catalog() -> None:
"""A two-candidate question (``qwen38`` + ``1panel``): the document
whose name carries BOTH (2 matches, 12 total chars) leads; the two
single-match documents tie on (1, 6) and fall to catalog order
(``dashboards/1panel-notes.md`` before ``quadlets/qwen3.8…``).
Hits carry ``fts_hit=True`` (the A8 gate answers), ``cosine=0.0``,
and the summary flag of their representative chunk."""
both = _doc("dashboards/1panel-qwen3.8.md", "body", title="1Panel Qwen 3.8")
panel = _doc("dashboards/1panel-notes.md", "body")
q38 = _doc("quadlets/qwen3.8-27b-juggernaut-vulkan.container", "body")
name_rows = [_name_row(d) for d in (panel, both, q38)] # catalog order
question = "what are the correct llama.cpp arguments for qwen 3.8 and the 1panel dashboard?"
def test_name_hit_chunks_digitless_exact_part_stem_subcomponent() -> None:
"""A DIGITLESS token EQUALS a normalized path part (the ``gitea/``
folder), the file stem (``gitea.md``), or a stem sub-component
(``kubernetes_gitea``, ``gitea-values``, ``test-gateway`` — the
stem split on non-alphanumeric runs) — the 2026-09-16 product-name
signal (LOCKED A2)."""
question = "how do i set up gitea or the gateway" # tokens: [gitea, gateway]
docs = [
_doc("deploy/reeseapps/gitea/README.md", "body"), # path part
_doc("notes/gitea.md", "body"), # file stem
_doc("deploy/k8s/kubernetes_gitea.md", "body"), # sub-component
_doc("deploy/k8s/gitea-values.yaml", "body"), # sub-component
_doc("deploy/istio/test-gateway.yaml", "body"), # sub-component (gateway)
_doc("notes/gitlab.md", "body"), # NO component matches — excluded
]
name_rows = [_name_row(d) for d in docs]
session = _FakeSession(name_rows, [_name_hit_lateral_row(d) for d in docs[:5]])
out = _name_hit_chunks(session, question) # pyright: ignore[reportArgumentType]
# Five one-token hits, catalog order (source, path):
assert [rc.document.path for rc in out] == [
"deploy/istio/test-gateway.yaml",
"deploy/k8s/gitea-values.yaml",
"deploy/k8s/kubernetes_gitea.md",
"deploy/reeseapps/gitea/README.md",
"notes/gitea.md",
]
assert all(rc.name_hit is True for rc in out)
assert all(rc.fts_hit is True for rc in out) # the lexical signal
assert all(rc.cosine == 0.0 for rc in out) # no vector rank
def test_name_hit_chunks_digitless_title_never_matched() -> None:
"""The owner-verified failure mode of the naive relaxation: a doc
under a ``Deployments/`` folder titled "Deployments" does NOT hit
the common token ``deploy`` (the part normalizes to
``deployments`` ≠ ``deploy``), and a doc titled "Gitea" with no
gitea path component does NOT hit ``gitea`` — TITLES ARE NEVER
MATCHED (LOCKED A2)."""
question = "how do i deploy gitea" # tokens: [deploy, gitea]
docs = [
_doc("Deployments/reeseapps/README.md", "body", title="Deployments"),
_doc("notes/internal-notes.md", "body", title="Gitea"), # title only
]
session = _FakeSession([_name_row(d) for d in docs], [])
assert _name_hit_chunks(session, question) == [] # pyright: ignore[reportArgumentType]
assert len(session.statements) == 1 # projection only — no LATERAL fetch
def test_name_hit_chunks_digit_bearing_prefix_not_midword() -> None:
"""A DIGIT-BEARING token is a PREFIX of a normalized part or stem
(``qwen38`` → ``qwen3.8-27b-epic-vulkan.container``) — a stem that
merely CONTAINS the token mid-word (``xqwen38y…``) does NOT hit;
sub-components are in the exact-match class only (LOCKED A2)."""
question = "what are the arguments for qwen 3.8" # tokens: what, arguments, qwen, qwen38
hit = _doc("quadlets/qwen3.8-27b-epic-vulkan.container", "body")
miss = _doc("quadlets/xqwen38y-test.container", "body") # mid-word containment
name_rows = [_name_row(hit), _name_row(miss)]
session = _FakeSession(name_rows, [_name_hit_lateral_row(hit)])
out = _name_hit_chunks(session, question) # pyright: ignore[reportArgumentType]
assert [rc.document.path for rc in out] == [hit.path]
assert out[0].name_hit is True
def test_name_hit_chunks_ranked_by_count_then_catalog() -> None:
"""A question (``deploy`` + ``gitea`` + ``qwen`` + ``qwen38``):
the document whose path carries BOTH a digitless component and a
digit-bearing prefix (2 matched tokens) leads; the two
single-token documents tie on count and fall to CATALOG ORDER —
the old total-matched-length tie-break is RETIRED (it would have
put the 6-char ``qwen38`` hit, ``quadlets/…``, before the 5-char
``gitea`` hit, ``gitea/notes.md`` — the flip is pinned). The
"Deployments"-titled doc and the title-only "Gitea" doc never
appear (titles are never matched)."""
precision = _doc("Deployments/reeseapps/README.md", "body", title="Deployments")
gitea_notes = _doc("gitea/notes.md", "body", title="Internal notes")
both = _doc("gitea/qwen3.8-model.container", "body", title="The model quadlet")
q38 = _doc("quadlets/qwen3.8-27b-juggernaut-vulkan.container", "body", title="juggernaut")
title_only = _doc("notes/internal-notes.md", "body", title="Gitea")
name_rows = [_name_row(d) for d in (precision, gitea_notes, both, q38, title_only)]
question = "how do i deploy gitea with qwen 3.8"
lateral_rows = [
_name_hit_lateral_row(q38, is_summary=True), # LATERAL may return any order
_name_hit_lateral_row(both),
_name_hit_lateral_row(panel),
_name_hit_lateral_row(gitea_notes),
]
session = _FakeSession(name_rows, lateral_rows)
out = _name_hit_chunks(session, question) # pyright: ignore[reportArgumentType]
assert [rc.document.path for rc in out] == [
"dashboards/1panel-qwen3.8.md", # 2 matched tokens — leads
"dashboards/1panel-notes.md", # (1, 6) — catalog order
"quadlets/qwen3.8-27b-juggernaut-vulkan.container", # (1, 6) — after
"gitea/qwen3.8-model.container", # 2 matched tokens (gitea + qwen38) — leads
"gitea/notes.md", # 1 token (gitea, 5 chars) — catalog order beats quadlets
"quadlets/qwen3.8-27b-juggernaut-vulkan.container", # 1 token (qwen38, 6 chars)
]
assert all(rc.name_hit is True for rc in out)
assert all(rc.fts_hit is True for rc in out) # the lexical signal
assert all(rc.cosine == 0.0 for rc in out) # no vector rank
assert all(rc.score == 0.0 for rc in out) # fuse fills the score
by_path = {rc.document.path: rc for rc in out}
# The representative chunk keeps its summary flag (the LATERAL
# choice: is_summary DESC, position ASC — chunk 0 otherwise).
assert by_path["quadlets/qwen3.8-27b-juggernaut-vulkan.container"].is_summary is True
assert by_path["quadlets/qwen3.8-27b-juggernaut-vulkan.container"].position == -1
assert by_path["dashboards/1panel-notes.md"].is_summary is False
assert by_path["gitea/notes.md"].is_summary is False
def test_name_hit_chunks_short_tokens_never_hit() -> None:
"""Short tokens (< 4 normalized — "3.8" → ``38``, the single
digits) are never candidates, so they can never hit, with or
without the versioned join."""
session = _FakeSession([]) # any call would surface a statement
assert _name_hit_chunks(session, "3.8 8 16 9") == [] # pyright: ignore[reportArgumentType]
assert session.statements == []
def test_name_hit_chunks_capped_at_limit() -> None:
"""Twelve tied name hits (one matched token each) yield exactly
``NAME_HIT_LIMIT`` of them — catalog order (the deterministic
tie-break)."""
docs = [_doc(f"quadlets/m{i:02d}.container", "body") for i in range(12)]
for d in docs: # give every document a name that carries the token
d.title = "qwen38 model i"
"""Twelve tied name hits (one matched token each — the ``qwen38``
stem prefix) yield exactly ``NAME_HIT_LIMIT`` of them — catalog
order (the deterministic tie-break)."""
docs = [_doc(f"quadlets/qwen3.8-m{i:02d}.container", "body") for i in range(12)]
name_rows = [_name_row(d) for d in docs]
# Only the ten winners (catalog order — the deterministic tie-break
# of the twelve identical scores) reach the LATERAL fetch; the fake
@@ -900,7 +1005,10 @@ def test_name_hit_chunks_capped_at_limit() -> None:
session, "tell me about the qwen 3.8 models" # pyright: ignore[reportArgumentType]
)
assert len(out) == NAME_HIT_LIMIT
assert [rc.document.path for rc in out] == [f"quadlets/m{i:02d}.container" for i in range(10)]
assert [rc.document.path for rc in out] == [
f"quadlets/qwen3.8-m{i:02d}.container" for i in range(10)
]
assert all(rc.name_hit is True for rc in out)
def test_lexical_candidates_name_hits_lead_and_dedupe_with_fts() -> None:
@@ -933,7 +1041,7 @@ def test_lexical_candidates_name_hits_lead_and_dedupe_with_fts() -> None:
session = _FakeSession(name_rows, lateral_rows, fts_rows)
out = _lexical_candidates(session, INCIDENT_QUESTION, limit=10) # pyright: ignore[reportArgumentType]
assert len(out) == 3 # q38 (once), other (name hit), other (FTS chunk)
# Both name hits tie on (1, 6) — catalog order: "qwen3." (ASCII 46)
# Both name hits tie on count (1) — catalog order: "qwen3." (ASCII 46)
# sorts before "qwen38" (ASCII 56).
assert out[0].document.path == "quadlets/qwen3.8-27b-juggernaut-vulkan.container"
assert out[1].document.path == "quadlets/qwen38-other.container"
@@ -942,3 +1050,299 @@ def test_lexical_candidates_name_hits_lead_and_dedupe_with_fts() -> None:
rc.chunk_id for rc in out
} == {q38_chunk, fts_rows[1].chunk_id, lateral_rows[1].chunk_id}
assert all(rc.fts_hit is True for rc in out)
# Phase 119: the name-hit representative rows are flagged, the plain
# FTS row is not (the selection tier's bonus input, task 02).
assert out[0].name_hit is True
assert out[1].name_hit is True
assert out[2].name_hit is False
# ---------------------------------------------------------------------------
# Phase 119, D1 — the name_hit flag through fusion
# ---------------------------------------------------------------------------
def test_fuse_keeps_name_hit_on_lexical_only_hit() -> None:
"""A name-hit row with no vector rank keeps ``name_hit=True``
through the fusion (the ``replace()`` copy carries the field)."""
nh = _rc("gitea/README.md")
nh.name_hit = True
out = fuse([], [nh], k=60)
assert len(out) == 1
assert out[0].name_hit is True
assert out[0].fts_hit is True
assert out[0].cosine == 0.0
def test_fuse_or_s_name_hit_on_double_hit() -> None:
"""A vector row that is ALSO the name hit's representative chunk
(the RRF merge dedupes by chunk id) keeps ``name_hit=True`` — the
merge ORs the flag in, so the selection tier (task 02) still sees
the name hit on the fused list."""
v = _rc("gitea/README.md", cosine=0.9)
l1 = _rc("gitea/README.md", cosine=0.1) # the lexical copy of the same chunk
l1.chunk_id = v.chunk_id
l1.name_hit = True
out = fuse([v], [l1], k=60)
assert len(out) == 1
assert out[0].name_hit is True
assert out[0].fts_hit is True
assert out[0].score == pytest.approx(2 / 61)
def test_fuse_default_name_hit_stays_false_for_ordinary_rows() -> None:
"""Neither list flagged ⇒ fusion never invents a name-hit flag —
ordinary vector and FTS rows are ``name_hit=False``."""
out = fuse([_rc("a.md", cosine=0.8)], [_rc("b.md", fts_hit=True)], k=60)
assert len(out) == 2
assert all(rc.name_hit is False for rc in out)
# ---------------------------------------------------------------------------
# Phase 119, D2 — the bounded name-hit bonus (LOCKED A3)
# ---------------------------------------------------------------------------
from app.rag.retriever import _selection_order, weak_hit_titles # noqa: E402
def _bonus_rc(
doc: Document,
score: float,
cosine: float,
position: int = 0,
name_hit: bool = False,
) -> RetrievedChunk:
"""One fused-list candidate (name-hit rows follow the D1 lexical
convention: ``cosine=0.0``, ``fts_hit=True``)."""
return RetrievedChunk(
chunk_id=uuid.uuid4(),
position=position,
content=doc.content[:20],
score=score,
document=doc,
cosine=cosine,
fts_hit=cosine == 0.0,
name_hit=name_hit,
)
def _bonus_chunks() -> list[RetrievedChunk]:
"""A mixed FUSED list — already in the ``fuse()`` key order
(−score, −cosine, path, position) — with one name-hit document
(``gitea/README.md``, the D1 convention: cosine 0.0) and ordinary
vector/FTS documents: ``a.md`` carries two chunks, and ``b.md`` /
``c.md`` tie on the fused score (separated only by cosine).
The pre-phase (bonus-0) document order this list walks — the golden
the kill switch must reproduce — is a (0.0200) → gitea (0.0160) →
b (0.0150, cos 0.7) → c (0.0150, cos 0.6) → d (0.0100).
"""
gitea = _doc("gitea/README.md", "G" * 50)
a = _doc("a.md", "A" * 50)
b = _doc("b.md", "B" * 50)
c = _doc("c.md", "C" * 50)
d = _doc("d.md", "D" * 50)
return [
_bonus_rc(a, 0.0200, 0.9, 0),
_bonus_rc(gitea, 0.0160, 0.0, 0, name_hit=True),
_bonus_rc(b, 0.0150, 0.7, 0),
_bonus_rc(c, 0.0150, 0.6, 0),
_bonus_rc(a, 0.0120, 0.5, 1),
_bonus_rc(d, 0.0100, 0.1, 0),
]
#: The golden document order the OLD pre-phase loop (stable score-
#: descending walk, first-seen-chunk dedupe) produces over
#: :func:`_bonus_chunks` — pinned byte-identical by the kill switch.
GOLDEN_PRE_PHASE_ORDER = ["a.md", "gitea/README.md", "b.md", "c.md", "d.md"]
def test_selection_order_bonus_zero_is_the_pre_phase_golden_walk() -> None:
"""LOCKED A3 kill switch: ``bonus=0`` returns the EXACT pre-phase
document order of the old score-descending first-seen walk — the
golden list pinned from the old loop over the mixed fused list
(incl. the b/c fused-score tie resolved by the input order the
fusion produced — the walk never re-sorts it away)."""
out = _selection_order(_bonus_chunks(), 0.0)
assert [d.path for d, _eff, _cos, _idx in out] == GOLDEN_PRE_PHASE_ORDER
# The re-rank inputs are exposed and exact: effective == best fused
# score (no bonus), best cosine tracked across a doc's chunks (a: 0.9
# from its rank-1 chunk, not 0.5), first-seen index in the
# score-descending walk.
assert [eff for _d, eff, _cos, _idx in out] == [
0.0200, 0.0160, 0.0150, 0.0150, 0.0100
]
assert [cos for _d, _eff, cos, _idx in out] == [
pytest.approx(v) for v in (0.9, 0.0, 0.7, 0.6, 0.1)
]
assert [idx for _d, _eff, _cos, idx in out] == [0, 1, 2, 3, 5]
def test_selection_order_bonus_inert_without_name_hits() -> None:
"""No name-hit chunk present → the bonus cannot fire: the order is
IDENTICAL to the pre-phase walk even with the default bonus on
(LOCKED A3)."""
chunks = _bonus_chunks()
for rc in chunks:
rc.name_hit = False
out = _selection_order(chunks, 0.005)
assert [d.path for d, _eff, _cos, _idx in out] == GOLDEN_PRE_PHASE_ORDER
def test_selection_order_bonus_lifts_name_hit_doc_below_bonus_gap() -> None:
"""The name-hit doc's effective 0.016 + 0.005 = 0.021 EXCEEDS a's
0.020 — a gap of 0.004 < bonus 0.005 — so the bonus lifts it to
rank 1; the rest keep their fused order (the bonus re-ranks, it
does not inflate)."""
out = _selection_order(_bonus_chunks(), 0.005)
assert [d.path for d, _eff, _cos, _idx in out] == [
"gitea/README.md", "a.md", "b.md", "c.md", "d.md"
]
assert out[0][1] == pytest.approx(0.016 + 0.005)
def test_selection_order_bonus_does_not_lift_above_bonus_gap() -> None:
"""A gap LARGER than the bonus is not closed: the name-hit doc's
best 0.010 + 0.005 = 0.015 ties b/c on effective and LOSES to both
on the (−effective, −best_cosine) tie-break (its D1 cosine is 0.0) —
a keeps the lead (0.020). A second name-hit doc (``e.md``, also
0.010/cos 0.0) trails gitea on the ``document.path`` tie-break —
the full re-rank key pinned."""
chunks = _bonus_chunks()
chunks[1].score = 0.010 # the name-hit doc drops to a 0.010 best
e = _doc("e.md", "E" * 50)
chunks.insert(2, _bonus_rc(e, 0.010, 0.0, 0, name_hit=True))
out = _selection_order(chunks, 0.005)
assert [d.path for d, _eff, _cos, _idx in out] == [
"a.md", "b.md", "c.md", "e.md", "gitea/README.md", "d.md"
]
def test_selection_order_first_seen_breaks_equal_path_ties() -> None:
"""Two documents sharing a path across sources (``notes.md`` in two
sources) can tie on (effective, cosine, path) — the pre-bonus
first-seen rank decides (the last key element)."""
s1 = _doc("notes.md", "X" * 50, source="Src1")
s2 = _doc("notes.md", "Y" * 50, source="Src2")
chunks = [
_bonus_rc(s1, 0.016, 0.0, 0, name_hit=True),
_bonus_rc(s2, 0.016, 0.0, 0, name_hit=True),
]
out = _selection_order(chunks, 0.005)
assert [d.source for d, *_ in out] == ["Src1", "Src2"]
def test_selection_order_bonus_applied_once_per_document() -> None:
"""The bonus is per DOCUMENT — applied ONCE no matter how many of
the doc's chunks are name hits (3×bonus would push the name-hit doc
above the 0.030 leader; one bonus cannot)."""
gitea = _doc("gitea/README.md", "G" * 50)
a = _doc("a.md", "A" * 50)
chunks = [
_bonus_rc(a, 0.030, 0.8),
_bonus_rc(gitea, 0.016, 0.0, 0, name_hit=True),
_bonus_rc(gitea, 0.010, 0.0, 1, name_hit=True),
_bonus_rc(gitea, 0.008, 0.0, 2, name_hit=True),
]
out = _selection_order(chunks, 0.005)
assert [d.path for d, _eff, _cos, _idx in out] == ["a.md", "gitea/README.md"]
assert out[1][1] == pytest.approx(0.016 + 0.005) # best + ONE bonus
def test_selection_order_bonus_fires_when_name_hit_is_not_first_chunk() -> None:
"""The bonus fires on ANY name-hit chunk of the document — including
when the doc's first-seen (best) chunk is an ordinary vector row and
only a lower-ranked chunk is the D1 name-hit representative (a
document's name hit and its best chunk can be different chunks).
The bonus still lands on the doc's BEST fused score, and the doc's
best cosine stays tracked across ALL its chunks."""
gitea = _doc("gitea/README.md", "G" * 50)
a = _doc("a.md", "A" * 50)
chunks = [
_bonus_rc(a, 0.024, 0.8),
_bonus_rc(gitea, 0.020, 0.2, 0), # the doc's best — an ordinary chunk
_bonus_rc(gitea, 0.016, 0.55, 2, name_hit=True), # the name-hit rep
]
assert [d.path for d, *_ in _selection_order(chunks, 0.0)] == [
"a.md", "gitea/README.md"
]
out = _selection_order(chunks, 0.005)
assert [d.path for d, *_ in out] == ["gitea/README.md", "a.md"]
assert out[0][1] == pytest.approx(0.020 + 0.005) # bonus on the BEST score
# The doc's best cosine is tracked across ALL its chunks — the
# lower-ranked name-hit chunk (0.55) beats the first-seen chunk's
# 0.2 (the re-rank tie-break input).
assert out[0][2] == pytest.approx(0.55)
def test_suggested_bonus_default_from_settings_and_kill_switch(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The *bonus* parameter defaults to the LIVE ``BOR_NAME_HIT_BONUS``
setting (the ``n`` parameter's settings-read pattern — the default
0.005 is the production value, not a frozen constant); an explicit
``bonus=0`` and a settings kill switch both reproduce the pre-phase
golden walk (LOCKED A3)."""
chunks = _bonus_chunks()
settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
assert settings.name_hit_bonus == 0.005 # the production default
monkeypatch.setattr(retriever, "get_settings", lambda: settings)
assert [d.path for d in select_suggested(chunks, n=5)] == [
"gitea/README.md", "a.md", "b.md", "c.md", "d.md"
]
# Explicit kill switch: the byte-identical pre-phase order.
assert [d.path for d in select_suggested(chunks, n=5, bonus=0.0)] == GOLDEN_PRE_PHASE_ORDER
# Settings kill switch (BOR_NAME_HIT_BONUS=0) — the same golden walk.
off = Settings(_env_file=None, name_hit_bonus=0.0) # pyright: ignore[reportCallIssue]
monkeypatch.setattr(retriever, "get_settings", lambda: off)
assert [d.path for d in select_suggested(chunks, n=5)] == GOLDEN_PRE_PHASE_ORDER
def test_related_skips_excluded_ids_under_the_bonus() -> None:
"""Exclusion is orthogonal to the bonus: excluded ids are skipped
exactly as before, on the bonus-adjusted walk — an excluded doc
never rides the related row even when the bonus would lift it to
the lead."""
chunks = _bonus_chunks()
b_id = chunks[2].document.id
d_id = chunks[5].document.id
out = select_related(chunks, {b_id, d_id}, cap=5, bonus=0.005)
assert [d.path for d in out] == ["gitea/README.md", "a.md", "c.md"]
# Kill switch: the same exclusions on the pre-phase walk.
out0 = select_related(chunks, {b_id, d_id}, cap=5, bonus=0.0)
assert [d.path for d in out0] == ["a.md", "gitea/README.md", "c.md"]
# The name-hit doc itself excluded → the lead goes to the next doc.
g_id = chunks[1].document.id
out2 = select_related(chunks, {g_id}, cap=5, bonus=0.005)
assert [d.path for d in out2][0] == "a.md"
def test_weak_hit_titles_bonus_adjusted_order() -> None:
"""Titles follow the bonus-adjusted selection walk (``_doc`` titles
equal paths here, so the title list mirrors the doc order); the
kill switch returns the pre-phase golden order."""
chunks = _bonus_chunks()
assert weak_hit_titles(chunks, bonus=0.005) == [
"gitea/README.md", "a.md", "b.md", "c.md", "d.md"
]
assert weak_hit_titles(chunks, bonus=0.0) == GOLDEN_PRE_PHASE_ORDER
def test_bonus_lives_in_the_selection_layer_only() -> None:
"""LOCKED A3: the bonus never touches the chunk objects —
``score``/``cosine``/``fts_hit`` (the A8 gate's inputs —
``query_log.top_score`` is the best fused chunk score, the same
values) are unchanged after every selection walk, even with the
bonus lifting a document."""
chunks = _bonus_chunks()
before = {
rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks
}
select_suggested(chunks, n=5, bonus=0.005)
select_related(chunks, set(), cap=5, bonus=0.005)
weak_hit_titles(chunks, bonus=0.005)
after = {
rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks
}
assert before == after
tests/unit/test_source_chip_quality.py
+33 -26
View File
@@ -633,6 +633,13 @@ class _FakeSession:
def get(self, _model: Any, _pk: Any) -> Any:
return None
def execute(self, *args: Any, **kwargs: Any) -> list[Any]:
# Phase 119 (D3): the endpoint's suggested-folder lines run the
# ls catalog fetchers on this session — the fake catalog is
# empty (header-only lines), keeping the prompt builds
# deterministic here.
return []
@pytest.fixture(autouse=True)
def _admin_signed_in(client: TestClient) -> None:
@@ -736,17 +743,19 @@ def test_shape_4_meta_question_deflected_frame_has_no_chips_or_row(
assert "108_history_wire_check/00_phase.md" in row.sources
def test_done_frame_carries_suggested_refs_strong_plus_weak(
def test_done_frame_zero_read_grounded_turn_carries_no_chips(
client: TestClient,
chip_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Shape 2 on the wire under phase 118 (LOCKED A4): the citation
surface is the suggested tier + the agent's reads (deduped) —
with two retrieved docs and no read, BOTH docs are ``sources``
refs (no floor — A3); nothing reaches rank 6+, so ``related`` is
empty; the tiers stay disjoint (the done frame's dedupe). The
durable record keeps the FULL retrieval (LOCKED A3)."""
"""Shape 2 on the wire under phase 119 (LOCKED A1 — the phase-118
A4 union retired): the citation surface is the agent's READ DOCS
only — with two retrieved docs and no read, ``sources`` is empty
(the zero-read grounded turn chips nothing — an accepted,
owner-directed consequence); the never-read suggested docs appear
NOWHERE in the frame (the explicit contrast against the retired
A4 union); nothing reaches rank 6+, so ``related`` is empty. The
durable record keeps the FULL retrieval (118-A3 stands)."""
session, _llm = chip_env
validate = _doc("brain-of-reese", ".agents/validate.sh", "validate.sh",
"VALIDATE_SH_CONTENT")
@@ -766,14 +775,11 @@ def test_done_frame_carries_suggested_refs_strong_plus_weak(
frames = _ask(client, "How does the phase gate decide to validate?")
done = frames[-1]
assert done["deflected"] is False
assert [(s["source"], s["path"]) for s in done["sources"]] == [
("brain-of-reese", ".agents/validate.sh"),
("ServMon", "README.md"), # A4: suggested + read — both suggested (A3)
]
# A1: read docs only — the agent read nothing ⇒ no chips, and the
# never-read suggested docs appear NOWHERE in the frame (the retired
# A4 union would have carried both).
assert done["sources"] == []
assert done["related"] == [] # nothing reaches rank 6+ for 2 docs
cited = {(s["source"], s["path"]) for s in done["sources"]}
related = {(s["source"], s["path"]) for s in done["related"]}
assert cited.isdisjoint(related)
(row,) = session.added
assert isinstance(row, QueryLog)
@@ -788,14 +794,15 @@ def test_agent_read_related_doc_is_cited_not_related(
chip_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The agent-read exemption (LOCKED A4, phase-118 tiering): a
rank-6+ doc — the related tier ("nearby docs") — that the agent
``read`` via the tool is cited by definition: the model read it, so
it was used. It joins ``sources`` (after the suggested docs — it
was not suggested, so the read appends it last, deduped) and is
EXCLUDED from ``related`` (a used doc must never read as "nearby");
the other rank-6+ doc stays in the tier. The read content reached
the model (the tool result in the follow-up request)."""
"""The agent-read exemption (phase-119 A1 × phase-118 tiering):
``sources`` is the READ DOCS only — a rank-6+ doc (the related
tier, "nearby docs") that the agent ``read`` via the tool is the
frame's ONLY chip: the model read it, so it was used. It is EXCLUDED
from ``related`` (a used doc must never read as "nearby" — unchanged
intent, the dedupe now keyed on read docs); the other rank-6+ doc
stays in the tier; the never-read suggested docs appear NOWHERE in
the frame (contrast against the retired A4 union). The read content
reached the model (the tool result in the follow-up request)."""
session, _default_llm = chip_env
strong = _doc("docs", "strong.md", "Strong", "STRONG_DOC_CONTENT")
fillers = [
@@ -838,10 +845,10 @@ def test_agent_read_related_doc_is_cited_not_related(
assert "WEAK_B_READ_BY_AGENT" in tool_msgs[0]["content"]
sources = [(s["source"], s["path"]) for s in done["sources"]]
# A4: suggested (5) + the read doc (last — it was not suggested).
assert sources[-1] == ("docs", "weak-b.md") # read ⇒ cited, last
assert len(sources) == 6
assert ("docs", "weak-c.md") not in sources # never suggested, never read
# A1: read docs only — the read related doc is the ONLY chip (the
# never-read suggested docs are nowhere in the frame).
assert sources == [("docs", "weak-b.md")] # read ⇒ chip
assert ("docs", "weak-c.md") not in sources # never read
related = [(s["source"], s["path"]) for s in done["related"]]
assert related == [("docs", "weak-c.md")] # the read doc is not "nearby"
assert set(sources).isdisjoint(set(related))