phase: 119_name_signal_read_chips
All verification complete. Final report: **Phase 119 final verification pass — all criteria verified, one stale pin fixed.** - Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry. - Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged. - New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2. - Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed). - Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met). - Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
@@ -280,14 +280,14 @@ def test_chat_streams_deltas_then_done_with_sources(client, db, seeded_kb: FakeR
|
||||
assert frames[-1]["type"] == "done" # done is the final event
|
||||
assert done[0]["deflected"] is False
|
||||
assert done[0]["suggestions"] == []
|
||||
sources = done[0]["sources"]
|
||||
assert sources, "done must carry the cited sources"
|
||||
# Phase 118 (A4): the citation surface is the suggested tier (top-5,
|
||||
# no floor) + the agent's reads (none on this turn) — deduped.
|
||||
assert len(sources) == get_settings().suggested_docs
|
||||
assert sources[0]["path"] == "homelab/kubernetes.md"
|
||||
assert sources[0]["source"] == "docs"
|
||||
assert sources[0]["title"] == "Kubernetes Homelab Cluster"
|
||||
# Phase 119 (LOCKED A1 — the phase-118 A4 union retired): the
|
||||
# citation surface is the agent's READ docs only — this turn's
|
||||
# canned LLM never emits a tool call, so nothing was read and the
|
||||
# grounded done frame chips nothing (an accepted, owner-directed
|
||||
# consequence — the answer prose names the doc it used). The
|
||||
# retrieval stays durably recorded (118-A3; pinned in the
|
||||
# query_log test below this one).
|
||||
assert done[0]["sources"] == []
|
||||
|
||||
# The LLM received the locked HIGH prompt — the ``<documents>`` block
|
||||
# seeds the document's stored SUMMARY (phase 118, LOCKED A6: summary
|
||||
@@ -334,13 +334,13 @@ def test_chat_streams_thinking_before_deltas(client, db, seeded_kb: FakeRagLLM)
|
||||
assert "".join(f["text"] for f in thinking) == thinker.thinking
|
||||
assert "".join(d["text"] for d in deltas) == thinker.answer
|
||||
|
||||
# Done still last; sources unchanged by the thinking extension.
|
||||
# Done still last; the citation surface is unchanged by the thinking
|
||||
# extension — phase 119 (A1): read docs only, nothing read ⇒ none.
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is False
|
||||
assert done["suggestions"] == []
|
||||
assert done["sources"][0]["path"] == "homelab/kubernetes.md"
|
||||
assert done["sources"][0]["source"] == "docs"
|
||||
assert done["sources"] == []
|
||||
assert not any(f.get("type") == "error" for f in frames)
|
||||
|
||||
|
||||
@@ -469,12 +469,13 @@ def test_off_topic_question_deflects_honestly(client, db, seeded_kb: FakeRagLLM)
|
||||
def test_done_frame_carries_related_tier_on_grounded_turn(
|
||||
client, db, seeded_kb: FakeRagLLM
|
||||
) -> None:
|
||||
"""Phase 118 (LOCKED A3/A4): a grounded turn's done frame carries the
|
||||
suggested tier in ``sources`` (top-5, no floor) and the related
|
||||
tier — the ranked docs from rank 6+ after the suggested set, capped
|
||||
at ``related_max_docs`` (2) — in ``related``, disjoint from the
|
||||
citation surface. The durable record keeps the FULL retrieval
|
||||
(suggested + related + read, LOCKED A3)."""
|
||||
"""Phase 118 (A3) × phase 119 (A1): a grounded turn's done frame
|
||||
carries the related tier — the ranked docs from rank 6+ after the
|
||||
suggested set, capped at ``related_max_docs`` (2) — in ``related``,
|
||||
disjoint from the read-only ``sources`` (nothing was read on this
|
||||
turn ⇒ no chips; the never-read suggested tier is not on the wire).
|
||||
The durable record keeps the FULL retrieval (suggested + related +
|
||||
read, 118-A3 stands)."""
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: seeded_kb
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
@@ -483,11 +484,11 @@ def test_done_frame_carries_related_tier_on_grounded_turn(
|
||||
|
||||
done = frames[-1]
|
||||
assert done["deflected"] is False
|
||||
# A4: the citation surface is the suggested tier (5, no read on this
|
||||
# turn) — ranks 1–5 for the Kubernetes question.
|
||||
# Phase 119 (A1): the citation surface is the agent's READ docs
|
||||
# only — no read on this turn ⇒ no chips (the retired A4 union
|
||||
# would have carried the suggested ranks 1–5 here).
|
||||
sources = [(s["source"], s["path"]) for s in done["sources"]]
|
||||
assert len(sources) == get_settings().suggested_docs
|
||||
assert sources[0] == ("docs", "homelab/kubernetes.md")
|
||||
assert sources == []
|
||||
related = done["related"]
|
||||
# Rank 6–7 for the Kubernetes question (after the top-5 suggested
|
||||
# set), capped at related_max_docs.
|
||||
@@ -546,12 +547,13 @@ def test_deflected_done_frame_carries_weak_hits_in_related(
|
||||
def test_related_doc_read_by_agent_is_cited_not_related(
|
||||
client, db, seeded_kb: FakeRagLLM
|
||||
) -> None:
|
||||
"""Phase 118 × phase 37: an agent-read doc is a citation by definition
|
||||
(LOCKED A4) — when the agent ``read``s a rank-6+ doc (the related
|
||||
tier, "nearby docs"), it joins done.sources (deduped, last — it was
|
||||
not suggested, so the read appends it) and is EXCLUDED from
|
||||
done.related (a "nearby doc" that was actually used must not read as
|
||||
nearby)."""
|
||||
"""Phase 119 (A1) × phase 37: an agent-read doc is a chip by
|
||||
definition — when the agent ``read``s a rank-6+ doc (the related
|
||||
tier, "nearby docs"), it is the done frame's ONLY chip (the read
|
||||
docs ARE the citation surface since phase 119; the never-read
|
||||
suggested tier is not on the wire) and is EXCLUDED from
|
||||
done.related (a "nearby doc" that was actually used must not read
|
||||
as nearby — unchanged intent, the dedupe keyed on read docs)."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
@@ -572,9 +574,9 @@ def test_related_doc_read_by_agent_is_cited_not_related(
|
||||
done = frames[-1]
|
||||
assert done["deflected"] is False
|
||||
sources = [(s["source"], s["path"]) for s in done["sources"]]
|
||||
# A4: suggested (5) + the read doc (last).
|
||||
assert len(sources) == get_settings().suggested_docs + 1
|
||||
assert sources[-1] == ("docs", "homelab/quadlet/cache.volume") # read ⇒ cited
|
||||
# A1: read docs only — the read related doc is the ONLY chip (the
|
||||
# retired A4 union would have carried 6 here).
|
||||
assert sources == [("docs", "homelab/quadlet/cache.volume")] # read ⇒ chip
|
||||
related = [(s["source"], s["path"]) for s in done["related"]]
|
||||
assert ("docs", "homelab/quadlet/cache.volume") not in related
|
||||
assert set(sources).isdisjoint(set(related))
|
||||
@@ -605,8 +607,9 @@ def test_keyword_question_grounded_by_lexical_hit_despite_weak_cosine(
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is False # weak cosine, but a lexical hit
|
||||
assert done["suggestions"] == []
|
||||
sources = done["sources"]
|
||||
assert sources and sources[0]["path"] == "homelab/networking/static-dns.json"
|
||||
# Phase 119 (A1): read docs only — nothing was read ⇒ no chips;
|
||||
# the lexical hit stays in the durable record (asserted below).
|
||||
assert done["sources"] == []
|
||||
|
||||
(system, _user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1]
|
||||
assert "<relevance>HIGH</relevance>" in system["content"] # grounded prompt
|
||||
@@ -1085,12 +1088,13 @@ def test_truncated_read_streams_tool_result_frame_after_tool_frame(
|
||||
assert body.startswith(f"Document docs/big-read.md:\ndate: 2024-06-15\n{content[:cap]}")
|
||||
assert TRUNCATION_MARKER in body
|
||||
assert READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) in body
|
||||
# The truncated read is still a SUCCESSFUL call — cited in done.
|
||||
# The truncated read is still a SUCCESSFUL call — it is the frame's
|
||||
# only chip (phase 119, A1: read docs only; the suggested kubernetes
|
||||
# doc was never read, so it is not on the wire).
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done" and done["deflected"] is False
|
||||
assert ("docs", "big-read.md") in [(s["source"], s["path"]) for s in done["sources"]]
|
||||
assert ("docs", "homelab/kubernetes.md") in [
|
||||
(s["source"], s["path"]) for s in done["sources"]
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
("docs", "big-read.md")
|
||||
]
|
||||
|
||||
|
||||
@@ -1159,11 +1163,12 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
|
||||
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""(a) Grounded turn with tool calls: the event sequence is
|
||||
``thinking?/tool/tool/delta…/done``; ``done.sources`` and the
|
||||
``query_log`` row include the read document (deduped, order
|
||||
preserved); the per-turn log line carries ``tool_calls=2``.
|
||||
Phase 45: the agent loop keeps offering the tools for the whole
|
||||
turn — the round cap (not per-tool budgets) is the bound."""
|
||||
``thinking?/tool/tool/delta…/done``; ``done.sources`` is the read
|
||||
document (phase 119, A1 — the read docs are the citation surface)
|
||||
and the ``query_log`` row includes it (deduped, order preserved);
|
||||
the per-turn log line carries ``tool_calls=2``. Phase 45: the
|
||||
agent loop keeps offering the tools for the whole turn — the round
|
||||
cap (not per-tool budgets) is the bound."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
@@ -1212,12 +1217,12 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
|
||||
|
||||
done = frames[-1]
|
||||
assert done["deflected"] is False
|
||||
# done.sources = the retrieval docs + the read doc, deduped, order kept.
|
||||
# Phase 119 (A1): done.sources = the agent's READ docs only — the
|
||||
# read doc is the frame's only chip (the suggested retrieval docs
|
||||
# are not on the wire; they stay in the durable record below).
|
||||
sources = [(s["source"], s["path"]) for s in done["sources"]]
|
||||
assert sources[-1] == ("docs", "homelab/backups.md") # the read doc is cited
|
||||
assert ("docs", "homelab/kubernetes.md") in sources # …after the retrieval docs
|
||||
assert len(sources) == len(set(sources)) # deduped by (source, path)
|
||||
assert done["sources"][-1]["title"] == "Backup Strategy"
|
||||
assert sources == [("docs", "homelab/backups.md")] # the read doc is cited
|
||||
assert done["sources"][0]["title"] == "Backup Strategy"
|
||||
|
||||
# Phase 45: the tools stay offered on every request — the round cap
|
||||
# (not spent budgets) bounds the loop, and the model answered while
|
||||
@@ -1250,7 +1255,8 @@ def test_grounded_turn_streams_grep_tool_frames(
|
||||
the raw pattern is the frame's ``argument`` (the UI renders the
|
||||
"searching for" line from it). A non-string pattern — a model error
|
||||
the backend refuses — yields ``argument: null``. A grep adds no
|
||||
source: ``done.sources`` stays the retrieval docs (locked A5)."""
|
||||
source (locked A5): with no read on the turn, ``done.sources`` is
|
||||
empty (phase 119, A1)."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
@@ -1292,9 +1298,8 @@ def test_grounded_turn_streams_grep_tool_frames(
|
||||
assert [f for f in frames if f["type"] == "delta"]
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done" and done["deflected"] is False
|
||||
paths = [s["path"] for s in done["sources"]]
|
||||
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
|
||||
assert "homelab/backups.md" not in paths # a grep adds no source
|
||||
# A grep adds no source (A5) and nothing was read (A1) ⇒ no chips.
|
||||
assert done["sources"] == []
|
||||
|
||||
|
||||
def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
|
||||
@@ -1306,7 +1311,8 @@ def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
|
||||
and frame emission is execution-independent: a rejected call (an
|
||||
unknown ``read`` path) still streams its frame with the model's
|
||||
argument as-is. The rejected read adds no source (``done.sources``
|
||||
stays the retrieval docs), and rejected calls count nothing
|
||||
carries no chip — the read failed and nothing else was read,
|
||||
phase 119 A1), and rejected calls count nothing
|
||||
(``tool_calls=1`` — only the executed scoped ``ls``)."""
|
||||
# The scoped ``ls`` source-name check reads the registry — insert a
|
||||
# row resolving to ``docs`` (the fixture's source name) and delete
|
||||
@@ -1350,12 +1356,12 @@ def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
|
||||
# passed — frame emission is execution-independent.
|
||||
assert read_frame["argument"] == "docs/homelab/nope.md"
|
||||
|
||||
# The rejected read adds no source — done.sources stays retrieval.
|
||||
# The rejected read adds no source — and the read-only surface is
|
||||
# empty on this turn (nothing was read; the refused read cites
|
||||
# nothing, phase 119 A1).
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done" and done["deflected"] is False
|
||||
paths = [s["path"] for s in done["sources"]]
|
||||
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
|
||||
assert "homelab/nope.md" not in paths # the refused read cites nothing
|
||||
assert done["sources"] == []
|
||||
|
||||
# The rejected call counts nothing — only the executed scoped ls.
|
||||
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
|
||||
@@ -1462,9 +1468,10 @@ def test_zero_max_rounds_reproduce_pre_phase_single_request(
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is False
|
||||
paths = [s["path"] for s in done["sources"]]
|
||||
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
|
||||
assert "homelab/backups.md" not in paths # nothing was read
|
||||
# Phase 119 (A1): nothing was read ⇒ no chips (the kill-switch
|
||||
# turn's citation surface is empty; the retrieval stays in the
|
||||
# durable record below).
|
||||
assert done["sources"] == []
|
||||
|
||||
# Exactly one request, and it carried no ``tools`` key at all — the
|
||||
# scripted tool calls were never even offered a chance.
|
||||
|
||||
Reference in New Issue
Block a user