phase: 119_name_signal_read_chips
All verification complete. Final report: **Phase 119 final verification pass — all criteria verified, one stale pin fixed.** - Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry. - Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged. - New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2. - Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed). - Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met). - Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
@@ -288,7 +288,10 @@ def test_token_user_surface_matrix(client: TestClient, db, seeded_kb: FakeRagLLM
|
||||
assert content_type.startswith("text/event-stream")
|
||||
assert frames[-1]["type"] == "done"
|
||||
assert frames[-1]["deflected"] is False
|
||||
assert frames[-1]["sources"][0]["path"] == "homelab/kubernetes.md"
|
||||
# Phase 119 (A1): the citation surface is the agent's READ docs
|
||||
# only — the canned turn reads nothing ⇒ no chips (the cited
|
||||
# document's content pin below is the surface's other half).
|
||||
assert frames[-1]["sources"] == []
|
||||
|
||||
# Suggestion chips + the cited document's content (the viewer).
|
||||
assert client.get("/api/suggestions").status_code == 200
|
||||
@@ -455,7 +458,8 @@ def test_admin_chat_still_streams(client: TestClient, db, seeded_kb: FakeRagLLM)
|
||||
assert len(deltas) >= 2 # genuinely streamed
|
||||
assert frames[-1]["type"] == "done"
|
||||
assert frames[-1]["deflected"] is False
|
||||
assert frames[-1]["sources"][0]["path"] == "homelab/kubernetes.md"
|
||||
# Phase 119 (A1): read docs only — nothing was read ⇒ no chips.
|
||||
assert frames[-1]["sources"] == []
|
||||
|
||||
|
||||
# ---------- phase 81: rate-limited failed sign-ins (audit SEC-03) ----------
|
||||
|
||||
@@ -280,14 +280,14 @@ def test_chat_streams_deltas_then_done_with_sources(client, db, seeded_kb: FakeR
|
||||
assert frames[-1]["type"] == "done" # done is the final event
|
||||
assert done[0]["deflected"] is False
|
||||
assert done[0]["suggestions"] == []
|
||||
sources = done[0]["sources"]
|
||||
assert sources, "done must carry the cited sources"
|
||||
# Phase 118 (A4): the citation surface is the suggested tier (top-5,
|
||||
# no floor) + the agent's reads (none on this turn) — deduped.
|
||||
assert len(sources) == get_settings().suggested_docs
|
||||
assert sources[0]["path"] == "homelab/kubernetes.md"
|
||||
assert sources[0]["source"] == "docs"
|
||||
assert sources[0]["title"] == "Kubernetes Homelab Cluster"
|
||||
# Phase 119 (LOCKED A1 — the phase-118 A4 union retired): the
|
||||
# citation surface is the agent's READ docs only — this turn's
|
||||
# canned LLM never emits a tool call, so nothing was read and the
|
||||
# grounded done frame chips nothing (an accepted, owner-directed
|
||||
# consequence — the answer prose names the doc it used). The
|
||||
# retrieval stays durably recorded (118-A3; pinned in the
|
||||
# query_log test below this one).
|
||||
assert done[0]["sources"] == []
|
||||
|
||||
# The LLM received the locked HIGH prompt — the ``<documents>`` block
|
||||
# seeds the document's stored SUMMARY (phase 118, LOCKED A6: summary
|
||||
@@ -334,13 +334,13 @@ def test_chat_streams_thinking_before_deltas(client, db, seeded_kb: FakeRagLLM)
|
||||
assert "".join(f["text"] for f in thinking) == thinker.thinking
|
||||
assert "".join(d["text"] for d in deltas) == thinker.answer
|
||||
|
||||
# Done still last; sources unchanged by the thinking extension.
|
||||
# Done still last; the citation surface is unchanged by the thinking
|
||||
# extension — phase 119 (A1): read docs only, nothing read ⇒ none.
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is False
|
||||
assert done["suggestions"] == []
|
||||
assert done["sources"][0]["path"] == "homelab/kubernetes.md"
|
||||
assert done["sources"][0]["source"] == "docs"
|
||||
assert done["sources"] == []
|
||||
assert not any(f.get("type") == "error" for f in frames)
|
||||
|
||||
|
||||
@@ -469,12 +469,13 @@ def test_off_topic_question_deflects_honestly(client, db, seeded_kb: FakeRagLLM)
|
||||
def test_done_frame_carries_related_tier_on_grounded_turn(
|
||||
client, db, seeded_kb: FakeRagLLM
|
||||
) -> None:
|
||||
"""Phase 118 (LOCKED A3/A4): a grounded turn's done frame carries the
|
||||
suggested tier in ``sources`` (top-5, no floor) and the related
|
||||
tier — the ranked docs from rank 6+ after the suggested set, capped
|
||||
at ``related_max_docs`` (2) — in ``related``, disjoint from the
|
||||
citation surface. The durable record keeps the FULL retrieval
|
||||
(suggested + related + read, LOCKED A3)."""
|
||||
"""Phase 118 (A3) × phase 119 (A1): a grounded turn's done frame
|
||||
carries the related tier — the ranked docs from rank 6+ after the
|
||||
suggested set, capped at ``related_max_docs`` (2) — in ``related``,
|
||||
disjoint from the read-only ``sources`` (nothing was read on this
|
||||
turn ⇒ no chips; the never-read suggested tier is not on the wire).
|
||||
The durable record keeps the FULL retrieval (suggested + related +
|
||||
read, 118-A3 stands)."""
|
||||
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: seeded_kb
|
||||
try:
|
||||
_, _, frames = _stream_chat(client, QUESTION)
|
||||
@@ -483,11 +484,11 @@ def test_done_frame_carries_related_tier_on_grounded_turn(
|
||||
|
||||
done = frames[-1]
|
||||
assert done["deflected"] is False
|
||||
# A4: the citation surface is the suggested tier (5, no read on this
|
||||
# turn) — ranks 1–5 for the Kubernetes question.
|
||||
# Phase 119 (A1): the citation surface is the agent's READ docs
|
||||
# only — no read on this turn ⇒ no chips (the retired A4 union
|
||||
# would have carried the suggested ranks 1–5 here).
|
||||
sources = [(s["source"], s["path"]) for s in done["sources"]]
|
||||
assert len(sources) == get_settings().suggested_docs
|
||||
assert sources[0] == ("docs", "homelab/kubernetes.md")
|
||||
assert sources == []
|
||||
related = done["related"]
|
||||
# Rank 6–7 for the Kubernetes question (after the top-5 suggested
|
||||
# set), capped at related_max_docs.
|
||||
@@ -546,12 +547,13 @@ def test_deflected_done_frame_carries_weak_hits_in_related(
|
||||
def test_related_doc_read_by_agent_is_cited_not_related(
|
||||
client, db, seeded_kb: FakeRagLLM
|
||||
) -> None:
|
||||
"""Phase 118 × phase 37: an agent-read doc is a citation by definition
|
||||
(LOCKED A4) — when the agent ``read``s a rank-6+ doc (the related
|
||||
tier, "nearby docs"), it joins done.sources (deduped, last — it was
|
||||
not suggested, so the read appends it) and is EXCLUDED from
|
||||
done.related (a "nearby doc" that was actually used must not read as
|
||||
nearby)."""
|
||||
"""Phase 119 (A1) × phase 37: an agent-read doc is a chip by
|
||||
definition — when the agent ``read``s a rank-6+ doc (the related
|
||||
tier, "nearby docs"), it is the done frame's ONLY chip (the read
|
||||
docs ARE the citation surface since phase 119; the never-read
|
||||
suggested tier is not on the wire) and is EXCLUDED from
|
||||
done.related (a "nearby doc" that was actually used must not read
|
||||
as nearby — unchanged intent, the dedupe keyed on read docs)."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
@@ -572,9 +574,9 @@ def test_related_doc_read_by_agent_is_cited_not_related(
|
||||
done = frames[-1]
|
||||
assert done["deflected"] is False
|
||||
sources = [(s["source"], s["path"]) for s in done["sources"]]
|
||||
# A4: suggested (5) + the read doc (last).
|
||||
assert len(sources) == get_settings().suggested_docs + 1
|
||||
assert sources[-1] == ("docs", "homelab/quadlet/cache.volume") # read ⇒ cited
|
||||
# A1: read docs only — the read related doc is the ONLY chip (the
|
||||
# retired A4 union would have carried 6 here).
|
||||
assert sources == [("docs", "homelab/quadlet/cache.volume")] # read ⇒ chip
|
||||
related = [(s["source"], s["path"]) for s in done["related"]]
|
||||
assert ("docs", "homelab/quadlet/cache.volume") not in related
|
||||
assert set(sources).isdisjoint(set(related))
|
||||
@@ -605,8 +607,9 @@ def test_keyword_question_grounded_by_lexical_hit_despite_weak_cosine(
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is False # weak cosine, but a lexical hit
|
||||
assert done["suggestions"] == []
|
||||
sources = done["sources"]
|
||||
assert sources and sources[0]["path"] == "homelab/networking/static-dns.json"
|
||||
# Phase 119 (A1): read docs only — nothing was read ⇒ no chips;
|
||||
# the lexical hit stays in the durable record (asserted below).
|
||||
assert done["sources"] == []
|
||||
|
||||
(system, _user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1]
|
||||
assert "<relevance>HIGH</relevance>" in system["content"] # grounded prompt
|
||||
@@ -1085,12 +1088,13 @@ def test_truncated_read_streams_tool_result_frame_after_tool_frame(
|
||||
assert body.startswith(f"Document docs/big-read.md:\ndate: 2024-06-15\n{content[:cap]}")
|
||||
assert TRUNCATION_MARKER in body
|
||||
assert READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) in body
|
||||
# The truncated read is still a SUCCESSFUL call — cited in done.
|
||||
# The truncated read is still a SUCCESSFUL call — it is the frame's
|
||||
# only chip (phase 119, A1: read docs only; the suggested kubernetes
|
||||
# doc was never read, so it is not on the wire).
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done" and done["deflected"] is False
|
||||
assert ("docs", "big-read.md") in [(s["source"], s["path"]) for s in done["sources"]]
|
||||
assert ("docs", "homelab/kubernetes.md") in [
|
||||
(s["source"], s["path"]) for s in done["sources"]
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
("docs", "big-read.md")
|
||||
]
|
||||
|
||||
|
||||
@@ -1159,11 +1163,12 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
|
||||
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""(a) Grounded turn with tool calls: the event sequence is
|
||||
``thinking?/tool/tool/delta…/done``; ``done.sources`` and the
|
||||
``query_log`` row include the read document (deduped, order
|
||||
preserved); the per-turn log line carries ``tool_calls=2``.
|
||||
Phase 45: the agent loop keeps offering the tools for the whole
|
||||
turn — the round cap (not per-tool budgets) is the bound."""
|
||||
``thinking?/tool/tool/delta…/done``; ``done.sources`` is the read
|
||||
document (phase 119, A1 — the read docs are the citation surface)
|
||||
and the ``query_log`` row includes it (deduped, order preserved);
|
||||
the per-turn log line carries ``tool_calls=2``. Phase 45: the
|
||||
agent loop keeps offering the tools for the whole turn — the round
|
||||
cap (not per-tool budgets) is the bound."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
@@ -1212,12 +1217,12 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
|
||||
|
||||
done = frames[-1]
|
||||
assert done["deflected"] is False
|
||||
# done.sources = the retrieval docs + the read doc, deduped, order kept.
|
||||
# Phase 119 (A1): done.sources = the agent's READ docs only — the
|
||||
# read doc is the frame's only chip (the suggested retrieval docs
|
||||
# are not on the wire; they stay in the durable record below).
|
||||
sources = [(s["source"], s["path"]) for s in done["sources"]]
|
||||
assert sources[-1] == ("docs", "homelab/backups.md") # the read doc is cited
|
||||
assert ("docs", "homelab/kubernetes.md") in sources # …after the retrieval docs
|
||||
assert len(sources) == len(set(sources)) # deduped by (source, path)
|
||||
assert done["sources"][-1]["title"] == "Backup Strategy"
|
||||
assert sources == [("docs", "homelab/backups.md")] # the read doc is cited
|
||||
assert done["sources"][0]["title"] == "Backup Strategy"
|
||||
|
||||
# Phase 45: the tools stay offered on every request — the round cap
|
||||
# (not spent budgets) bounds the loop, and the model answered while
|
||||
@@ -1250,7 +1255,8 @@ def test_grounded_turn_streams_grep_tool_frames(
|
||||
the raw pattern is the frame's ``argument`` (the UI renders the
|
||||
"searching for" line from it). A non-string pattern — a model error
|
||||
the backend refuses — yields ``argument: null``. A grep adds no
|
||||
source: ``done.sources`` stays the retrieval docs (locked A5)."""
|
||||
source (locked A5): with no read on the turn, ``done.sources`` is
|
||||
empty (phase 119, A1)."""
|
||||
scripted = FakeRagLLM(
|
||||
tool_script=[
|
||||
[
|
||||
@@ -1292,9 +1298,8 @@ def test_grounded_turn_streams_grep_tool_frames(
|
||||
assert [f for f in frames if f["type"] == "delta"]
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done" and done["deflected"] is False
|
||||
paths = [s["path"] for s in done["sources"]]
|
||||
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
|
||||
assert "homelab/backups.md" not in paths # a grep adds no source
|
||||
# A grep adds no source (A5) and nothing was read (A1) ⇒ no chips.
|
||||
assert done["sources"] == []
|
||||
|
||||
|
||||
def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
|
||||
@@ -1306,7 +1311,8 @@ def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
|
||||
and frame emission is execution-independent: a rejected call (an
|
||||
unknown ``read`` path) still streams its frame with the model's
|
||||
argument as-is. The rejected read adds no source (``done.sources``
|
||||
stays the retrieval docs), and rejected calls count nothing
|
||||
carries no chip — the read failed and nothing else was read,
|
||||
phase 119 A1), and rejected calls count nothing
|
||||
(``tool_calls=1`` — only the executed scoped ``ls``)."""
|
||||
# The scoped ``ls`` source-name check reads the registry — insert a
|
||||
# row resolving to ``docs`` (the fixture's source name) and delete
|
||||
@@ -1350,12 +1356,12 @@ def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
|
||||
# passed — frame emission is execution-independent.
|
||||
assert read_frame["argument"] == "docs/homelab/nope.md"
|
||||
|
||||
# The rejected read adds no source — done.sources stays retrieval.
|
||||
# The rejected read adds no source — and the read-only surface is
|
||||
# empty on this turn (nothing was read; the refused read cites
|
||||
# nothing, phase 119 A1).
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done" and done["deflected"] is False
|
||||
paths = [s["path"] for s in done["sources"]]
|
||||
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
|
||||
assert "homelab/nope.md" not in paths # the refused read cites nothing
|
||||
assert done["sources"] == []
|
||||
|
||||
# The rejected call counts nothing — only the executed scoped ls.
|
||||
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
|
||||
@@ -1462,9 +1468,10 @@ def test_zero_max_rounds_reproduce_pre_phase_single_request(
|
||||
done = frames[-1]
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is False
|
||||
paths = [s["path"] for s in done["sources"]]
|
||||
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
|
||||
assert "homelab/backups.md" not in paths # nothing was read
|
||||
# Phase 119 (A1): nothing was read ⇒ no chips (the kill-switch
|
||||
# turn's citation surface is empty; the retrieval stays in the
|
||||
# durable record below).
|
||||
assert done["sources"] == []
|
||||
|
||||
# Exactly one request, and it carried no ``tools`` key at all — the
|
||||
# scripted tool calls were never even offered a chance.
|
||||
|
||||
@@ -25,15 +25,17 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import select, text
|
||||
from sqlalchemy import text
|
||||
from test_chat_api import FakeRagLLM, _stream_chat, _token_vec
|
||||
|
||||
from app.api import chat as chat_api
|
||||
from app.config import get_settings
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import Document, KbOverview
|
||||
from app.rag.agent import suggested_folder_lines
|
||||
from app.rag.importer import import_sources
|
||||
from app.rag.prompts import build_deflect_prompt, build_high_prompt
|
||||
from app.rag.retriever import retrieve, weak_hit_titles
|
||||
from app.rag.retriever import retrieve, select_suggested, weak_hit_titles
|
||||
from tests.conftest import ADMIN_PASSWORD
|
||||
|
||||
FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "docs"
|
||||
@@ -90,15 +92,27 @@ def _seed_overview(db) -> None:
|
||||
db.commit()
|
||||
|
||||
|
||||
def _cited_docs(db, frames: list[dict]) -> list[Document]:
|
||||
"""The documents the done event cited, in citation order — the same
|
||||
list ``plan_turn`` passed to the prompt builder."""
|
||||
docs = []
|
||||
for s in frames[-1]["sources"]:
|
||||
doc = db.scalar(select(Document).where(Document.path == s["path"]))
|
||||
assert doc is not None, f"done source {s['path']!r} missing from the KB"
|
||||
docs.append(doc)
|
||||
return docs
|
||||
def _suggested_docs(db) -> list[Document]:
|
||||
"""The documents the grounded turn's HIGH prompt seeded, in the same
|
||||
order ``plan_turn`` walked them — the SAME deterministic suggested
|
||||
walk over the retrieval. Phase 119 (A1) moved the citation surface
|
||||
to the agent's READ docs, so the seeded tier can no longer be read
|
||||
off the done frame (the canned LLM here reads nothing ⇒
|
||||
``done.sources`` is empty) — reconstructing from the retrieval is
|
||||
the faithful source now."""
|
||||
chunks = retrieve(db, QUESTION, _token_vec(QUESTION))
|
||||
return list(select_suggested(chunks, n=get_settings().suggested_docs))
|
||||
|
||||
|
||||
def _suggested_folder_lines(db) -> list[str]:
|
||||
"""The suggested-folder context lines the endpoint computed (phase
|
||||
119, D3, LOCKED A4) — the SAME deterministic suggested walk
|
||||
``plan_turn`` performs internally (one extra walk, no shared
|
||||
state), over the real seeded catalog."""
|
||||
chunks = retrieve(db, QUESTION, _token_vec(QUESTION))
|
||||
return suggested_folder_lines(
|
||||
db, select_suggested(chunks, n=get_settings().suggested_docs)
|
||||
)
|
||||
|
||||
|
||||
def _turn_log_lines(caplog: pytest.LogCaptureFixture) -> list[str]:
|
||||
@@ -124,7 +138,14 @@ def test_no_row_high_prompt_byte_identical_to_pre_phase(
|
||||
assert frames[-1]["deflected"] is False
|
||||
(system, user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1]
|
||||
assert user["content"] == QUESTION
|
||||
expected = build_high_prompt(_cited_docs(db, frames), notes=[], kb_overview=None)
|
||||
# Phase 119 (D3): the endpoint's suggested-folder lines ride the
|
||||
# HIGH prompt — reconstructed the same deterministic way.
|
||||
expected = build_high_prompt(
|
||||
_suggested_docs(db),
|
||||
notes=[],
|
||||
kb_overview=None,
|
||||
folder_lines=_suggested_folder_lines(db),
|
||||
)
|
||||
assert system["content"] == expected
|
||||
assert "<knowledge_base>" not in system["content"]
|
||||
|
||||
@@ -179,8 +200,14 @@ def test_row_high_prompt_carries_kb_section_exactly(
|
||||
|
||||
assert frames[-1]["deflected"] is False
|
||||
(system, _user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1]
|
||||
# Phase 119 (D3): the suggested-folder lines ride the HIGH prompt
|
||||
# alongside the <knowledge_base> section — reconstructed the same
|
||||
# deterministic way.
|
||||
expected = build_high_prompt(
|
||||
_cited_docs(db, frames), notes=[], kb_overview=OVERVIEW
|
||||
_suggested_docs(db),
|
||||
notes=[],
|
||||
kb_overview=OVERVIEW,
|
||||
folder_lines=_suggested_folder_lines(db),
|
||||
)
|
||||
assert system["content"] == expected
|
||||
|
||||
|
||||
@@ -6,10 +6,12 @@ mapping with fake rows; this suite covers the SQL side on real
|
||||
Postgres: the document-projection scan, the LATERAL representative-
|
||||
chunk fetch (the ``is_summary`` chunk wins, chunk 0 otherwise, and a
|
||||
chunk-less name match is EXCLUDED — the ``c.id IS NOT NULL`` guard),
|
||||
the (count, length, catalog) ranking, the name-hits-lead-the-lexical-
|
||||
list union with the FTS rows (chunk-id dedup), and the full
|
||||
``retrieve()`` → ``select_documents()`` path putting the versioned-
|
||||
name document into the seeded top-N.
|
||||
the (matched-token count, catalog) ranking (phase 119, LOCKED A2 —
|
||||
the two-class component rule: digit-bearing prefix, digitless exact,
|
||||
titles never matched), the name-hits-lead-the-lexical-list union with
|
||||
the FTS rows (chunk-id dedup), and the full ``retrieve()`` →
|
||||
``select_documents()`` path putting the versioned-name document into
|
||||
the seeded top-N.
|
||||
|
||||
Requires: ``podman compose up -d db``.
|
||||
"""
|
||||
@@ -137,19 +139,29 @@ def kb(db) -> Iterator[None]:
|
||||
|
||||
|
||||
def test_name_hit_chunks_real_sql(kb, db) -> None:
|
||||
"""Real Postgres: the projection scan finds exactly the qwen3.8
|
||||
quadlet (the qwen3.6 sibling and the chunk-less name match are
|
||||
excluded), and the LATERAL fetch hands back the SUMMARY chunk as
|
||||
the representative (position −1, is_summary)."""
|
||||
"""Real Postgres: the projection scan finds the qwen3.8 quadlet
|
||||
(the ``qwen38`` stem prefix — the incident's original case) AND
|
||||
the ``notes/llama.cpp.md`` doc (the question names "llama.cpp" —
|
||||
the dotted token ``llamacpp`` exact-matches the file stem, the
|
||||
phase-119 two-class rule). The qwen3.6 sibling (``qwen38`` is not
|
||||
a prefix of ``qwen36…``) and the chunk-less name match are
|
||||
excluded; the LATERAL fetch hands back the SUMMARY chunk as the
|
||||
representative for the quadlet (position −1, is_summary) and chunk
|
||||
0 for the single-chunk notes doc."""
|
||||
out = _name_hit_chunks(db, INCIDENT_QUESTION)
|
||||
assert [rc.document.path for rc in out] == [
|
||||
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container"
|
||||
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container",
|
||||
"notes/llama.cpp.md", # the question names it — stem exact (A2)
|
||||
]
|
||||
rc = out[0]
|
||||
assert rc.position == -1 # the summary chunk wins the LATERAL order
|
||||
assert rc.is_summary is True
|
||||
assert rc.fts_hit is True # the lexical signal — the A8 gate answers
|
||||
assert rc.cosine == 0.0 # no vector rank on the name-hit row
|
||||
rc_notes = out[1]
|
||||
assert rc_notes.position == 0 # chunk 0 (no summary chunk)
|
||||
assert rc_notes.is_summary is False
|
||||
assert all(rc.name_hit is True for rc in out) # phase 119 — D2 bonus input
|
||||
assert "qwen3.8-empty.container" not in [r.document.path for r in out] # chunk-less guard
|
||||
|
||||
|
||||
@@ -163,13 +175,19 @@ def test_lexical_candidates_name_hit_leads_real_sql(kb, db) -> None:
|
||||
assert paths[0] == (
|
||||
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container"
|
||||
)
|
||||
assert paths[1] == "notes/llama.cpp.md" # the second name hit (stem exact)
|
||||
# The FTS pollution is still present (the incident's shape) — but
|
||||
# behind the name hit, no longer ahead of it.
|
||||
# behind the name hits, no longer ahead of them.
|
||||
assert (
|
||||
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.6-27b-juggernaut-vulkan.container"
|
||||
in paths
|
||||
)
|
||||
assert all(rc.fts_hit is True for rc in out)
|
||||
# Phase 119: the two name-hit representative rows are flagged, the
|
||||
# plain FTS rows are not.
|
||||
assert out[0].name_hit is True
|
||||
assert out[1].name_hit is True
|
||||
assert all(not rc.name_hit for rc in out[2:])
|
||||
|
||||
|
||||
def test_retrieve_selects_name_hit_doc_into_top_n(kb, db) -> None:
|
||||
@@ -177,7 +195,12 @@ def test_retrieve_selects_name_hit_doc_into_top_n(kb, db) -> None:
|
||||
fused) → ``select_documents`` puts the qwen3.8 quadlet in the
|
||||
seeded top-N — the incident's seed miss (the two overview docs
|
||||
only) is fixed. The question vector is parallel to the q38 chunk
|
||||
embeddings (cosine 1.0), orthogonal to q36 (0.0)."""
|
||||
embeddings (cosine 1.0), orthogonal to q36 (0.0).
|
||||
|
||||
Phase 119: the ``name_hit`` flag survives the fusion — the two
|
||||
name-hit representative chunks are double hits (vector ∪ lexical),
|
||||
and the double-hit merge ORs the flag into the surviving row;
|
||||
the plain q36 vector+FTS row stays False."""
|
||||
question_vec = _vec(1)
|
||||
chunks = retrieve(db, INCIDENT_QUESTION, question_vec)
|
||||
docs = select_documents(chunks, n=2)
|
||||
@@ -185,16 +208,31 @@ def test_retrieve_selects_name_hit_doc_into_top_n(kb, db) -> None:
|
||||
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container",
|
||||
"notes/llama.cpp.md",
|
||||
]
|
||||
by_path: dict[str, list[bool]] = {}
|
||||
for rc in chunks:
|
||||
by_path.setdefault(rc.document.path, []).append(rc.name_hit)
|
||||
# The q38 doc's summary chunk is the double-hit name hit (flagged);
|
||||
# its plain chunk 0 (vector + FTS) is not — the doc has a flagged row.
|
||||
assert any(by_path[
|
||||
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container"
|
||||
])
|
||||
assert any(by_path["notes/llama.cpp.md"]) # its only chunk is the name hit
|
||||
assert not any(by_path[
|
||||
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.6-27b-juggernaut-vulkan.container"
|
||||
]) # plain vector+FTS — never a name hit
|
||||
|
||||
|
||||
def test_name_hit_limit_real_sql(db) -> None:
|
||||
"""Twelve identical (1, 6) name hits — the LATERAL fetch (and the
|
||||
output) carries exactly ``NAME_HIT_LIMIT`` winners, catalog order."""
|
||||
"""Twelve tied name hits (one matched token each — the ``qwen38``
|
||||
stem PREFIX; the token must lead the stem, the old mid-stem
|
||||
containment no longer matches) — the LATERAL fetch (and the
|
||||
output) carries exactly ``NAME_HIT_LIMIT`` winners, catalog
|
||||
order."""
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
for i in range(12):
|
||||
doc = _doc(
|
||||
db, "S", f"quadlets/m{i:02d}-qwen38.container", f"m{i:02d}-qwen38", "llama cpp qwen38\n"
|
||||
db, "S", f"quadlets/qwen38-m{i:02d}.container", f"qwen38-m{i:02d}", "llama cpp qwen38\n"
|
||||
)
|
||||
db.flush()
|
||||
c = _chunk(db, doc, 0, f"llama cpp qwen38 doc {i}")
|
||||
@@ -203,5 +241,6 @@ def test_name_hit_limit_real_sql(db) -> None:
|
||||
out = _name_hit_chunks(db, "what are the llama.cpp arguments for qwen 3.8")
|
||||
assert len(out) == NAME_HIT_LIMIT
|
||||
assert [rc.document.path for rc in out] == [
|
||||
f"quadlets/m{i:02d}-qwen38.container" for i in range(NAME_HIT_LIMIT)
|
||||
f"quadlets/qwen38-m{i:02d}.container" for i in range(NAME_HIT_LIMIT)
|
||||
]
|
||||
assert all(rc.name_hit is True for rc in out)
|
||||
|
||||
@@ -12,13 +12,21 @@ margins below are exact floats, not flaky measurements.
|
||||
**Measured geometry (recorded per task step 4/5):**
|
||||
|
||||
* Owner scenario — A (``backups/retention.md``, created 2020-01-01,
|
||||
the exact answer, cosine 1.0) lands at vector rank 1 + FTS rank 1
|
||||
(fused 0.03278689); B (``backups/retention-draft.md``, created
|
||||
the exact answer, cosine 1.0) lands at vector rank 1 + lexical rank
|
||||
3 (fused 0.03226646); B (``backups/retention-draft.md``, created
|
||||
yesterday, the "under review, no decision yet" draft, cosine
|
||||
0.707107) lands at vector rank 10 + FTS rank 3 (fused 0.03015873 —
|
||||
a solid FTS hit at rank 3, as the task describes). Pre-boost fused
|
||||
margin **A−B = 0.00262816** (asserted ≥ 3× the zero-age boost =
|
||||
0.002100 at the default → ratio 1.25, the "comfortable margin").
|
||||
0.707107) lands at vector rank 10 + lexical rank 1 (fused
|
||||
0.03067915). Pre-boost fused margin **A−B = 1/63 − 1/70 =
|
||||
0.00158730** — re-pinned by phase 119 (task 01): the two-class
|
||||
name-hit rule (LOCKED A2) makes the digitless question token
|
||||
"retention" exact-match the stem sub-component of all three
|
||||
``retention*`` docs, so the lexical list now LEADS with those name
|
||||
hits in catalog order (draft, review, retention) and B takes the
|
||||
rank-1 lexical slot (pre-119 it was a plain FTS rank-1 / rank-3
|
||||
hit). The flip-safety bar the margin is pinned against — margin >
|
||||
the zero-age boost, so a zero-age B's full boost cannot close it
|
||||
(A's own boost being ≈ 0 at its 2020 age) — still holds, at 2.27×
|
||||
the default.
|
||||
* Twin near-tie — C (``twin/c-older.md``, 2019) and D
|
||||
(``twin/d-newer.md``, yesterday) with IDENTICAL chunk text and
|
||||
near-identical vectors (cosine 1.0 vs 0.9999 — a literal identical
|
||||
@@ -29,12 +37,16 @@ margins below are exact floats, not flaky measurements.
|
||||
scale.
|
||||
* The DEFAULT was tuned from the design starting point (0.001) down to
|
||||
**0.0007** (task step 5: "tune the DEFAULTS … until old-correct wins
|
||||
comfortably"): on the k=60 scale the owner scenario's margin is
|
||||
comfortably"): on the k=60 scale the owner scenario's margin was
|
||||
0.00262816 < 3×0.001, and a 0.001 zero-age boost (+0.000997 for a
|
||||
yesterday doc) would have FLIPPED the pinned scenario. 0.0007 keeps
|
||||
the flip margin comfortable (0.000698 > 0.00052882, lead
|
||||
+0.000169) while staying 1.25× under the 3×-boost margin bar. The
|
||||
owner re-tunes live via ``BOR_RECENCY_BOOST``.
|
||||
+0.000169). Phase 119 (task 01) re-pinned the owner margin to
|
||||
0.00158730 (above): still above the zero-age boost (2.27×), so the
|
||||
worst-case flip — a zero-age B against A's ≈ 0 boost — keeps A
|
||||
first; the twin-flip figures (gap 0.00052882, boost 0.000698) are
|
||||
untouched (the twin KB has no name hits). The owner re-tunes live
|
||||
via ``BOR_RECENCY_BOOST``.
|
||||
|
||||
Requires: ``podman compose up -d db``.
|
||||
"""
|
||||
@@ -239,11 +251,12 @@ def test_owner_scenario_old_correct_beats_new_similar(
|
||||
owner_kb, db, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""THE owner scenario, pinned at the DEFAULTS: the older doc that
|
||||
answers ranks above the newer similar one — AND the pre-boost fused
|
||||
margin is ≥ 3× the zero-age boost (the "comfortable margin"; the
|
||||
measured 0.00262816 vs the 0.0021 bar is recorded in the module
|
||||
docstring). Re-pinned with the boost OFF: relevance alone already
|
||||
ordered them (no regression — the boost is not what makes A win)."""
|
||||
answers ranks above the newer similar one — AND the pre-boost
|
||||
fused margin exceeds the zero-age boost (the flip-safety bar; the
|
||||
phase-119-re-pinned measured 0.00158730 vs the 0.0007 default is
|
||||
recorded in the module docstring). Re-pinned with the boost OFF:
|
||||
relevance alone already ordered them (no regression — the boost is
|
||||
not what makes A win)."""
|
||||
chunks = retrieve(db, QUESTION, QUESTION_VEC)
|
||||
assert select_documents(chunks, n=2)[0].path == "backups/retention.md"
|
||||
|
||||
@@ -254,7 +267,15 @@ def test_owner_scenario_old_correct_beats_new_similar(
|
||||
fused = fuse(vector, lexical, s.rrf_k)
|
||||
by_path = {rc.document.path: rc.score for rc in fused}
|
||||
margin = by_path["backups/retention.md"] - by_path["backups/retention-draft.md"]
|
||||
assert margin >= 3 * s.recency_boost
|
||||
# Phase 119 (task 01) re-pinned geometry: the three ``retention*``
|
||||
# docs are name hits (the digitless token "retention" exact-matches
|
||||
# the stem sub-component), ranked in catalog order (draft, review,
|
||||
# retention) — so A sits at vector rank 1 + lexical rank 3 and B
|
||||
# at vector rank 10 + lexical rank 1: the margin is exactly
|
||||
# 1/63 − 1/70. The flip-safety bar: a zero-age B's full boost
|
||||
# cannot close it (A's own boost is ≈ 0 at its 2020 age).
|
||||
assert margin == pytest.approx(1 / 63 - 1 / 70)
|
||||
assert margin > s.recency_boost # 2.27× at the default
|
||||
|
||||
# The kill switch: A still first (relevance alone), and the
|
||||
# weight-0 scores are the pre-phase fused scores byte-identical.
|
||||
|
||||
Reference in New Issue
Block a user