phase: 119_name_signal_read_chips
Build and Push Containers / build-and-push-app (push) Successful in 2m1s
Build and Push Containers / build-and-push-db (push) Successful in 18s

All verification complete. Final report:

**Phase 119 final verification pass — all criteria verified, one stale pin fixed.**
- Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry.
- Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged.
- New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2.
- Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed).
- Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met).
- Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
2026-09-16 15:50:48 -04:00
parent 795fb56425
commit a5b63f83ad
89 changed files with 4377 additions and 655 deletions
+6 -2
View File
@@ -288,7 +288,10 @@ def test_token_user_surface_matrix(client: TestClient, db, seeded_kb: FakeRagLLM
assert content_type.startswith("text/event-stream")
assert frames[-1]["type"] == "done"
assert frames[-1]["deflected"] is False
assert frames[-1]["sources"][0]["path"] == "homelab/kubernetes.md"
# Phase 119 (A1): the citation surface is the agent's READ docs
# only — the canned turn reads nothing ⇒ no chips (the cited
# document's content pin below is the surface's other half).
assert frames[-1]["sources"] == []
# Suggestion chips + the cited document's content (the viewer).
assert client.get("/api/suggestions").status_code == 200
@@ -455,7 +458,8 @@ def test_admin_chat_still_streams(client: TestClient, db, seeded_kb: FakeRagLLM)
assert len(deltas) >= 2 # genuinely streamed
assert frames[-1]["type"] == "done"
assert frames[-1]["deflected"] is False
assert frames[-1]["sources"][0]["path"] == "homelab/kubernetes.md"
# Phase 119 (A1): read docs only — nothing was read ⇒ no chips.
assert frames[-1]["sources"] == []
# ---------- phase 81: rate-limited failed sign-ins (audit SEC-03) ----------
+65 -58
View File
@@ -280,14 +280,14 @@ def test_chat_streams_deltas_then_done_with_sources(client, db, seeded_kb: FakeR
assert frames[-1]["type"] == "done" # done is the final event
assert done[0]["deflected"] is False
assert done[0]["suggestions"] == []
sources = done[0]["sources"]
assert sources, "done must carry the cited sources"
# Phase 118 (A4): the citation surface is the suggested tier (top-5,
# no floor) + the agent's reads (none on this turn) — deduped.
assert len(sources) == get_settings().suggested_docs
assert sources[0]["path"] == "homelab/kubernetes.md"
assert sources[0]["source"] == "docs"
assert sources[0]["title"] == "Kubernetes Homelab Cluster"
# Phase 119 (LOCKED A1 — the phase-118 A4 union retired): the
# citation surface is the agent's READ docs only — this turn's
# canned LLM never emits a tool call, so nothing was read and the
# grounded done frame chips nothing (an accepted, owner-directed
# consequence — the answer prose names the doc it used). The
# retrieval stays durably recorded (118-A3; pinned in the
# query_log test below this one).
assert done[0]["sources"] == []
# The LLM received the locked HIGH prompt — the ``<documents>`` block
# seeds the document's stored SUMMARY (phase 118, LOCKED A6: summary
@@ -334,13 +334,13 @@ def test_chat_streams_thinking_before_deltas(client, db, seeded_kb: FakeRagLLM)
assert "".join(f["text"] for f in thinking) == thinker.thinking
assert "".join(d["text"] for d in deltas) == thinker.answer
# Done still last; sources unchanged by the thinking extension.
# Done still last; the citation surface is unchanged by the thinking
# extension — phase 119 (A1): read docs only, nothing read ⇒ none.
done = frames[-1]
assert done["type"] == "done"
assert done["deflected"] is False
assert done["suggestions"] == []
assert done["sources"][0]["path"] == "homelab/kubernetes.md"
assert done["sources"][0]["source"] == "docs"
assert done["sources"] == []
assert not any(f.get("type") == "error" for f in frames)
@@ -469,12 +469,13 @@ def test_off_topic_question_deflects_honestly(client, db, seeded_kb: FakeRagLLM)
def test_done_frame_carries_related_tier_on_grounded_turn(
client, db, seeded_kb: FakeRagLLM
) -> None:
"""Phase 118 (LOCKED A3/A4): a grounded turn's done frame carries the
suggested tier in ``sources`` (top-5, no floor) and the related
tier — the ranked docs from rank 6+ after the suggested set, capped
at ``related_max_docs`` (2) — in ``related``, disjoint from the
citation surface. The durable record keeps the FULL retrieval
(suggested + related + read, LOCKED A3)."""
"""Phase 118 (A3) × phase 119 (A1): a grounded turn's done frame
carries the related tier — the ranked docs from rank 6+ after the
suggested set, capped at ``related_max_docs`` (2) — in ``related``,
disjoint from the read-only ``sources`` (nothing was read on this
turn ⇒ no chips; the never-read suggested tier is not on the wire).
The durable record keeps the FULL retrieval (suggested + related +
read, 118-A3 stands)."""
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: seeded_kb
try:
_, _, frames = _stream_chat(client, QUESTION)
@@ -483,11 +484,11 @@ def test_done_frame_carries_related_tier_on_grounded_turn(
done = frames[-1]
assert done["deflected"] is False
# A4: the citation surface is the suggested tier (5, no read on this
# turn) — ranks 1–5 for the Kubernetes question.
# Phase 119 (A1): the citation surface is the agent's READ docs
# only — no read on this turn ⇒ no chips (the retired A4 union
# would have carried the suggested ranks 1–5 here).
sources = [(s["source"], s["path"]) for s in done["sources"]]
assert len(sources) == get_settings().suggested_docs
assert sources[0] == ("docs", "homelab/kubernetes.md")
assert sources == []
related = done["related"]
# Rank 6–7 for the Kubernetes question (after the top-5 suggested
# set), capped at related_max_docs.
@@ -546,12 +547,13 @@ def test_deflected_done_frame_carries_weak_hits_in_related(
def test_related_doc_read_by_agent_is_cited_not_related(
client, db, seeded_kb: FakeRagLLM
) -> None:
"""Phase 118 × phase 37: an agent-read doc is a citation by definition
(LOCKED A4) — when the agent ``read``s a rank-6+ doc (the related
tier, "nearby docs"), it joins done.sources (deduped, last — it was
not suggested, so the read appends it) and is EXCLUDED from
done.related (a "nearby doc" that was actually used must not read as
nearby)."""
"""Phase 119 (A1) × phase 37: an agent-read doc is a chip by
definition — when the agent ``read``s a rank-6+ doc (the related
tier, "nearby docs"), it is the done frame's ONLY chip (the read
docs ARE the citation surface since phase 119; the never-read
suggested tier is not on the wire) and is EXCLUDED from
done.related (a "nearby doc" that was actually used must not read
as nearby — unchanged intent, the dedupe keyed on read docs)."""
scripted = FakeRagLLM(
tool_script=[
[
@@ -572,9 +574,9 @@ def test_related_doc_read_by_agent_is_cited_not_related(
done = frames[-1]
assert done["deflected"] is False
sources = [(s["source"], s["path"]) for s in done["sources"]]
# A4: suggested (5) + the read doc (last).
assert len(sources) == get_settings().suggested_docs + 1
assert sources[-1] == ("docs", "homelab/quadlet/cache.volume") # read ⇒ cited
# A1: read docs only — the read related doc is the ONLY chip (the
# retired A4 union would have carried 6 here).
assert sources == [("docs", "homelab/quadlet/cache.volume")] # read ⇒ chip
related = [(s["source"], s["path"]) for s in done["related"]]
assert ("docs", "homelab/quadlet/cache.volume") not in related
assert set(sources).isdisjoint(set(related))
@@ -605,8 +607,9 @@ def test_keyword_question_grounded_by_lexical_hit_despite_weak_cosine(
assert done["type"] == "done"
assert done["deflected"] is False # weak cosine, but a lexical hit
assert done["suggestions"] == []
sources = done["sources"]
assert sources and sources[0]["path"] == "homelab/networking/static-dns.json"
# Phase 119 (A1): read docs only — nothing was read ⇒ no chips;
# the lexical hit stays in the durable record (asserted below).
assert done["sources"] == []
(system, _user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1]
assert "<relevance>HIGH</relevance>" in system["content"] # grounded prompt
@@ -1085,12 +1088,13 @@ def test_truncated_read_streams_tool_result_frame_after_tool_frame(
assert body.startswith(f"Document docs/big-read.md:\ndate: 2024-06-15\n{content[:cap]}")
assert TRUNCATION_MARKER in body
assert READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) in body
# The truncated read is still a SUCCESSFUL call — cited in done.
# The truncated read is still a SUCCESSFUL call — it is the frame's
# only chip (phase 119, A1: read docs only; the suggested kubernetes
# doc was never read, so it is not on the wire).
done = frames[-1]
assert done["type"] == "done" and done["deflected"] is False
assert ("docs", "big-read.md") in [(s["source"], s["path"]) for s in done["sources"]]
assert ("docs", "homelab/kubernetes.md") in [
(s["source"], s["path"]) for s in done["sources"]
assert [(s["source"], s["path"]) for s in done["sources"]] == [
("docs", "big-read.md")
]
@@ -1159,11 +1163,12 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
) -> None:
"""(a) Grounded turn with tool calls: the event sequence is
``thinking?/tool/tool/delta…/done``; ``done.sources`` and the
``query_log`` row include the read document (deduped, order
preserved); the per-turn log line carries ``tool_calls=2``.
Phase 45: the agent loop keeps offering the tools for the whole
turn — the round cap (not per-tool budgets) is the bound."""
``thinking?/tool/tool/delta…/done``; ``done.sources`` is the read
document (phase 119, A1 — the read docs are the citation surface)
and the ``query_log`` row includes it (deduped, order preserved);
the per-turn log line carries ``tool_calls=2``. Phase 45: the
agent loop keeps offering the tools for the whole turn — the round
cap (not per-tool budgets) is the bound."""
scripted = FakeRagLLM(
tool_script=[
[
@@ -1212,12 +1217,12 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
done = frames[-1]
assert done["deflected"] is False
# done.sources = the retrieval docs + the read doc, deduped, order kept.
# Phase 119 (A1): done.sources = the agent's READ docs only — the
# read doc is the frame's only chip (the suggested retrieval docs
# are not on the wire; they stay in the durable record below).
sources = [(s["source"], s["path"]) for s in done["sources"]]
assert sources[-1] == ("docs", "homelab/backups.md") # the read doc is cited
assert ("docs", "homelab/kubernetes.md") in sources # …after the retrieval docs
assert len(sources) == len(set(sources)) # deduped by (source, path)
assert done["sources"][-1]["title"] == "Backup Strategy"
assert sources == [("docs", "homelab/backups.md")] # the read doc is cited
assert done["sources"][0]["title"] == "Backup Strategy"
# Phase 45: the tools stay offered on every request — the round cap
# (not spent budgets) bounds the loop, and the model answered while
@@ -1250,7 +1255,8 @@ def test_grounded_turn_streams_grep_tool_frames(
the raw pattern is the frame's ``argument`` (the UI renders the
"searching for" line from it). A non-string pattern — a model error
the backend refuses — yields ``argument: null``. A grep adds no
source: ``done.sources`` stays the retrieval docs (locked A5)."""
source (locked A5): with no read on the turn, ``done.sources`` is
empty (phase 119, A1)."""
scripted = FakeRagLLM(
tool_script=[
[
@@ -1292,9 +1298,8 @@ def test_grounded_turn_streams_grep_tool_frames(
assert [f for f in frames if f["type"] == "delta"]
done = frames[-1]
assert done["type"] == "done" and done["deflected"] is False
paths = [s["path"] for s in done["sources"]]
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
assert "homelab/backups.md" not in paths # a grep adds no source
# A grep adds no source (A5) and nothing was read (A1) ⇒ no chips.
assert done["sources"] == []
def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
@@ -1306,7 +1311,8 @@ def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
and frame emission is execution-independent: a rejected call (an
unknown ``read`` path) still streams its frame with the model's
argument as-is. The rejected read adds no source (``done.sources``
stays the retrieval docs), and rejected calls count nothing
carries no chip — the read failed and nothing else was read,
phase 119 A1), and rejected calls count nothing
(``tool_calls=1`` — only the executed scoped ``ls``)."""
# The scoped ``ls`` source-name check reads the registry — insert a
# row resolving to ``docs`` (the fixture's source name) and delete
@@ -1350,12 +1356,12 @@ def test_tool_frames_carry_the_model_arguments_regardless_of_execution(
# passed — frame emission is execution-independent.
assert read_frame["argument"] == "docs/homelab/nope.md"
# The rejected read adds no source — done.sources stays retrieval.
# The rejected read adds no source — and the read-only surface is
# empty on this turn (nothing was read; the refused read cites
# nothing, phase 119 A1).
done = frames[-1]
assert done["type"] == "done" and done["deflected"] is False
paths = [s["path"] for s in done["sources"]]
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
assert "homelab/nope.md" not in paths # the refused read cites nothing
assert done["sources"] == []
# The rejected call counts nothing — only the executed scoped ls.
lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
@@ -1462,9 +1468,10 @@ def test_zero_max_rounds_reproduce_pre_phase_single_request(
done = frames[-1]
assert done["type"] == "done"
assert done["deflected"] is False
paths = [s["path"] for s in done["sources"]]
assert "homelab/kubernetes.md" in paths # retrieval docs, unchanged
assert "homelab/backups.md" not in paths # nothing was read
# Phase 119 (A1): nothing was read ⇒ no chips (the kill-switch
# turn's citation surface is empty; the retrieval stays in the
# durable record below).
assert done["sources"] == []
# Exactly one request, and it carried no ``tools`` key at all — the
# scripted tool calls were never even offered a chance.
+40 -13
View File
@@ -25,15 +25,17 @@ from pathlib import Path
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import select, text
from sqlalchemy import text
from test_chat_api import FakeRagLLM, _stream_chat, _token_vec
from app.api import chat as chat_api
from app.config import get_settings
from app.main import app as fastapi_app
from app.models import Document, KbOverview
from app.rag.agent import suggested_folder_lines
from app.rag.importer import import_sources
from app.rag.prompts import build_deflect_prompt, build_high_prompt
from app.rag.retriever import retrieve, weak_hit_titles
from app.rag.retriever import retrieve, select_suggested, weak_hit_titles
from tests.conftest import ADMIN_PASSWORD
FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "docs"
@@ -90,15 +92,27 @@ def _seed_overview(db) -> None:
db.commit()
def _cited_docs(db, frames: list[dict]) -> list[Document]:
"""The documents the done event cited, in citation order — the same
list ``plan_turn`` passed to the prompt builder."""
docs = []
for s in frames[-1]["sources"]:
doc = db.scalar(select(Document).where(Document.path == s["path"]))
assert doc is not None, f"done source {s['path']!r} missing from the KB"
docs.append(doc)
return docs
def _suggested_docs(db) -> list[Document]:
"""The documents the grounded turn's HIGH prompt seeded, in the same
order ``plan_turn`` walked them — the SAME deterministic suggested
walk over the retrieval. Phase 119 (A1) moved the citation surface
to the agent's READ docs, so the seeded tier can no longer be read
off the done frame (the canned LLM here reads nothing ⇒
``done.sources`` is empty) — reconstructing from the retrieval is
the faithful source now."""
chunks = retrieve(db, QUESTION, _token_vec(QUESTION))
return list(select_suggested(chunks, n=get_settings().suggested_docs))
def _suggested_folder_lines(db) -> list[str]:
"""The suggested-folder context lines the endpoint computed (phase
119, D3, LOCKED A4) — the SAME deterministic suggested walk
``plan_turn`` performs internally (one extra walk, no shared
state), over the real seeded catalog."""
chunks = retrieve(db, QUESTION, _token_vec(QUESTION))
return suggested_folder_lines(
db, select_suggested(chunks, n=get_settings().suggested_docs)
)
def _turn_log_lines(caplog: pytest.LogCaptureFixture) -> list[str]:
@@ -124,7 +138,14 @@ def test_no_row_high_prompt_byte_identical_to_pre_phase(
assert frames[-1]["deflected"] is False
(system, user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1]
assert user["content"] == QUESTION
expected = build_high_prompt(_cited_docs(db, frames), notes=[], kb_overview=None)
# Phase 119 (D3): the endpoint's suggested-folder lines ride the
# HIGH prompt — reconstructed the same deterministic way.
expected = build_high_prompt(
_suggested_docs(db),
notes=[],
kb_overview=None,
folder_lines=_suggested_folder_lines(db),
)
assert system["content"] == expected
assert "<knowledge_base>" not in system["content"]
@@ -179,8 +200,14 @@ def test_row_high_prompt_carries_kb_section_exactly(
assert frames[-1]["deflected"] is False
(system, _user) = seeded_kb.seen_messages[0][0], seeded_kb.seen_messages[0][1]
# Phase 119 (D3): the suggested-folder lines ride the HIGH prompt
# alongside the <knowledge_base> section — reconstructed the same
# deterministic way.
expected = build_high_prompt(
_cited_docs(db, frames), notes=[], kb_overview=OVERVIEW
_suggested_docs(db),
notes=[],
kb_overview=OVERVIEW,
folder_lines=_suggested_folder_lines(db),
)
assert system["content"] == expected
+54 -15
View File
@@ -6,10 +6,12 @@ mapping with fake rows; this suite covers the SQL side on real
Postgres: the document-projection scan, the LATERAL representative-
chunk fetch (the ``is_summary`` chunk wins, chunk 0 otherwise, and a
chunk-less name match is EXCLUDED — the ``c.id IS NOT NULL`` guard),
the (count, length, catalog) ranking, the name-hits-lead-the-lexical-
list union with the FTS rows (chunk-id dedup), and the full
``retrieve()`` → ``select_documents()`` path putting the versioned-
name document into the seeded top-N.
the (matched-token count, catalog) ranking (phase 119, LOCKED A2 —
the two-class component rule: digit-bearing prefix, digitless exact,
titles never matched), the name-hits-lead-the-lexical-list union with
the FTS rows (chunk-id dedup), and the full ``retrieve()`` →
``select_documents()`` path putting the versioned-name document into
the seeded top-N.
Requires: ``podman compose up -d db``.
"""
@@ -137,19 +139,29 @@ def kb(db) -> Iterator[None]:
def test_name_hit_chunks_real_sql(kb, db) -> None:
"""Real Postgres: the projection scan finds exactly the qwen3.8
quadlet (the qwen3.6 sibling and the chunk-less name match are
excluded), and the LATERAL fetch hands back the SUMMARY chunk as
the representative (position −1, is_summary)."""
"""Real Postgres: the projection scan finds the qwen3.8 quadlet
(the ``qwen38`` stem prefix — the incident's original case) AND
the ``notes/llama.cpp.md`` doc (the question names "llama.cpp" —
the dotted token ``llamacpp`` exact-matches the file stem, the
phase-119 two-class rule). The qwen3.6 sibling (``qwen38`` is not
a prefix of ``qwen36…``) and the chunk-less name match are
excluded; the LATERAL fetch hands back the SUMMARY chunk as the
representative for the quadlet (position −1, is_summary) and chunk
0 for the single-chunk notes doc."""
out = _name_hit_chunks(db, INCIDENT_QUESTION)
assert [rc.document.path for rc in out] == [
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container"
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container",
"notes/llama.cpp.md", # the question names it — stem exact (A2)
]
rc = out[0]
assert rc.position == -1 # the summary chunk wins the LATERAL order
assert rc.is_summary is True
assert rc.fts_hit is True # the lexical signal — the A8 gate answers
assert rc.cosine == 0.0 # no vector rank on the name-hit row
rc_notes = out[1]
assert rc_notes.position == 0 # chunk 0 (no summary chunk)
assert rc_notes.is_summary is False
assert all(rc.name_hit is True for rc in out) # phase 119 — D2 bonus input
assert "qwen3.8-empty.container" not in [r.document.path for r in out] # chunk-less guard
@@ -163,13 +175,19 @@ def test_lexical_candidates_name_hit_leads_real_sql(kb, db) -> None:
assert paths[0] == (
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container"
)
assert paths[1] == "notes/llama.cpp.md" # the second name hit (stem exact)
# The FTS pollution is still present (the incident's shape) — but
# behind the name hit, no longer ahead of it.
# behind the name hits, no longer ahead of them.
assert (
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.6-27b-juggernaut-vulkan.container"
in paths
)
assert all(rc.fts_hit is True for rc in out)
# Phase 119: the two name-hit representative rows are flagged, the
# plain FTS rows are not.
assert out[0].name_hit is True
assert out[1].name_hit is True
assert all(not rc.name_hit for rc in out[2:])
def test_retrieve_selects_name_hit_doc_into_top_n(kb, db) -> None:
@@ -177,7 +195,12 @@ def test_retrieve_selects_name_hit_doc_into_top_n(kb, db) -> None:
fused) → ``select_documents`` puts the qwen3.8 quadlet in the
seeded top-N — the incident's seed miss (the two overview docs
only) is fixed. The question vector is parallel to the q38 chunk
embeddings (cosine 1.0), orthogonal to q36 (0.0)."""
embeddings (cosine 1.0), orthogonal to q36 (0.0).
Phase 119: the ``name_hit`` flag survives the fusion — the two
name-hit representative chunks are double hits (vector ∪ lexical),
and the double-hit merge ORs the flag into the surviving row;
the plain q36 vector+FTS row stays False."""
question_vec = _vec(1)
chunks = retrieve(db, INCIDENT_QUESTION, question_vec)
docs = select_documents(chunks, n=2)
@@ -185,16 +208,31 @@ def test_retrieve_selects_name_hit_doc_into_top_n(kb, db) -> None:
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container",
"notes/llama.cpp.md",
]
by_path: dict[str, list[bool]] = {}
for rc in chunks:
by_path.setdefault(rc.document.path, []).append(rc.name_hit)
# The q38 doc's summary chunk is the double-hit name hit (flagged);
# its plain chunk 0 (vector + FTS) is not — the doc has a flagged row.
assert any(by_path[
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.8-27b-juggernaut-vulkan.container"
])
assert any(by_path["notes/llama.cpp.md"]) # its only chunk is the name hit
assert not any(by_path[
"reeseapps/ai/deployments/juggernaut/quadlets/qwen3.6-27b-juggernaut-vulkan.container"
]) # plain vector+FTS — never a name hit
def test_name_hit_limit_real_sql(db) -> None:
"""Twelve identical (1, 6) name hits — the LATERAL fetch (and the
output) carries exactly ``NAME_HIT_LIMIT`` winners, catalog order."""
"""Twelve tied name hits (one matched token each — the ``qwen38``
stem PREFIX; the token must lead the stem, the old mid-stem
containment no longer matches) — the LATERAL fetch (and the
output) carries exactly ``NAME_HIT_LIMIT`` winners, catalog
order."""
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
for i in range(12):
doc = _doc(
db, "S", f"quadlets/m{i:02d}-qwen38.container", f"m{i:02d}-qwen38", "llama cpp qwen38\n"
db, "S", f"quadlets/qwen38-m{i:02d}.container", f"qwen38-m{i:02d}", "llama cpp qwen38\n"
)
db.flush()
c = _chunk(db, doc, 0, f"llama cpp qwen38 doc {i}")
@@ -203,5 +241,6 @@ def test_name_hit_limit_real_sql(db) -> None:
out = _name_hit_chunks(db, "what are the llama.cpp arguments for qwen 3.8")
assert len(out) == NAME_HIT_LIMIT
assert [rc.document.path for rc in out] == [
f"quadlets/m{i:02d}-qwen38.container" for i in range(NAME_HIT_LIMIT)
f"quadlets/qwen38-m{i:02d}.container" for i in range(NAME_HIT_LIMIT)
]
assert all(rc.name_hit is True for rc in out)
+36 -15
View File
@@ -12,13 +12,21 @@ margins below are exact floats, not flaky measurements.
**Measured geometry (recorded per task step 4/5):**
* Owner scenario — A (``backups/retention.md``, created 2020-01-01,
the exact answer, cosine 1.0) lands at vector rank 1 + FTS rank 1
(fused 0.03278689); B (``backups/retention-draft.md``, created
the exact answer, cosine 1.0) lands at vector rank 1 + lexical rank
3 (fused 0.03226646); B (``backups/retention-draft.md``, created
yesterday, the "under review, no decision yet" draft, cosine
0.707107) lands at vector rank 10 + FTS rank 3 (fused 0.03015873 —
a solid FTS hit at rank 3, as the task describes). Pre-boost fused
margin **A−B = 0.00262816** (asserted ≥ 3× the zero-age boost =
0.002100 at the default → ratio 1.25, the "comfortable margin").
0.707107) lands at vector rank 10 + lexical rank 1 (fused
0.03067915). Pre-boost fused margin **A−B = 1/63 − 1/70 =
0.00158730** — re-pinned by phase 119 (task 01): the two-class
name-hit rule (LOCKED A2) makes the digitless question token
"retention" exact-match the stem sub-component of all three
``retention*`` docs, so the lexical list now LEADS with those name
hits in catalog order (draft, review, retention) and B takes the
rank-1 lexical slot (pre-119 it was a plain FTS rank-1 / rank-3
hit). The flip-safety bar the margin is pinned against — margin >
the zero-age boost, so a zero-age B's full boost cannot close it
(A's own boost being ≈ 0 at its 2020 age) — still holds, at 2.27×
the default.
* Twin near-tie — C (``twin/c-older.md``, 2019) and D
(``twin/d-newer.md``, yesterday) with IDENTICAL chunk text and
near-identical vectors (cosine 1.0 vs 0.9999 — a literal identical
@@ -29,12 +37,16 @@ margins below are exact floats, not flaky measurements.
scale.
* The DEFAULT was tuned from the design starting point (0.001) down to
**0.0007** (task step 5: "tune the DEFAULTS … until old-correct wins
comfortably"): on the k=60 scale the owner scenario's margin is
comfortably"): on the k=60 scale the owner scenario's margin was
0.00262816 < 3×0.001, and a 0.001 zero-age boost (+0.000997 for a
yesterday doc) would have FLIPPED the pinned scenario. 0.0007 keeps
the flip margin comfortable (0.000698 > 0.00052882, lead
+0.000169) while staying 1.25× under the 3×-boost margin bar. The
owner re-tunes live via ``BOR_RECENCY_BOOST``.
+0.000169). Phase 119 (task 01) re-pinned the owner margin to
0.00158730 (above): still above the zero-age boost (2.27×), so the
worst-case flip — a zero-age B against A's ≈ 0 boost — keeps A
first; the twin-flip figures (gap 0.00052882, boost 0.000698) are
untouched (the twin KB has no name hits). The owner re-tunes live
via ``BOR_RECENCY_BOOST``.
Requires: ``podman compose up -d db``.
"""
@@ -239,11 +251,12 @@ def test_owner_scenario_old_correct_beats_new_similar(
owner_kb, db, monkeypatch: pytest.MonkeyPatch
) -> None:
"""THE owner scenario, pinned at the DEFAULTS: the older doc that
answers ranks above the newer similar one — AND the pre-boost fused
margin is ≥ 3× the zero-age boost (the "comfortable margin"; the
measured 0.00262816 vs the 0.0021 bar is recorded in the module
docstring). Re-pinned with the boost OFF: relevance alone already
ordered them (no regression — the boost is not what makes A win)."""
answers ranks above the newer similar one — AND the pre-boost
fused margin exceeds the zero-age boost (the flip-safety bar; the
phase-119-re-pinned measured 0.00158730 vs the 0.0007 default is
recorded in the module docstring). Re-pinned with the boost OFF:
relevance alone already ordered them (no regression — the boost is
not what makes A win)."""
chunks = retrieve(db, QUESTION, QUESTION_VEC)
assert select_documents(chunks, n=2)[0].path == "backups/retention.md"
@@ -254,7 +267,15 @@ def test_owner_scenario_old_correct_beats_new_similar(
fused = fuse(vector, lexical, s.rrf_k)
by_path = {rc.document.path: rc.score for rc in fused}
margin = by_path["backups/retention.md"] - by_path["backups/retention-draft.md"]
assert margin >= 3 * s.recency_boost
# Phase 119 (task 01) re-pinned geometry: the three ``retention*``
# docs are name hits (the digitless token "retention" exact-matches
# the stem sub-component), ranked in catalog order (draft, review,
# retention) — so A sits at vector rank 1 + lexical rank 3 and B
# at vector rank 10 + lexical rank 1: the margin is exactly
# 1/63 − 1/70. The flip-safety bar: a zero-age B's full boost
# cannot close it (A's own boost is ≈ 0 at its 2020 age).
assert margin == pytest.approx(1 / 63 - 1 / 70)
assert margin > s.recency_boost # 2.27× at the default
# The kill switch: A still first (relevance alone), and the
# weight-0 scores are the pre-phase fused scores byte-identical.