phase: 119_name_signal_read_chips
All verification complete. Final report: **Phase 119 final verification pass — all criteria verified, one stale pin fixed.** - Verified implementation of all 6 tasks: D1 component name-hit rule (`name_hit` flag, titles never matched, retired length tie-break), D2 `BOR_NAME_HIT_BONUS` (0.005 default, 0 = byte-identical kill switch, negative fails startup, selection-layer only, `eval_retrieval` `suggested:` line), D3 suggested-folder lines (after `SUGGEST_INTRO`, before first block), D4 cite-discipline `SUGGEST_INTRO` sentence (PERSONA/LOW/`TOOLS_SECTION` byte-pins intact), D5 `done.sources` = read docs only (frontend no-op on empty confirmed), D6 mock `repeat your folder map` echo + new suite + telemetry. - Battery (replica restored per skill, fingerprint docs=1000/chunks=8866 verified, `eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` re-run): **GATE PASS** — gitea README #4 in suggested top-5, forgejo 5/5 (README #1), gateway README in top-5 (#4), qwen3.8-27b quadlets top-5, Mongolia HIGH/fts=5 unchanged. - New E2E in isolation: `4 passed` ×2 (deterministic). All 27 modified E2E suites in isolation: 26 green; **1 stale pin fixed** — `test_source_chip_quality.py` durable-record order pin pre-dated the D1 re-rank (`aliases` stem sub-component name-hits `ssh_aliases.txt`, deterministically lifting `backups.md` over `kubernetes.md`; probe-verified 0.016277 vs 0.016036, 4/4 stable) — re-pinned with the phase-119 rationale; suite green ×2. - Gates: `uv run pytest --cov=app --cov-report=term-missing` → **2547 passed, app coverage 99%** (>90%); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Completion criteria: 1 ✅ (battery, recorded), 2 ✅ (folder lines; block/LOW byte-identical pins green), 3 ✅ (read-only chips, zero-read chips nothing, related row + durable record untouched — unit+E2E agree), 4 ✅ (all green), 5 → commit/phase-move left to the harness per pass rules (nothing committed). - Deviations: battery output + real-model telemetry recorded in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md` and `TOOL_CALLING_TESTING.md` §11 (task files in `complete/` are immutable to this pass); gateway canonical doc at #4 vs overview's #3 was already documented at task 06 (containment gate met). - Next pending phase: **none** — `todo/` holds only phase 119.
This commit is contained in:
+449
-45
@@ -721,14 +721,15 @@ def _lexical_row(is_summary: bool, doc_path: str) -> object:
|
||||
def test_lexical_candidates_carry_is_summary_flag() -> None:
|
||||
"""The lexical list reads ``c.is_summary`` from the raw row.
|
||||
|
||||
The question carries no digit-bearing name token (no bare, no
|
||||
numeric-join), so the name-hit path issues NO queries at all — the
|
||||
single FTS rowset answers the only (FTS) call, and the list is the
|
||||
plain FTS rows: the pre-name-hit behavior, unchanged.
|
||||
The question's tokens are name candidates (class-agnostic, phase
|
||||
119), so the name-hit projection runs — but the (empty) catalog
|
||||
yields no path match, the LATERAL fetch is skipped, and the FTS
|
||||
rowset answers the second call: the list is the plain FTS rows,
|
||||
every one ``name_hit=False``.
|
||||
"""
|
||||
rows = [_lexical_row(True, "summary-src.yaml"), _lexical_row(False, "other.md")]
|
||||
out = _lexical_candidates(
|
||||
_FakeSession(rows), "how do i configure the thing", limit=10 # pyright: ignore[reportArgumentType]
|
||||
_FakeSession([], rows), "how do i configure the thing", limit=10 # pyright: ignore[reportArgumentType]
|
||||
)
|
||||
assert len(out) == 2
|
||||
by_path = {rc.document.path: rc for rc in out}
|
||||
@@ -736,6 +737,7 @@ def test_lexical_candidates_carry_is_summary_flag() -> None:
|
||||
assert by_path["summary-src.yaml"].position == -1
|
||||
assert by_path["other.md"].is_summary is False
|
||||
assert all(rc.fts_hit is True for rc in out)
|
||||
assert all(rc.name_hit is False for rc in out) # ordinary FTS rows
|
||||
|
||||
|
||||
def test_fuse_keeps_is_summary_on_double_hit() -> None:
|
||||
@@ -783,30 +785,54 @@ def test_normalize_name() -> None:
|
||||
|
||||
|
||||
def test_name_hit_tokens_incident_question() -> None:
|
||||
"""The incident question yields EXACTLY the versioned join
|
||||
``qwen38`` — the token the document names actually carry. Plain
|
||||
prose words (``what``, ``llamacpp``, ``arguments``, ``server`` —
|
||||
no digit) never name-match (the precision guard); the single
|
||||
digits ("3", "8") and the bare "38" are < 4 chars; the
|
||||
digit-leading ``38show`` boundary artifact is dropped."""
|
||||
"""Phase 119 (LOCKED A2): the candidate list is CLASS-AGNOSTIC —
|
||||
every normalized token of length >= 4 (dotted kept whole:
|
||||
``llama.cpp`` → ``llamacpp``) plus the versioned join ``qwen38``.
|
||||
The digit distinction moved to the match side (:func:`_name_hit_chunks`) —
|
||||
prose precision now comes from the match class (a digitless token
|
||||
must EQUAL a whole path component). The single digits ("3", "8")
|
||||
and the bare "38" are < 4 chars; the digit-leading ``38show``
|
||||
boundary artifact cannot survive (the join only fires on a purely
|
||||
numeric SECOND token)."""
|
||||
tokens = name_hit_tokens(INCIDENT_QUESTION)
|
||||
assert tokens == ["qwen38"]
|
||||
for absent in ("what", "qwen", "llamacpp", "arguments", "3", "8", "38", "38show", "server"):
|
||||
assert tokens == ["what", "correct", "llamacpp", "arguments", "qwen", "qwen38"]
|
||||
for absent in ("3", "8", "38", "38show", "server"):
|
||||
assert absent not in tokens
|
||||
|
||||
|
||||
def test_name_hit_tokens_no_digit_question_returns_empty() -> None:
|
||||
"""A question with no digit-bearing token (bare or joined) yields
|
||||
no name candidates — prose joins like ``correctllama`` never count."""
|
||||
assert name_hit_tokens("what is the correct caddy config") == []
|
||||
def test_name_hit_tokens_digitless_question_yields_long_tokens() -> None:
|
||||
"""A question with NO digit-bearing token still yields candidates
|
||||
(every normalized token of length >= 4) — the 2026-09-16 fix:
|
||||
product names without digits ("gitea", "gateway") must get a name
|
||||
signal. Prose joins (``correctcaddy``) never count (the second
|
||||
token is not purely numeric)."""
|
||||
assert name_hit_tokens("what is the correct caddy config") == [
|
||||
"what", "correct", "caddy", "config",
|
||||
]
|
||||
assert name_hit_tokens("a e i o u 3 8") == []
|
||||
|
||||
|
||||
def test_name_hit_tokens_bare_digit_bearing_token() -> None:
|
||||
"""A single written token that carries a digit (``1panel``) is a
|
||||
name candidate on its own — no join needed."""
|
||||
name candidate on its own — no join needed — alongside the plain
|
||||
prose tokens of the same question (class-agnostic list)."""
|
||||
tokens = name_hit_tokens("what is my 1panel dashboard setup")
|
||||
assert tokens == ["1panel"]
|
||||
assert tokens == ["what", "1panel", "dashboard", "setup"]
|
||||
|
||||
|
||||
def test_name_hit_tokens_versioned_join_and_short_tokens() -> None:
|
||||
"""The versioned join survives the class-agnostic change ("Qwen
|
||||
3.8" → ``qwen38``), and short tokens (< :data:`NAME_TOKEN_MIN_LEN`
|
||||
normalized — the single digits, "3.8" → ``38``) never become
|
||||
candidates, with or without a join."""
|
||||
tokens = name_hit_tokens("help me with Qwen 3.8 please")
|
||||
# The join is appended at its FIRST token's position (after "qwen").
|
||||
assert tokens == ["help", "with", "qwen", "qwen38", "please"]
|
||||
assert name_hit_tokens("3.8 8 16 9") == [] # 38 / 8 / 16 / 9 / 816 / 169 all < 4
|
||||
# Word-after-version: the word itself is a candidate, but the
|
||||
# digit-leading join artifact ("38show") cannot survive (the join
|
||||
# only fires on a purely numeric SECOND token).
|
||||
assert name_hit_tokens("3.8 show") == ["show"]
|
||||
|
||||
|
||||
def _name_row(doc: Document) -> tuple:
|
||||
@@ -834,7 +860,8 @@ def _name_hit_lateral_row(doc: Document, is_summary: bool = False) -> SimpleName
|
||||
|
||||
|
||||
def test_name_hit_chunks_no_tokens_skips_all_queries() -> None:
|
||||
"""A question with no name tokens issues no queries at all."""
|
||||
"""A question with no name tokens (every normalized token < 4)
|
||||
issues no queries at all."""
|
||||
session = _FakeSession([]) # any call would surface a statement
|
||||
assert _name_hit_chunks(session, "a e i o u 3 8") == [] # pyright: ignore[reportArgumentType]
|
||||
assert session.statements == []
|
||||
@@ -850,46 +877,124 @@ def test_name_hit_chunks_no_matching_doc_returns_empty() -> None:
|
||||
assert len(session.statements) == 1 # projection only — no LATERAL fetch
|
||||
|
||||
|
||||
def test_name_hit_chunks_ranked_by_count_length_catalog() -> None:
|
||||
"""A two-candidate question (``qwen38`` + ``1panel``): the document
|
||||
whose name carries BOTH (2 matches, 12 total chars) leads; the two
|
||||
single-match documents tie on (1, 6) and fall to catalog order
|
||||
(``dashboards/1panel-notes.md`` before ``quadlets/qwen3.8…``).
|
||||
Hits carry ``fts_hit=True`` (the A8 gate answers), ``cosine=0.0``,
|
||||
and the summary flag of their representative chunk."""
|
||||
both = _doc("dashboards/1panel-qwen3.8.md", "body", title="1Panel Qwen 3.8")
|
||||
panel = _doc("dashboards/1panel-notes.md", "body")
|
||||
q38 = _doc("quadlets/qwen3.8-27b-juggernaut-vulkan.container", "body")
|
||||
name_rows = [_name_row(d) for d in (panel, both, q38)] # catalog order
|
||||
question = "what are the correct llama.cpp arguments for qwen 3.8 and the 1panel dashboard?"
|
||||
def test_name_hit_chunks_digitless_exact_part_stem_subcomponent() -> None:
|
||||
"""A DIGITLESS token EQUALS a normalized path part (the ``gitea/``
|
||||
folder), the file stem (``gitea.md``), or a stem sub-component
|
||||
(``kubernetes_gitea``, ``gitea-values``, ``test-gateway`` — the
|
||||
stem split on non-alphanumeric runs) — the 2026-09-16 product-name
|
||||
signal (LOCKED A2)."""
|
||||
question = "how do i set up gitea or the gateway" # tokens: [gitea, gateway]
|
||||
docs = [
|
||||
_doc("deploy/reeseapps/gitea/README.md", "body"), # path part
|
||||
_doc("notes/gitea.md", "body"), # file stem
|
||||
_doc("deploy/k8s/kubernetes_gitea.md", "body"), # sub-component
|
||||
_doc("deploy/k8s/gitea-values.yaml", "body"), # sub-component
|
||||
_doc("deploy/istio/test-gateway.yaml", "body"), # sub-component (gateway)
|
||||
_doc("notes/gitlab.md", "body"), # NO component matches — excluded
|
||||
]
|
||||
name_rows = [_name_row(d) for d in docs]
|
||||
session = _FakeSession(name_rows, [_name_hit_lateral_row(d) for d in docs[:5]])
|
||||
out = _name_hit_chunks(session, question) # pyright: ignore[reportArgumentType]
|
||||
# Five one-token hits, catalog order (source, path):
|
||||
assert [rc.document.path for rc in out] == [
|
||||
"deploy/istio/test-gateway.yaml",
|
||||
"deploy/k8s/gitea-values.yaml",
|
||||
"deploy/k8s/kubernetes_gitea.md",
|
||||
"deploy/reeseapps/gitea/README.md",
|
||||
"notes/gitea.md",
|
||||
]
|
||||
assert all(rc.name_hit is True for rc in out)
|
||||
assert all(rc.fts_hit is True for rc in out) # the lexical signal
|
||||
assert all(rc.cosine == 0.0 for rc in out) # no vector rank
|
||||
|
||||
|
||||
def test_name_hit_chunks_digitless_title_never_matched() -> None:
|
||||
"""The owner-verified failure mode of the naive relaxation: a doc
|
||||
under a ``Deployments/`` folder titled "Deployments" does NOT hit
|
||||
the common token ``deploy`` (the part normalizes to
|
||||
``deployments`` ≠ ``deploy``), and a doc titled "Gitea" with no
|
||||
gitea path component does NOT hit ``gitea`` — TITLES ARE NEVER
|
||||
MATCHED (LOCKED A2)."""
|
||||
question = "how do i deploy gitea" # tokens: [deploy, gitea]
|
||||
docs = [
|
||||
_doc("Deployments/reeseapps/README.md", "body", title="Deployments"),
|
||||
_doc("notes/internal-notes.md", "body", title="Gitea"), # title only
|
||||
]
|
||||
session = _FakeSession([_name_row(d) for d in docs], [])
|
||||
assert _name_hit_chunks(session, question) == [] # pyright: ignore[reportArgumentType]
|
||||
assert len(session.statements) == 1 # projection only — no LATERAL fetch
|
||||
|
||||
|
||||
def test_name_hit_chunks_digit_bearing_prefix_not_midword() -> None:
|
||||
"""A DIGIT-BEARING token is a PREFIX of a normalized part or stem
|
||||
(``qwen38`` → ``qwen3.8-27b-epic-vulkan.container``) — a stem that
|
||||
merely CONTAINS the token mid-word (``xqwen38y…``) does NOT hit;
|
||||
sub-components are in the exact-match class only (LOCKED A2)."""
|
||||
question = "what are the arguments for qwen 3.8" # tokens: what, arguments, qwen, qwen38
|
||||
hit = _doc("quadlets/qwen3.8-27b-epic-vulkan.container", "body")
|
||||
miss = _doc("quadlets/xqwen38y-test.container", "body") # mid-word containment
|
||||
name_rows = [_name_row(hit), _name_row(miss)]
|
||||
session = _FakeSession(name_rows, [_name_hit_lateral_row(hit)])
|
||||
out = _name_hit_chunks(session, question) # pyright: ignore[reportArgumentType]
|
||||
assert [rc.document.path for rc in out] == [hit.path]
|
||||
assert out[0].name_hit is True
|
||||
|
||||
|
||||
def test_name_hit_chunks_ranked_by_count_then_catalog() -> None:
|
||||
"""A question (``deploy`` + ``gitea`` + ``qwen`` + ``qwen38``):
|
||||
the document whose path carries BOTH a digitless component and a
|
||||
digit-bearing prefix (2 matched tokens) leads; the two
|
||||
single-token documents tie on count and fall to CATALOG ORDER —
|
||||
the old total-matched-length tie-break is RETIRED (it would have
|
||||
put the 6-char ``qwen38`` hit, ``quadlets/…``, before the 5-char
|
||||
``gitea`` hit, ``gitea/notes.md`` — the flip is pinned). The
|
||||
"Deployments"-titled doc and the title-only "Gitea" doc never
|
||||
appear (titles are never matched)."""
|
||||
precision = _doc("Deployments/reeseapps/README.md", "body", title="Deployments")
|
||||
gitea_notes = _doc("gitea/notes.md", "body", title="Internal notes")
|
||||
both = _doc("gitea/qwen3.8-model.container", "body", title="The model quadlet")
|
||||
q38 = _doc("quadlets/qwen3.8-27b-juggernaut-vulkan.container", "body", title="juggernaut")
|
||||
title_only = _doc("notes/internal-notes.md", "body", title="Gitea")
|
||||
name_rows = [_name_row(d) for d in (precision, gitea_notes, both, q38, title_only)]
|
||||
question = "how do i deploy gitea with qwen 3.8"
|
||||
lateral_rows = [
|
||||
_name_hit_lateral_row(q38, is_summary=True), # LATERAL may return any order
|
||||
_name_hit_lateral_row(both),
|
||||
_name_hit_lateral_row(panel),
|
||||
_name_hit_lateral_row(gitea_notes),
|
||||
]
|
||||
session = _FakeSession(name_rows, lateral_rows)
|
||||
out = _name_hit_chunks(session, question) # pyright: ignore[reportArgumentType]
|
||||
assert [rc.document.path for rc in out] == [
|
||||
"dashboards/1panel-qwen3.8.md", # 2 matched tokens — leads
|
||||
"dashboards/1panel-notes.md", # (1, 6) — catalog order
|
||||
"quadlets/qwen3.8-27b-juggernaut-vulkan.container", # (1, 6) — after
|
||||
"gitea/qwen3.8-model.container", # 2 matched tokens (gitea + qwen38) — leads
|
||||
"gitea/notes.md", # 1 token (gitea, 5 chars) — catalog order beats quadlets
|
||||
"quadlets/qwen3.8-27b-juggernaut-vulkan.container", # 1 token (qwen38, 6 chars)
|
||||
]
|
||||
assert all(rc.name_hit is True for rc in out)
|
||||
assert all(rc.fts_hit is True for rc in out) # the lexical signal
|
||||
assert all(rc.cosine == 0.0 for rc in out) # no vector rank
|
||||
assert all(rc.score == 0.0 for rc in out) # fuse fills the score
|
||||
by_path = {rc.document.path: rc for rc in out}
|
||||
# The representative chunk keeps its summary flag (the LATERAL
|
||||
# choice: is_summary DESC, position ASC — chunk 0 otherwise).
|
||||
assert by_path["quadlets/qwen3.8-27b-juggernaut-vulkan.container"].is_summary is True
|
||||
assert by_path["quadlets/qwen3.8-27b-juggernaut-vulkan.container"].position == -1
|
||||
assert by_path["dashboards/1panel-notes.md"].is_summary is False
|
||||
assert by_path["gitea/notes.md"].is_summary is False
|
||||
|
||||
|
||||
def test_name_hit_chunks_short_tokens_never_hit() -> None:
|
||||
"""Short tokens (< 4 normalized — "3.8" → ``38``, the single
|
||||
digits) are never candidates, so they can never hit, with or
|
||||
without the versioned join."""
|
||||
session = _FakeSession([]) # any call would surface a statement
|
||||
assert _name_hit_chunks(session, "3.8 8 16 9") == [] # pyright: ignore[reportArgumentType]
|
||||
assert session.statements == []
|
||||
|
||||
|
||||
def test_name_hit_chunks_capped_at_limit() -> None:
|
||||
"""Twelve tied name hits (one matched token each) yield exactly
|
||||
``NAME_HIT_LIMIT`` of them — catalog order (the deterministic
|
||||
tie-break)."""
|
||||
docs = [_doc(f"quadlets/m{i:02d}.container", "body") for i in range(12)]
|
||||
for d in docs: # give every document a name that carries the token
|
||||
d.title = "qwen38 model i"
|
||||
"""Twelve tied name hits (one matched token each — the ``qwen38``
|
||||
stem prefix) yield exactly ``NAME_HIT_LIMIT`` of them — catalog
|
||||
order (the deterministic tie-break)."""
|
||||
docs = [_doc(f"quadlets/qwen3.8-m{i:02d}.container", "body") for i in range(12)]
|
||||
name_rows = [_name_row(d) for d in docs]
|
||||
# Only the ten winners (catalog order — the deterministic tie-break
|
||||
# of the twelve identical scores) reach the LATERAL fetch; the fake
|
||||
@@ -900,7 +1005,10 @@ def test_name_hit_chunks_capped_at_limit() -> None:
|
||||
session, "tell me about the qwen 3.8 models" # pyright: ignore[reportArgumentType]
|
||||
)
|
||||
assert len(out) == NAME_HIT_LIMIT
|
||||
assert [rc.document.path for rc in out] == [f"quadlets/m{i:02d}.container" for i in range(10)]
|
||||
assert [rc.document.path for rc in out] == [
|
||||
f"quadlets/qwen3.8-m{i:02d}.container" for i in range(10)
|
||||
]
|
||||
assert all(rc.name_hit is True for rc in out)
|
||||
|
||||
|
||||
def test_lexical_candidates_name_hits_lead_and_dedupe_with_fts() -> None:
|
||||
@@ -933,7 +1041,7 @@ def test_lexical_candidates_name_hits_lead_and_dedupe_with_fts() -> None:
|
||||
session = _FakeSession(name_rows, lateral_rows, fts_rows)
|
||||
out = _lexical_candidates(session, INCIDENT_QUESTION, limit=10) # pyright: ignore[reportArgumentType]
|
||||
assert len(out) == 3 # q38 (once), other (name hit), other (FTS chunk)
|
||||
# Both name hits tie on (1, 6) — catalog order: "qwen3." (ASCII 46)
|
||||
# Both name hits tie on count (1) — catalog order: "qwen3." (ASCII 46)
|
||||
# sorts before "qwen38" (ASCII 56).
|
||||
assert out[0].document.path == "quadlets/qwen3.8-27b-juggernaut-vulkan.container"
|
||||
assert out[1].document.path == "quadlets/qwen38-other.container"
|
||||
@@ -942,3 +1050,299 @@ def test_lexical_candidates_name_hits_lead_and_dedupe_with_fts() -> None:
|
||||
rc.chunk_id for rc in out
|
||||
} == {q38_chunk, fts_rows[1].chunk_id, lateral_rows[1].chunk_id}
|
||||
assert all(rc.fts_hit is True for rc in out)
|
||||
# Phase 119: the name-hit representative rows are flagged, the plain
|
||||
# FTS row is not (the selection tier's bonus input, task 02).
|
||||
assert out[0].name_hit is True
|
||||
assert out[1].name_hit is True
|
||||
assert out[2].name_hit is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 119, D1 — the name_hit flag through fusion
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_fuse_keeps_name_hit_on_lexical_only_hit() -> None:
|
||||
"""A name-hit row with no vector rank keeps ``name_hit=True``
|
||||
through the fusion (the ``replace()`` copy carries the field)."""
|
||||
nh = _rc("gitea/README.md")
|
||||
nh.name_hit = True
|
||||
out = fuse([], [nh], k=60)
|
||||
assert len(out) == 1
|
||||
assert out[0].name_hit is True
|
||||
assert out[0].fts_hit is True
|
||||
assert out[0].cosine == 0.0
|
||||
|
||||
|
||||
def test_fuse_or_s_name_hit_on_double_hit() -> None:
|
||||
"""A vector row that is ALSO the name hit's representative chunk
|
||||
(the RRF merge dedupes by chunk id) keeps ``name_hit=True`` — the
|
||||
merge ORs the flag in, so the selection tier (task 02) still sees
|
||||
the name hit on the fused list."""
|
||||
v = _rc("gitea/README.md", cosine=0.9)
|
||||
l1 = _rc("gitea/README.md", cosine=0.1) # the lexical copy of the same chunk
|
||||
l1.chunk_id = v.chunk_id
|
||||
l1.name_hit = True
|
||||
out = fuse([v], [l1], k=60)
|
||||
assert len(out) == 1
|
||||
assert out[0].name_hit is True
|
||||
assert out[0].fts_hit is True
|
||||
assert out[0].score == pytest.approx(2 / 61)
|
||||
|
||||
|
||||
def test_fuse_default_name_hit_stays_false_for_ordinary_rows() -> None:
|
||||
"""Neither list flagged ⇒ fusion never invents a name-hit flag —
|
||||
ordinary vector and FTS rows are ``name_hit=False``."""
|
||||
out = fuse([_rc("a.md", cosine=0.8)], [_rc("b.md", fts_hit=True)], k=60)
|
||||
assert len(out) == 2
|
||||
assert all(rc.name_hit is False for rc in out)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 119, D2 — the bounded name-hit bonus (LOCKED A3)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
from app.rag.retriever import _selection_order, weak_hit_titles # noqa: E402
|
||||
|
||||
|
||||
def _bonus_rc(
|
||||
doc: Document,
|
||||
score: float,
|
||||
cosine: float,
|
||||
position: int = 0,
|
||||
name_hit: bool = False,
|
||||
) -> RetrievedChunk:
|
||||
"""One fused-list candidate (name-hit rows follow the D1 lexical
|
||||
convention: ``cosine=0.0``, ``fts_hit=True``)."""
|
||||
return RetrievedChunk(
|
||||
chunk_id=uuid.uuid4(),
|
||||
position=position,
|
||||
content=doc.content[:20],
|
||||
score=score,
|
||||
document=doc,
|
||||
cosine=cosine,
|
||||
fts_hit=cosine == 0.0,
|
||||
name_hit=name_hit,
|
||||
)
|
||||
|
||||
|
||||
def _bonus_chunks() -> list[RetrievedChunk]:
|
||||
"""A mixed FUSED list — already in the ``fuse()`` key order
|
||||
(−score, −cosine, path, position) — with one name-hit document
|
||||
(``gitea/README.md``, the D1 convention: cosine 0.0) and ordinary
|
||||
vector/FTS documents: ``a.md`` carries two chunks, and ``b.md`` /
|
||||
``c.md`` tie on the fused score (separated only by cosine).
|
||||
|
||||
The pre-phase (bonus-0) document order this list walks — the golden
|
||||
the kill switch must reproduce — is a (0.0200) → gitea (0.0160) →
|
||||
b (0.0150, cos 0.7) → c (0.0150, cos 0.6) → d (0.0100).
|
||||
"""
|
||||
gitea = _doc("gitea/README.md", "G" * 50)
|
||||
a = _doc("a.md", "A" * 50)
|
||||
b = _doc("b.md", "B" * 50)
|
||||
c = _doc("c.md", "C" * 50)
|
||||
d = _doc("d.md", "D" * 50)
|
||||
return [
|
||||
_bonus_rc(a, 0.0200, 0.9, 0),
|
||||
_bonus_rc(gitea, 0.0160, 0.0, 0, name_hit=True),
|
||||
_bonus_rc(b, 0.0150, 0.7, 0),
|
||||
_bonus_rc(c, 0.0150, 0.6, 0),
|
||||
_bonus_rc(a, 0.0120, 0.5, 1),
|
||||
_bonus_rc(d, 0.0100, 0.1, 0),
|
||||
]
|
||||
|
||||
|
||||
#: The golden document order the OLD pre-phase loop (stable score-
|
||||
#: descending walk, first-seen-chunk dedupe) produces over
|
||||
#: :func:`_bonus_chunks` — pinned byte-identical by the kill switch.
|
||||
GOLDEN_PRE_PHASE_ORDER = ["a.md", "gitea/README.md", "b.md", "c.md", "d.md"]
|
||||
|
||||
|
||||
def test_selection_order_bonus_zero_is_the_pre_phase_golden_walk() -> None:
|
||||
"""LOCKED A3 kill switch: ``bonus=0`` returns the EXACT pre-phase
|
||||
document order of the old score-descending first-seen walk — the
|
||||
golden list pinned from the old loop over the mixed fused list
|
||||
(incl. the b/c fused-score tie resolved by the input order the
|
||||
fusion produced — the walk never re-sorts it away)."""
|
||||
out = _selection_order(_bonus_chunks(), 0.0)
|
||||
assert [d.path for d, _eff, _cos, _idx in out] == GOLDEN_PRE_PHASE_ORDER
|
||||
# The re-rank inputs are exposed and exact: effective == best fused
|
||||
# score (no bonus), best cosine tracked across a doc's chunks (a: 0.9
|
||||
# from its rank-1 chunk, not 0.5), first-seen index in the
|
||||
# score-descending walk.
|
||||
assert [eff for _d, eff, _cos, _idx in out] == [
|
||||
0.0200, 0.0160, 0.0150, 0.0150, 0.0100
|
||||
]
|
||||
assert [cos for _d, _eff, cos, _idx in out] == [
|
||||
pytest.approx(v) for v in (0.9, 0.0, 0.7, 0.6, 0.1)
|
||||
]
|
||||
assert [idx for _d, _eff, _cos, idx in out] == [0, 1, 2, 3, 5]
|
||||
|
||||
|
||||
def test_selection_order_bonus_inert_without_name_hits() -> None:
|
||||
"""No name-hit chunk present → the bonus cannot fire: the order is
|
||||
IDENTICAL to the pre-phase walk even with the default bonus on
|
||||
(LOCKED A3)."""
|
||||
chunks = _bonus_chunks()
|
||||
for rc in chunks:
|
||||
rc.name_hit = False
|
||||
out = _selection_order(chunks, 0.005)
|
||||
assert [d.path for d, _eff, _cos, _idx in out] == GOLDEN_PRE_PHASE_ORDER
|
||||
|
||||
|
||||
def test_selection_order_bonus_lifts_name_hit_doc_below_bonus_gap() -> None:
|
||||
"""The name-hit doc's effective 0.016 + 0.005 = 0.021 EXCEEDS a's
|
||||
0.020 — a gap of 0.004 < bonus 0.005 — so the bonus lifts it to
|
||||
rank 1; the rest keep their fused order (the bonus re-ranks, it
|
||||
does not inflate)."""
|
||||
out = _selection_order(_bonus_chunks(), 0.005)
|
||||
assert [d.path for d, _eff, _cos, _idx in out] == [
|
||||
"gitea/README.md", "a.md", "b.md", "c.md", "d.md"
|
||||
]
|
||||
assert out[0][1] == pytest.approx(0.016 + 0.005)
|
||||
|
||||
|
||||
def test_selection_order_bonus_does_not_lift_above_bonus_gap() -> None:
|
||||
"""A gap LARGER than the bonus is not closed: the name-hit doc's
|
||||
best 0.010 + 0.005 = 0.015 ties b/c on effective and LOSES to both
|
||||
on the (−effective, −best_cosine) tie-break (its D1 cosine is 0.0) —
|
||||
a keeps the lead (0.020). A second name-hit doc (``e.md``, also
|
||||
0.010/cos 0.0) trails gitea on the ``document.path`` tie-break —
|
||||
the full re-rank key pinned."""
|
||||
chunks = _bonus_chunks()
|
||||
chunks[1].score = 0.010 # the name-hit doc drops to a 0.010 best
|
||||
e = _doc("e.md", "E" * 50)
|
||||
chunks.insert(2, _bonus_rc(e, 0.010, 0.0, 0, name_hit=True))
|
||||
out = _selection_order(chunks, 0.005)
|
||||
assert [d.path for d, _eff, _cos, _idx in out] == [
|
||||
"a.md", "b.md", "c.md", "e.md", "gitea/README.md", "d.md"
|
||||
]
|
||||
|
||||
|
||||
def test_selection_order_first_seen_breaks_equal_path_ties() -> None:
|
||||
"""Two documents sharing a path across sources (``notes.md`` in two
|
||||
sources) can tie on (effective, cosine, path) — the pre-bonus
|
||||
first-seen rank decides (the last key element)."""
|
||||
s1 = _doc("notes.md", "X" * 50, source="Src1")
|
||||
s2 = _doc("notes.md", "Y" * 50, source="Src2")
|
||||
chunks = [
|
||||
_bonus_rc(s1, 0.016, 0.0, 0, name_hit=True),
|
||||
_bonus_rc(s2, 0.016, 0.0, 0, name_hit=True),
|
||||
]
|
||||
out = _selection_order(chunks, 0.005)
|
||||
assert [d.source for d, *_ in out] == ["Src1", "Src2"]
|
||||
|
||||
|
||||
def test_selection_order_bonus_applied_once_per_document() -> None:
|
||||
"""The bonus is per DOCUMENT — applied ONCE no matter how many of
|
||||
the doc's chunks are name hits (3×bonus would push the name-hit doc
|
||||
above the 0.030 leader; one bonus cannot)."""
|
||||
gitea = _doc("gitea/README.md", "G" * 50)
|
||||
a = _doc("a.md", "A" * 50)
|
||||
chunks = [
|
||||
_bonus_rc(a, 0.030, 0.8),
|
||||
_bonus_rc(gitea, 0.016, 0.0, 0, name_hit=True),
|
||||
_bonus_rc(gitea, 0.010, 0.0, 1, name_hit=True),
|
||||
_bonus_rc(gitea, 0.008, 0.0, 2, name_hit=True),
|
||||
]
|
||||
out = _selection_order(chunks, 0.005)
|
||||
assert [d.path for d, _eff, _cos, _idx in out] == ["a.md", "gitea/README.md"]
|
||||
assert out[1][1] == pytest.approx(0.016 + 0.005) # best + ONE bonus
|
||||
|
||||
|
||||
def test_selection_order_bonus_fires_when_name_hit_is_not_first_chunk() -> None:
|
||||
"""The bonus fires on ANY name-hit chunk of the document — including
|
||||
when the doc's first-seen (best) chunk is an ordinary vector row and
|
||||
only a lower-ranked chunk is the D1 name-hit representative (a
|
||||
document's name hit and its best chunk can be different chunks).
|
||||
The bonus still lands on the doc's BEST fused score, and the doc's
|
||||
best cosine stays tracked across ALL its chunks."""
|
||||
gitea = _doc("gitea/README.md", "G" * 50)
|
||||
a = _doc("a.md", "A" * 50)
|
||||
chunks = [
|
||||
_bonus_rc(a, 0.024, 0.8),
|
||||
_bonus_rc(gitea, 0.020, 0.2, 0), # the doc's best — an ordinary chunk
|
||||
_bonus_rc(gitea, 0.016, 0.55, 2, name_hit=True), # the name-hit rep
|
||||
]
|
||||
assert [d.path for d, *_ in _selection_order(chunks, 0.0)] == [
|
||||
"a.md", "gitea/README.md"
|
||||
]
|
||||
out = _selection_order(chunks, 0.005)
|
||||
assert [d.path for d, *_ in out] == ["gitea/README.md", "a.md"]
|
||||
assert out[0][1] == pytest.approx(0.020 + 0.005) # bonus on the BEST score
|
||||
# The doc's best cosine is tracked across ALL its chunks — the
|
||||
# lower-ranked name-hit chunk (0.55) beats the first-seen chunk's
|
||||
# 0.2 (the re-rank tie-break input).
|
||||
assert out[0][2] == pytest.approx(0.55)
|
||||
|
||||
|
||||
def test_suggested_bonus_default_from_settings_and_kill_switch(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The *bonus* parameter defaults to the LIVE ``BOR_NAME_HIT_BONUS``
|
||||
setting (the ``n`` parameter's settings-read pattern — the default
|
||||
0.005 is the production value, not a frozen constant); an explicit
|
||||
``bonus=0`` and a settings kill switch both reproduce the pre-phase
|
||||
golden walk (LOCKED A3)."""
|
||||
chunks = _bonus_chunks()
|
||||
settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||||
assert settings.name_hit_bonus == 0.005 # the production default
|
||||
monkeypatch.setattr(retriever, "get_settings", lambda: settings)
|
||||
assert [d.path for d in select_suggested(chunks, n=5)] == [
|
||||
"gitea/README.md", "a.md", "b.md", "c.md", "d.md"
|
||||
]
|
||||
# Explicit kill switch: the byte-identical pre-phase order.
|
||||
assert [d.path for d in select_suggested(chunks, n=5, bonus=0.0)] == GOLDEN_PRE_PHASE_ORDER
|
||||
# Settings kill switch (BOR_NAME_HIT_BONUS=0) — the same golden walk.
|
||||
off = Settings(_env_file=None, name_hit_bonus=0.0) # pyright: ignore[reportCallIssue]
|
||||
monkeypatch.setattr(retriever, "get_settings", lambda: off)
|
||||
assert [d.path for d in select_suggested(chunks, n=5)] == GOLDEN_PRE_PHASE_ORDER
|
||||
|
||||
|
||||
def test_related_skips_excluded_ids_under_the_bonus() -> None:
|
||||
"""Exclusion is orthogonal to the bonus: excluded ids are skipped
|
||||
exactly as before, on the bonus-adjusted walk — an excluded doc
|
||||
never rides the related row even when the bonus would lift it to
|
||||
the lead."""
|
||||
chunks = _bonus_chunks()
|
||||
b_id = chunks[2].document.id
|
||||
d_id = chunks[5].document.id
|
||||
out = select_related(chunks, {b_id, d_id}, cap=5, bonus=0.005)
|
||||
assert [d.path for d in out] == ["gitea/README.md", "a.md", "c.md"]
|
||||
# Kill switch: the same exclusions on the pre-phase walk.
|
||||
out0 = select_related(chunks, {b_id, d_id}, cap=5, bonus=0.0)
|
||||
assert [d.path for d in out0] == ["a.md", "gitea/README.md", "c.md"]
|
||||
# The name-hit doc itself excluded → the lead goes to the next doc.
|
||||
g_id = chunks[1].document.id
|
||||
out2 = select_related(chunks, {g_id}, cap=5, bonus=0.005)
|
||||
assert [d.path for d in out2][0] == "a.md"
|
||||
|
||||
|
||||
def test_weak_hit_titles_bonus_adjusted_order() -> None:
|
||||
"""Titles follow the bonus-adjusted selection walk (``_doc`` titles
|
||||
equal paths here, so the title list mirrors the doc order); the
|
||||
kill switch returns the pre-phase golden order."""
|
||||
chunks = _bonus_chunks()
|
||||
assert weak_hit_titles(chunks, bonus=0.005) == [
|
||||
"gitea/README.md", "a.md", "b.md", "c.md", "d.md"
|
||||
]
|
||||
assert weak_hit_titles(chunks, bonus=0.0) == GOLDEN_PRE_PHASE_ORDER
|
||||
|
||||
|
||||
def test_bonus_lives_in_the_selection_layer_only() -> None:
|
||||
"""LOCKED A3: the bonus never touches the chunk objects —
|
||||
``score``/``cosine``/``fts_hit`` (the A8 gate's inputs —
|
||||
``query_log.top_score`` is the best fused chunk score, the same
|
||||
values) are unchanged after every selection walk, even with the
|
||||
bonus lifting a document."""
|
||||
chunks = _bonus_chunks()
|
||||
before = {
|
||||
rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks
|
||||
}
|
||||
select_suggested(chunks, n=5, bonus=0.005)
|
||||
select_related(chunks, set(), cap=5, bonus=0.005)
|
||||
weak_hit_titles(chunks, bonus=0.005)
|
||||
after = {
|
||||
rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks
|
||||
}
|
||||
assert before == after
|
||||
|
||||
Reference in New Issue
Block a user