phase: 162_search_recall_small_sources
Build and Push Containers / build-and-push-app (push) Successful in 1m50s
Build and Push Containers / build-and-push-db (push) Successful in 14s

**Phase 162 — final verification pass (all tasks already complete; verified + docs pass)**

- Verified: `_fold` (NFKD+casefold, stdlib) inside `_normalize_name` — `BESTÅ`→`besta` question side, `BÅsta.md`→`basta` path side; A2 match classes/ranking/caps/bonus untouched; FTS/cosine/A8 untouched.
- Verified: `grep_document` folds both sides via the imported `_fold` (no second copy); regex-teaching lines byte-identical; 3 copy sites updated.
- Verified: `regenerate_overview` appends the deterministic `Sources: <name> (<k> doc|docs)` tail (happy path only; zero-doc + `LLMError` rows byte-identical; `build_overview_prompt`/`load_kb_overview`/`build_kb_section` untouched).
- Verified: `TOOLS_SECTION` grep sentence updated + D4 search-discipline sentence present **verbatim**, immediately after the grep sentence; `PERSONA`/LOW pins byte-identical (`test_prompt_lock.py`).
- Measured evidence (in `.agents/reports/162_…/task04_prefix_evidence_and_battery.md` + re-measured every run by `test_under_rank_is_measured_not_assumed`): pre-fix tokens `['build','best']` → IKEA doc rank 6 (outside top-5); post-fix `['build','besta']` → suggested #1, cosine unchanged 0.382.
- `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated); neighbours `test_kb_overview{,_freshness} test_name_signal_read_chips test_grep_regex_teaching test_harness_aligned_tools test_agent_document_tools test_tool_call_compact` → **24 passed**.
- `uv run pytest --cov=app --cov-report=term-missing` → **3756 passed**, app coverage **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**.
- Completion criteria: fold both ways ✅ · `Sources:` tail ✅ · TOOLS_SECTION + D4 verbatim + re-cut pins ✅ · E2E green/battery recorded/full suite/coverage/lint ✅ · docs pass ✅ · commit + dir move = harness.
- Battery: 5 phase-119 questions byte-identical to the phase-119 record (GATE PASS); the 2 Besta questions **skipped loudly** — verified the 2026-09-05 replica dump has zero `ikea` matches (`grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0); rank gate stays with the owner's live KB.
- Docs fixed: README — overview-tail doc + `BOR_KB_OVERVIEW_MAX_CHARS` tail-cut constraint, the phase-162 search-discipline paragraph, per-phase (not per-story) E2E wording, new "Retrieval regression battery" section; AGENTS.md — rule 4 per-phase wording + battery command in Quick reference. (The overview's expected "AGENTS.md §6 tool table" does not exist in this repo — the grep contract line lives in README's tool table, already updated.)
- Deviations (recorded, not assumed): `_fold` adds `casefold` (required for the pinned `ß → ss`); `①` NFKD-decomposes to `1` (the overview's parenthetical was wrong) and is excluded by `NAME_TOKEN_MIN_LEN`; pre-fix evidence lives in the reports dir because task files are read-only for this pass.
- Owner follow-up A6 (runtime, not code): add the Tuning-tab steering note "IKEA's BESTÅ assembly PDF is a live source — search it as `BESTÅ` or `AA-1272068`".
- Next pending phase: none — 162 is the only entry in `todo/` (harness moves it to `complete/`).
This commit is contained in:
2026-10-07 01:42:01 -04:00
parent 690fd98036
commit eebc824636
48 changed files with 2685 additions and 80 deletions
+28 -3
View File
@@ -367,7 +367,8 @@ def test_agent_tools_names_and_parameters() -> None:
# teaching no-match line instead of a trusted miss.
assert grep["description"] == (
"Search the indexed documents for an exact string "
"(case-insensitive) and return up to 20 matching lines "
"(case- and accent-insensitive — accents fold: `besta` "
"matches `BESTÅ`) and return up to 20 matching lines "
"as `source/path:line: text` — a locator, not a "
"context-adder: read the winner with `read` — for a "
"large document, use the reported line number as "
@@ -387,8 +388,11 @@ def test_agent_tools_names_and_parameters() -> None:
assert set(grep_params["properties"]) == {"pattern", "path"}
assert all(p["type"] == "string" for p in grep_params["properties"].values())
assert grep_params["properties"]["pattern"]["description"] == (
"The exact text to search for (a plain substring, "
"not a regex — no '.*', no '\\.', no character classes)"
"The exact text to search for (a plain "
"substring — case- and accent-insensitive, "
"accents fold: `besta` matches `BESTÅ` — "
"not a regex — no '.*', no "
"'\\.', no character classes)"
)
# Phase 72 (task 02): the bare-path contract is stated up front;
# task 05 (live gate iterations 1-8): the one-known-document clause
@@ -4000,6 +4004,27 @@ def test_origin_lines_image_doc_keeps_marker_position(
# ---------- grep (the phase-68 A5 contract under the new name) ----------
def test_grep_document_folds_accents_both_ways() -> None:
"""Phase 162 (LOCKED A3): grep folds BOTH the pattern and each line
through the shared ``app.rag.retriever._fold`` plus lowercase — an
ASCII pattern hits accented content and an accented pattern hits
ASCII content, one fold, never drifting from the name-hit path."""
# ASCII pattern, accented content.
assert agent.grep_document("The BESTÅ frame (AA-1272068-11-100)", "besta") == [
(1, "The BESTÅ frame (AA-1272068-11-100)")
]
assert agent.grep_document("line one\nBÅsta shelf\nline three", "basta") == [
(2, "BÅsta shelf")
]
assert agent.grep_document("café-au-lait", "cafe") == [(1, "café-au-lait")]
# Accented pattern, ASCII content — the pattern's own accents fold.
assert agent.grep_document("The besta frame", "BESTÅ") == [(1, "The besta frame")]
# A genuinely non-matching folded pattern still returns no lines.
assert agent.grep_document("The besta frame", "bestb") == []
# The fold bridges Ä/Ö/Ü/ß too (the same _fold as name matching).
assert agent.grep_document("Straße", "strasse") == [(1, "Straße")]
def test_grep_document_case_insensitive_line_numbers() -> None:
"""Case-insensitive fixed substring, 1-based line numbers, file order,
repeated matches within a line collapse to one match (grep semantics)."""
+144 -6
View File
@@ -5,7 +5,11 @@ per-document user lines (source/path/title/first summary line), and the
input cap with the shared ``[…truncated…]`` marker. The loader and
regenerator tests run against the local compose Postgres (preferred —
real single-row upsert), skipping with clear instructions when the stack
is not up — same pattern as ``test_importer.py``.
is not up — same pattern as ``test_importer.py``. The phase-162 section
pins the deterministic ``Sources: …`` inventory tail (LOCKED A2): the
format/order/counts, the 1-doc survival, the happy-path stored content,
the byte-identical zero-doc / ``LLMError`` paths, and the byte-identical
model input.
"""
from __future__ import annotations
@@ -15,7 +19,7 @@ import uuid
from datetime import UTC, datetime, timedelta
import pytest
from sqlalchemy import text
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.config import Settings, get_settings
@@ -25,6 +29,7 @@ from app.rag.overview import (
KB_OVERVIEW_MODE,
OVERVIEW_INSTRUCTION,
SYSTEM_PROMPT,
_source_inventory,
build_overview_prompt,
load_kb_overview,
regenerate_overview,
@@ -302,11 +307,13 @@ def test_regenerate_happy_path_creates_single_row_id_1(
row = db.get(KbOverview, 1)
assert row is not None, "the single row must be upserted"
assert row.id == 1
assert row.content == REPLY
# Phase 162 (LOCKED A2): the stored content carries the
# deterministic source-inventory tail after the model text.
assert row.content == REPLY + "\nSources: Homelab (2 docs)"
assert row.updated_at is not None
age = datetime.now(UTC) - row.updated_at
assert age.total_seconds() < 300, "updated_at must be a fresh UTC timestamp"
assert f"overview: regenerated docs=2 chars={len(REPLY)}" in caplog.text
assert f"overview: regenerated docs=2 chars={len(row.content)}" in caplog.text
finally:
_truncate_documents(db)
@@ -324,7 +331,7 @@ def test_regenerate_updates_existing_row_in_place(db: Session, clean_overview) -
assert count == 1, "still exactly one row after the re-generation"
row = db.get(KbOverview, 1)
assert row is not None
assert row.content == REPLY # replaced, not appended
assert row.content == REPLY + "\nSources: Homelab (1 doc)" # replaced, not appended
assert row.updated_at is not None and row.updated_at > past
finally:
_truncate_documents(db)
@@ -511,4 +518,135 @@ def test_regenerate_opens_own_session_when_none(db: Session, clean_overview) ->
assert ok is True
row = db.get(KbOverview, 1) # fresh lookup via the test session
assert row is not None and row.content == REPLY
assert row is not None and row.content == REPLY + "\nSources: Homelab (1 doc)"
# ---------- phase 162: the deterministic source inventory (LOCKED A2) -------
def test_source_inventory_format_counts_and_catalog_order() -> None:
"""Exact format, singular/plural, and catalog order: one entry per
DISTINCT source — ``<name> (<k> doc|docs)`` joined with ``, ``
behind ``Sources: ``, ordered ``source, path`` (the scrambled
input rows prove the catalog order is the OUTPUT order, not the
recency-first order of the rows the generator reads)."""
rows = [
("b", "x.md", "X", None),
("a", "y.md", "Y", None),
("c", "p1.md", "P1", None),
("b", "z.md", "Z", None),
("c", "p2.md", "P2", None),
("c", "p3.md", "P3", None),
]
assert _source_inventory(rows) == "Sources: a (1 doc), b (2 docs), c (3 docs)"
def test_source_inventory_keeps_a_one_doc_source_visible() -> None:
"""The 2026-10-07 ``ikea_instructions`` case: a 1-doc source among
many appears with ``(1 doc)`` — no source is invisible to
``<knowledge_base>``, whatever the ``lite`` outline says."""
rows = [
("gufo", "a.md", "A", None),
(
"ikea_instructions",
"besta-frame-white__AA-1272068-11-100.md",
"Assembly Instructions",
None,
),
("gufo", "b.md", "B", None),
("players_handbook", "core.md", "Core", None),
]
assert _source_inventory(rows) == (
"Sources: gufo (2 docs), ikea_instructions (1 doc), "
"players_handbook (1 doc)"
)
def test_regenerate_appends_the_inventory_tail_to_the_stored_content(
db: Session, clean_overview, caplog: pytest.LogCaptureFixture
) -> None:
"""Happy path, exact pin on the fake's fixed output: the stored row
= model text (trailing newlines rstripped) + ``\\n`` + the inventory
line; the log's chars = the FULL stored content, tail included."""
_truncate_documents(db)
try:
_add_doc(db, "players_handbook", "rpg.md", "RPG")
_add_doc(db, "Homelab", "b.md", "B")
_add_doc(db, "Homelab", "a.md", "A")
llm = _FakeLLM(reply=REPLY + "\n\n")
expected = REPLY + "\nSources: Homelab (2 docs), players_handbook (1 doc)"
with caplog.at_level(logging.INFO, logger="app.rag.overview"):
ok = asyncio.run(regenerate_overview(llm, db))
assert ok is True
row = db.get(KbOverview, 1)
assert row is not None
assert row.content == expected # no blank line before the tail
assert f"overview: regenerated docs=3 chars={len(expected)}" in caplog.text
finally:
_truncate_documents(db)
def test_regenerate_model_input_never_gains_the_inventory_tail(
db: Session, clean_overview
) -> None:
"""The model input is byte-identical to the pre-phase prompt: the
user message the fake received equals ``build_overview_prompt`` for
the same catalogue rows — the tail is never sent to ``lite``."""
_truncate_documents(db)
try:
_add_doc(db, "Homelab", "a.md", "A", "A summary.\nSource: Homelab/a.md")
_add_doc(db, "Deployments", "b.yaml", "B")
llm = _FakeLLM()
assert asyncio.run(regenerate_overview(llm, db)) is True
rows = db.execute(
select(Document.source, Document.path, Document.title, Document.summary)
.order_by(Document.created_at.desc(), Document.source, Document.path)
).all()
finally:
_truncate_documents(db)
user = next(m["content"] for m in llm.messages if m["role"] == "user")
_, expected_user = build_overview_prompt([tuple(r) for r in rows])
assert user == expected_user
assert "Sources:" not in user
def test_regenerate_zero_docs_keeps_tailless_pre_phase_row(
db: Session, clean_overview
) -> None:
"""Zero-doc early return: byte-identical pre-phase behavior — the
pre-phase (tail-less) row survives untouched, no tail is invented."""
db.execute(text("DELETE FROM kb_overview"))
pre_phase_row = "- Old outline\n- from before phase 162"
db.add(KbOverview(id=1, content=pre_phase_row))
db.commit()
_truncate_documents(db)
llm = _FakeLLM()
ok = asyncio.run(regenerate_overview(llm, db))
assert ok is False
assert llm.calls == 0
row = db.get(KbOverview, 1)
assert row is not None and row.content == pre_phase_row
assert "Sources:" not in row.content
def test_regenerate_llm_error_preserves_an_existing_inventory_tail(
db: Session, clean_overview
) -> None:
"""The tail is CONTENT: a previous row that already carried a
``Sources: …`` tail survives an ``LLMError`` byte-identically —
including the tail (an old outline is better than none)."""
db.execute(text("DELETE FROM kb_overview"))
previous = "- Old outline\nSources: Homelab (3 docs)"
db.add(KbOverview(id=1, content=previous))
db.commit()
_truncate_documents(db)
try:
_add_doc(db, "Homelab", "a.md", "A")
llm = _FakeLLM(fail=LLMError("simulated lite-model failure"))
ok = asyncio.run(regenerate_overview(llm, db))
assert ok is False
row = db.get(KbOverview, 1)
assert row is not None and row.content == previous
finally:
_truncate_documents(db)
+69 -16
View File
@@ -17,6 +17,7 @@ import hashlib
from app.rag.prompts import (
ORIGIN_RULE,
PERSONA,
SEARCH_DISCIPLINE_RULE,
SUGGEST_INTRO,
TOOLS_SECTION,
_base,
@@ -87,9 +88,34 @@ def test_high_and_low_bases_byte_locked() -> None:
#: inserted — the prefix (the ``ls``-clause opening) and the suffix
#: (the discipline-rules ending) survived byte-identical, so they are
#: the same anchors as pre-phase-145; the pre-phase-145 sha/length are
#: kept below for the diff-style pin.
TOOLS_SECTION_SHA256 = "41a84885be98d4e15b95205694d59d364296a30e27758d39ae737881d5bacaa7"
TOOLS_SECTION_LEN = 3279
#: kept below for the chained diff-style pin. RE-CUT again for phase 162
#: (task 03, LOCKED A3+A4): the ``grep`` sentence's ``(case-insensitive)``
#: parenthetical widened to the folded contract ``(case- and
#: accent-insensitive — accents fold: besta matches BESTÅ)``, and the
#: LOCKED search-discipline sentence (:data:`SEARCH_DISCIPLINE_RULE`) was
#: inserted immediately after the grep sentence (before the ``For a normal
#: search pass`` clause). Only those two byte-groups moved — the prefix
#: (the ``ls``-clause opening) and the suffix (the discipline-rules
#: ending) survive byte-identical; the pre-phase-162 sha/length are kept
#: below for the chained diff-style pin.
TOOLS_SECTION_SHA256 = "988642f9dc6bb2a4a1d44a83b5368ff4cdb9df4e181b2fe3dc2fb241b1857595"
TOOLS_SECTION_LEN = 3818
# The pre-phase-162 anchors — the section BEFORE phase 162's grep-fold +
# search-discipline insertion. Kept for the chained diff-style pin.
TOOLS_SECTION_PRE_PHASE_162_SHA256 = (
"41a84885be98d4e15b95205694d59d364296a30e27758d39ae737881d5bacaa7"
)
TOOLS_SECTION_PRE_PHASE_162_LEN = 3279
# The grep contract parenthetical exactly as it changed in phase 162
# (LOCKED A3): the pre-phase ``(case-insensitive)`` vs the folded one now
# embedded in ``TOOLS_SECTION``. Used to revert phase 162 for the diff pin.
_GREP_CONTRACT_PRE_PHASE_162 = "(case-insensitive)"
_GREP_CONTRACT_PHASE_162 = (
"(case- and accent-insensitive — accents fold: `besta` matches `BESTÅ`)"
)
TOOLS_SECTION_PREFIX = (
"<tools>\n"
"You may extend your context with three tools. `ls` lists the "
@@ -159,19 +185,46 @@ def test_tools_section_origin_rule_verbatim_at_its_position() -> None:
) in TOOLS_SECTION
def test_tools_section_origin_rule_is_the_only_byte_change() -> None:
"""Phase 145 (task 02) diff-style pin: the section's OTHER bytes are
byte-identical to pre-phase — the pre-phase section with ONLY the new
sentence-group inserted. Removing the rule (and the single space
that now separates it from the ``grep`` clause) reproduces the
pre-phase-145 section byte-for-byte (sha + length), and the length
delta is exactly the rule's length plus that one inserted space."""
assert len(TOOLS_SECTION) == (
TOOLS_SECTION_PRE_PHASE_145_LEN + len(ORIGIN_RULE) + 1
)
pre_phase = TOOLS_SECTION.replace(ORIGIN_RULE + " ", "")
assert len(pre_phase) == TOOLS_SECTION_PRE_PHASE_145_LEN
assert _sha256(pre_phase) == TOOLS_SECTION_PRE_PHASE_145_SHA256
def test_tools_section_search_discipline_rule_verbatim_at_its_position() -> None:
"""Phase 162 (task 03, LOCKED A4): the search-discipline rule is
LOCKED verbatim and sits at its EXACT position — immediately after
the grep sentence (the ``…retry with the plain text you expect to
see.`` clause) and BEFORE the ``For a normal search pass`` clause.
The rule + its exact boundaries are pinned as one contiguous span, so
a move or re-spelling of ANY byte fails."""
assert SEARCH_DISCIPLINE_RULE in TOOLS_SECTION
assert TOOLS_SECTION.count(SEARCH_DISCIPLINE_RULE) == 1 # exactly once
assert (
"retry with the plain text you expect to see. "
+ SEARCH_DISCIPLINE_RULE
+ " For a normal search pass only `pattern`"
) in TOOLS_SECTION
def test_tools_section_grep_fold_contract_present() -> None:
"""Phase 162 (task 03, LOCKED A3): the grep sentence now states the
FOLDED contract — the ``(case-insensitive)`` parenthetical is replaced
byte-for-byte by the folded one, and the old one is gone."""
assert _GREP_CONTRACT_PHASE_162 in TOOLS_SECTION
assert _GREP_CONTRACT_PRE_PHASE_162 not in TOOLS_SECTION
def test_tools_section_phase162_and_145_are_the_only_byte_changes() -> None:
"""Chained diff-style pin: the ONLY byte changes are phase 162's (the
folded grep contract + :data:`SEARCH_DISCIPLINE_RULE`) and phase 145's
(:data:`ORIGIN_RULE`). Reverting phase 162 (drop the rule + its
separating space, restore the pre-phase grep parenthetical) reproduces
the pre-phase-162 section byte-for-byte; reverting the origin rule on
top of that reproduces the pre-phase-145 section byte-for-byte."""
# Revert phase 162 -> pre-phase-162.
pre_162 = TOOLS_SECTION.replace(SEARCH_DISCIPLINE_RULE + " ", "")
pre_162 = pre_162.replace(_GREP_CONTRACT_PHASE_162, _GREP_CONTRACT_PRE_PHASE_162)
assert len(pre_162) == TOOLS_SECTION_PRE_PHASE_162_LEN
assert _sha256(pre_162) == TOOLS_SECTION_PRE_PHASE_162_SHA256
# Revert phase 145 -> pre-phase-145.
pre_145 = pre_162.replace(ORIGIN_RULE + " ", "")
assert len(pre_145) == TOOLS_SECTION_PRE_PHASE_145_LEN
assert _sha256(pre_145) == TOOLS_SECTION_PRE_PHASE_145_SHA256
# ---------- the LOW prompt's locked DEFLECT_MODE body ----------
+19 -10
View File
@@ -50,6 +50,7 @@ from app.models import Document
from app.rag.prompts import (
ORIGIN_RULE,
PERSONA,
SEARCH_DISCIPLINE_RULE,
SHOW_IMAGE_CLAUSE,
SUGGEST_INTRO,
TOOLS_SECTION,
@@ -286,14 +287,14 @@ def test_tools_section_markers_and_new_tool_names() -> None:
def test_tools_section_teaches_the_harness_shapes() -> None:
"""Copy pins: ``ls``'s phase-63 catalog-line format, ``grep``'s
case-insensitive exact-string locator contract (up to 20
case- and accent-insensitive exact-string locator contract (up to 20
``source/path:line: text`` lines, a locator not a context-adder),
and ``read``'s combined ``source/path`` + full content. (Phase 72:
the source-name scope clause and the combined-identity clause are
pinned byte-for-byte in
:func:`test_tools_section_phase72_contract_clauses`.)"""
assert "source: X | path: Y | title: Z" in TOOLS_SECTION
assert "case-insensitive" in TOOLS_SECTION
assert "case- and accent-insensitive" in TOOLS_SECTION # phase 162 fold
assert "up to 20" in TOOLS_SECTION
assert "source/path:line: text" in TOOLS_SECTION
assert "locator, not a context-adder" in TOOLS_SECTION
@@ -414,20 +415,28 @@ def test_tools_section_phase118_summary_seed_read_clause() -> None:
"| path: Y | title: Z`; to find one specific document without "
"listing, use `grep`."
) in TOOLS_SECTION
# The grep clause — byte-identical.
# The grep clause — folded contract (phase 162, LOCKED A3), up to the
# search-discipline sentence, byte-identical.
assert (
"`grep` locates an exact string (case-insensitive) in the "
"`grep` locates an exact string (case- and accent-insensitive — "
"accents fold: `besta` matches `BESTÅ`) in the "
"indexed documents and returns up to 20 matching "
"`source/path:line: text` lines — a locator, not a "
"context-adder: read the winner with `read`. A grep pattern "
"is a plain substring, NEVER a regex — '.*' and '\\.' are "
"literal text there; if such a pattern returns no matches, "
"retry with the plain text you expect to see. For a normal "
"search pass only `pattern` — its optional `path` argument "
"limits the search to one document you already know, by the "
"same combined `source/path` string; never a source name — a "
"bare document path (without the source name) will not "
"resolve there either."
"retry with the plain text you expect to see. "
) in TOOLS_SECTION
# The LOCKED search-discipline sentence sits immediately after the
# grep sentence, before the path clause (phase 162, LOCKED A4).
assert (
"retry with the plain text you expect to see. "
+ SEARCH_DISCIPLINE_RULE
+ " For a normal search pass only `pattern` — its optional "
"`path` argument limits the search to one document you already "
"know, by the same combined `source/path` string; never a "
"source name — a bare document path (without the source name) "
"will not resolve there either."
) in TOOLS_SECTION
# The discipline rules — byte-identical.
assert (
+153
View File
@@ -1346,3 +1346,156 @@ def test_bonus_lives_in_the_selection_layer_only() -> None:
rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks
}
assert before == after
# ---------------------------------------------------------------------------
# Phase 162, D1 — the accent-folded name normalization (LOCKED A1): the
# 2026-10-07 live gap, the ASCII filename vs the accented content layer
# (``besta`` <-> ``BESTÅ``, both directions, ONE function)
# ---------------------------------------------------------------------------
from app.rag.retriever import _fold, _name_parts # noqa: E402
def test_fold_accent_ligature_and_sharp_s_cases() -> None:
"""The shared accent fold (phase 162, LOCKED A1): NFKD decomposition,
combining marks dropped — ``Å/å → a``, ``Ä/Ö/Ü → a/o/u`` (the letter
survives, the mark drops), ``ß → ss`` (the Unicode fold —
``str.casefold``; NFKD alone leaves ``ß`` intact), the ``fi`` ligature
``→ fi``, ``ç → c``."""
assert _fold("BESTÅ") == "besta"
assert _fold("Åsa") == "asa"
assert _fold("ÄÖÜäöü") == "aouaou"
assert _fold("Straße") == "strasse"
assert _fold("fine") == "fine"
assert _fold("garçon") == "garcon"
def test_fold_survives_non_decomposing_characters() -> None:
"""Characters with NO decomposition (CJK) survive the fold unchanged —
the caller's alnum scan drops them, exactly as pre-phase. Compat-only
characters (``①``) NFKD to their ASCII base (``1``); that base is
below :data:`NAME_TOKEN_MIN_LEN`, so it never becomes a name token
(the name-signal behavior for such characters is unchanged)."""
assert _fold("漢字") == "漢字"
assert _fold("①") == "1"
def test_normalize_name_is_accent_folded() -> None:
"""``_normalize_name`` = the alnum scan of ``_fold(s.lower())`` —
``Qwen 3.8 → qwen38`` UNCHANGED, ``BESTÅ → besta`` NEW (pre-phase it
was ``best``); non-folding characters still drop out (``漢字 → ""``)."""
assert _normalize_name("Qwen 3.8") == "qwen38"
assert _normalize_name("BESTÅ") == "besta"
assert _normalize_name("BÅsta") == "basta"
assert _normalize_name("straße") == "strasse"
assert _normalize_name("fine") == "fine"
assert _normalize_name("garçon") == "garcon"
assert _normalize_name("漢字") == ""
# Compat-only base survives (``①`` → ``1``) but is a 1-char token —
# below the minimum, never a name candidate (see the token test).
assert _normalize_name("①") == "1"
def test_name_hit_tokens_accented_question_bridges() -> None:
"""The question side of the bridge: the 2026-10-07 battery question
produces the ``besta`` token (pre-phase: ``best`` — the exact list is
pinned; ``how``/``do``/``i``/``the`` are below NAME_TOKEN_MIN_LEN).
The fold covers every pinned case: ``BESTÅ``, umlauts, ``ß``, the
``fi`` ligature, ``ç``."""
assert name_hit_tokens("How do I build the BESTÅ?") == ["build", "besta"]
assert name_hit_tokens("how do I fine the Über strasse") == [
"fine", "uber", "strasse",
]
assert name_hit_tokens("MÖRK ÖVER BÄSTA GARÇON") == [
"mork", "over", "basta", "garcon",
]
def test_name_hit_tokens_min_length_on_the_folded_form() -> None:
"""NAME_TOKEN_MIN_LEN is measured on the FOLDED form (``BÄST`` →
``bast`` (4, kept) while ``BÄ`` → ``ba`` (2, dropped)), and
non-folding tokens still drop out: ``漢字`` normalizes to ``""`` and
``①`` to the 1-char ``1`` — neither can ever be a candidate. The
``1`` is compat-decomposition (NFKD — LOCKED A1): the ``①`` token
itself never survives, and the only footprint a digit-base can leave
is a versioned-JOIN artifact (``the`` + ``1`` → ``the1``) whose sole
match route is PREFIXING a normalized path part — no real path does,
so the name signal for such characters is unchanged."""
assert name_hit_tokens("BÄST BÄ") == ["bast"]
assert name_hit_tokens("tell me about it ① 漢字") == ["tell", "about"]
# The compat-decomposed ``1`` acts as a version token for the join —
# pinned as the real behavior (a prefix-class candidate no path hits).
assert name_hit_tokens("tell me about the ① 漢字") == [
"tell", "about", "the1",
]
def test_name_parts_fold_both_the_ascii_and_accented_stem() -> None:
"""The path side of the bridge: the ASCII IKEA filename yields
``besta`` in the EQUAL set (the stem sub-component — the phase-119
A2 case, unchanged), and the accented filename ``BÅsta.md`` yields
``basta`` (the folded stem — already in the equal set via the
normalized-stem member, no _name_parts code change)."""
_prefix, equal = _name_parts(
"ikea/besta-frame-white__AA-1272068-11-100.pdf"
)
assert "besta" in equal # sub-component, unchanged
_prefix2, equal2 = _name_parts("manuals/BÅsta.md")
assert "basta" in equal2 # folded stem — an ASCII question matches it
def test_name_hit_chunks_accented_question_hits_ascii_doc() -> None:
"""The bridge, end to end (direction 1): the accented question
``BESTÅ`` name-hits the ASCII-stem document — the token ``besta``
EQUALS the stem sub-component of
``besta-frame-white__AA-1272068-11-100`` (the 2026-10-07 live miss;
pre-phase the token was ``best`` and the hit was impossible)."""
ikea = _doc(
"ikea/besta-frame-white__AA-1272068-11-100.pdf",
"MONTERING BÄSTA",
source="ikea_instructions",
)
distractor = _doc("notes/meeting.md", "we discussed besta", source="Brain-of-Reese")
session = _FakeSession(
[_name_row(ikea), _name_row(distractor)],
[_name_hit_lateral_row(ikea)],
)
out = _name_hit_chunks(session, "How do I build the BESTÅ?") # pyright: ignore[reportArgumentType]
assert [rc.document.path for rc in out] == [ikea.path]
assert out[0].name_hit is True
assert out[0].fts_hit is True
assert out[0].cosine == 0.0
def test_name_hit_chunks_ascii_question_hits_accented_doc() -> None:
"""The bridge, end to end (direction 2): the ASCII question token
``basta`` name-hits the ACCENTED filename ``BÅsta.md`` — the folded
stem enters the equal set as ``basta`` (pre-phase the accented stem
normalized to ``bsta`` — the ``å`` simply dropped out of the alnum
scan — so the match was impossible)."""
accented = _doc("manuals/BÅsta.md", "MONTERING", source="ikea_instructions")
miss = _doc("manuals/hesta.md", "MONTERING", source="ikea_instructions")
session = _FakeSession(
[_name_row(accented), _name_row(miss)],
[_name_hit_lateral_row(accented)],
)
out = _name_hit_chunks(session, "How do I assemble the basta shelf?") # pyright: ignore[reportArgumentType]
assert [rc.document.path for rc in out] == [accented.path]
assert out[0].name_hit is True
def test_name_hit_chunks_fold_preserves_digitless_precision() -> None:
"""The fold does NOT relax the A2 match classes: the accented token
``déploy`` folds to ``deploy`` and STILL does not hit the
``Deployments/`` document (the part normalizes to ``deployments`` ≠
``deploy`` — titles are never matched, digits never enter); only the
exact ``gitea`` component hits."""
precision = _doc("Deployments/reeseapps/README.md", "body", title="Deployments")
gitea = _doc("notes/gitea.md", "body")
session = _FakeSession(
[_name_row(precision), _name_row(gitea)],
[_name_hit_lateral_row(gitea)],
)
out = _name_hit_chunks(session, "how do I déploy gitea") # pyright: ignore[reportArgumentType]
assert [rc.document.path for rc in out] == [gitea.path]