phase: 162_search_recall_small_sources

**Phase 162 — final verification pass (all tasks already complete; verified + docs pass)**

- Verified: `_fold` (NFKD+casefold, stdlib) inside `_normalize_name` — `BESTÅ`→`besta` question side, `BÅsta.md`→`basta` path side; A2 match classes/ranking/caps/bonus untouched; FTS/cosine/A8 untouched.
- Verified: `grep_document` folds both sides via the imported `_fold` (no second copy); regex-teaching lines byte-identical; 3 copy sites updated.
- Verified: `regenerate_overview` appends the deterministic `Sources: <name> (<k> doc|docs)` tail (happy path only; zero-doc + `LLMError` rows byte-identical; `build_overview_prompt`/`load_kb_overview`/`build_kb_section` untouched).
- Verified: `TOOLS_SECTION` grep sentence updated + D4 search-discipline sentence present **verbatim**, immediately after the grep sentence; `PERSONA`/LOW pins byte-identical (`test_prompt_lock.py`).
- Measured evidence (in `.agents/reports/162_…/task04_prefix_evidence_and_battery.md` + re-measured every run by `test_under_rank_is_measured_not_assumed`): pre-fix tokens `['build','best']` → IKEA doc rank 6 (outside top-5); post-fix `['build','besta']` → suggested #1, cosine unchanged 0.382.
- `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated); neighbours `test_kb_overview{,_freshness} test_name_signal_read_chips test_grep_regex_teaching test_harness_aligned_tools test_agent_document_tools test_tool_call_compact` → **24 passed**.
- `uv run pytest --cov=app --cov-report=term-missing` → **3756 passed**, app coverage **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**.
- Completion criteria: fold both ways ✅ · `Sources:` tail ✅ · TOOLS_SECTION + D4 verbatim + re-cut pins ✅ · E2E green/battery recorded/full suite/coverage/lint ✅ · docs pass ✅ · commit + dir move = harness.
- Battery: 5 phase-119 questions byte-identical to the phase-119 record (GATE PASS); the 2 Besta questions **skipped loudly** — verified the 2026-09-05 replica dump has zero `ikea` matches (`grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0); rank gate stays with the owner's live KB.
- Docs fixed: README — overview-tail doc + `BOR_KB_OVERVIEW_MAX_CHARS` tail-cut constraint, the phase-162 search-discipline paragraph, per-phase (not per-story) E2E wording, new "Retrieval regression battery" section; AGENTS.md — rule 4 per-phase wording + battery command in Quick reference. (The overview's expected "AGENTS.md §6 tool table" does not exist in this repo — the grep contract line lives in README's tool table, already updated.)
- Deviations (recorded, not assumed): `_fold` adds `casefold` (required for the pinned `ß → ss`); `①` NFKD-decomposes to `1` (the overview's parenthetical was wrong) and is excluded by `NAME_TOKEN_MIN_LEN`; pre-fix evidence lives in the reports dir because task files are read-only for this pass.
- Owner follow-up A6 (runtime, not code): add the Tuning-tab steering note "IKEA's BESTÅ assembly PDF is a live source — search it as `BESTÅ` or `AA-1272068`".
- Next pending phase: none — 162 is the only entry in `todo/` (harness moves it to `complete/`).

tests/unit/test_retriever.py

@@ -1346,3 +1346,156 @@ def test_bonus_lives_in_the_selection_layer_only() -> None:
         rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks
     }
     assert before == after
+
+
+# ---------------------------------------------------------------------------
+# Phase 162, D1 — the accent-folded name normalization (LOCKED A1): the
+# 2026-10-07 live gap, the ASCII filename vs the accented content layer
+# (``besta`` <-> ``BESTÅ``, both directions, ONE function)
+# ---------------------------------------------------------------------------
+
+from app.rag.retriever import _fold, _name_parts  # noqa: E402
+
+
+def test_fold_accent_ligature_and_sharp_s_cases() -> None:
+    """The shared accent fold (phase 162, LOCKED A1): NFKD decomposition,
+    combining marks dropped — ``Å/å → a``, ``Ä/Ö/Ü → a/o/u`` (the letter
+    survives, the mark drops), ``ß → ss`` (the Unicode fold —
+    ``str.casefold``; NFKD alone leaves ``ß`` intact), the ``fi`` ligature
+    ``→ fi``, ``ç → c``."""
+    assert _fold("BESTÅ") == "besta"
+    assert _fold("Åsa") == "asa"
+    assert _fold("ÄÖÜäöü") == "aouaou"
+    assert _fold("Straße") == "strasse"
+    assert _fold("fine") == "fine"
+    assert _fold("garçon") == "garcon"
+
+
+def test_fold_survives_non_decomposing_characters() -> None:
+    """Characters with NO decomposition (CJK) survive the fold unchanged —
+    the caller's alnum scan drops them, exactly as pre-phase. Compat-only
+    characters (``①``) NFKD to their ASCII base (``1``); that base is
+    below :data:`NAME_TOKEN_MIN_LEN`, so it never becomes a name token
+    (the name-signal behavior for such characters is unchanged)."""
+    assert _fold("漢字") == "漢字"
+    assert _fold("①") == "1"
+
+
+def test_normalize_name_is_accent_folded() -> None:
+    """``_normalize_name`` = the alnum scan of ``_fold(s.lower())`` —
+    ``Qwen 3.8 → qwen38`` UNCHANGED, ``BESTÅ → besta`` NEW (pre-phase it
+    was ``best``); non-folding characters still drop out (``漢字 → ""``)."""
+    assert _normalize_name("Qwen 3.8") == "qwen38"
+    assert _normalize_name("BESTÅ") == "besta"
+    assert _normalize_name("BÅsta") == "basta"
+    assert _normalize_name("straße") == "strasse"
+    assert _normalize_name("fine") == "fine"
+    assert _normalize_name("garçon") == "garcon"
+    assert _normalize_name("漢字") == ""
+    # Compat-only base survives (``①`` → ``1``) but is a 1-char token —
+    # below the minimum, never a name candidate (see the token test).
+    assert _normalize_name("①") == "1"
+
+
+def test_name_hit_tokens_accented_question_bridges() -> None:
+    """The question side of the bridge: the 2026-10-07 battery question
+    produces the ``besta`` token (pre-phase: ``best`` — the exact list is
+    pinned; ``how``/``do``/``i``/``the`` are below NAME_TOKEN_MIN_LEN).
+    The fold covers every pinned case: ``BESTÅ``, umlauts, ``ß``, the
+    ``fi`` ligature, ``ç``."""
+    assert name_hit_tokens("How do I build the BESTÅ?") == ["build", "besta"]
+    assert name_hit_tokens("how do I fine the Über strasse") == [
+        "fine", "uber", "strasse",
+    ]
+    assert name_hit_tokens("MÖRK ÖVER BÄSTA GARÇON") == [
+        "mork", "over", "basta", "garcon",
+    ]
+
+
+def test_name_hit_tokens_min_length_on_the_folded_form() -> None:
+    """NAME_TOKEN_MIN_LEN is measured on the FOLDED form (``BÄST`` →
+    ``bast`` (4, kept) while ``BÄ`` → ``ba`` (2, dropped)), and
+    non-folding tokens still drop out: ``漢字`` normalizes to ``""`` and
+    ``①`` to the 1-char ``1`` — neither can ever be a candidate. The
+    ``1`` is compat-decomposition (NFKD — LOCKED A1): the ``①`` token
+    itself never survives, and the only footprint a digit-base can leave
+    is a versioned-JOIN artifact (``the`` + ``1`` → ``the1``) whose sole
+    match route is PREFIXING a normalized path part — no real path does,
+    so the name signal for such characters is unchanged."""
+    assert name_hit_tokens("BÄST BÄ") == ["bast"]
+    assert name_hit_tokens("tell me about it ① 漢字") == ["tell", "about"]
+    # The compat-decomposed ``1`` acts as a version token for the join —
+    # pinned as the real behavior (a prefix-class candidate no path hits).
+    assert name_hit_tokens("tell me about the ① 漢字") == [
+        "tell", "about", "the1",
+    ]
+
+
+def test_name_parts_fold_both_the_ascii_and_accented_stem() -> None:
+    """The path side of the bridge: the ASCII IKEA filename yields
+    ``besta`` in the EQUAL set (the stem sub-component — the phase-119
+    A2 case, unchanged), and the accented filename ``BÅsta.md`` yields
+    ``basta`` (the folded stem — already in the equal set via the
+    normalized-stem member, no _name_parts code change)."""
+    _prefix, equal = _name_parts(
+        "ikea/besta-frame-white__AA-1272068-11-100.pdf"
+    )
+    assert "besta" in equal  # sub-component, unchanged
+    _prefix2, equal2 = _name_parts("manuals/BÅsta.md")
+    assert "basta" in equal2  # folded stem — an ASCII question matches it
+
+
+def test_name_hit_chunks_accented_question_hits_ascii_doc() -> None:
+    """The bridge, end to end (direction 1): the accented question
+    ``BESTÅ`` name-hits the ASCII-stem document — the token ``besta``
+    EQUALS the stem sub-component of
+    ``besta-frame-white__AA-1272068-11-100`` (the 2026-10-07 live miss;
+    pre-phase the token was ``best`` and the hit was impossible)."""
+    ikea = _doc(
+        "ikea/besta-frame-white__AA-1272068-11-100.pdf",
+        "MONTERING BÄSTA",
+        source="ikea_instructions",
+    )
+    distractor = _doc("notes/meeting.md", "we discussed besta", source="Brain-of-Reese")
+    session = _FakeSession(
+        [_name_row(ikea), _name_row(distractor)],
+        [_name_hit_lateral_row(ikea)],
+    )
+    out = _name_hit_chunks(session, "How do I build the BESTÅ?")  # pyright: ignore[reportArgumentType]
+    assert [rc.document.path for rc in out] == [ikea.path]
+    assert out[0].name_hit is True
+    assert out[0].fts_hit is True
+    assert out[0].cosine == 0.0
+
+
+def test_name_hit_chunks_ascii_question_hits_accented_doc() -> None:
+    """The bridge, end to end (direction 2): the ASCII question token
+    ``basta`` name-hits the ACCENTED filename ``BÅsta.md`` — the folded
+    stem enters the equal set as ``basta`` (pre-phase the accented stem
+    normalized to ``bsta`` — the ``å`` simply dropped out of the alnum
+    scan — so the match was impossible)."""
+    accented = _doc("manuals/BÅsta.md", "MONTERING", source="ikea_instructions")
+    miss = _doc("manuals/hesta.md", "MONTERING", source="ikea_instructions")
+    session = _FakeSession(
+        [_name_row(accented), _name_row(miss)],
+        [_name_hit_lateral_row(accented)],
+    )
+    out = _name_hit_chunks(session, "How do I assemble the basta shelf?")  # pyright: ignore[reportArgumentType]
+    assert [rc.document.path for rc in out] == [accented.path]
+    assert out[0].name_hit is True
+
+
+def test_name_hit_chunks_fold_preserves_digitless_precision() -> None:
+    """The fold does NOT relax the A2 match classes: the accented token
+    ``déploy`` folds to ``deploy`` and STILL does not hit the
+    ``Deployments/`` document (the part normalizes to ``deployments`` ≠
+    ``deploy`` — titles are never matched, digits never enter); only the
+    exact ``gitea`` component hits."""
+    precision = _doc("Deployments/reeseapps/README.md", "body", title="Deployments")
+    gitea = _doc("notes/gitea.md", "body")
+    session = _FakeSession(
+        [_name_row(precision), _name_row(gitea)],
+        [_name_hit_lateral_row(gitea)],
+    )
+    out = _name_hit_chunks(session, "how do I déploy gitea")  # pyright: ignore[reportArgumentType]
+    assert [rc.document.path for rc in out] == [gitea.path]