phase: 162_search_recall_small_sources

**Phase 162 — final verification pass (all tasks already complete; verified + docs pass)**

- Verified: `_fold` (NFKD+casefold, stdlib) inside `_normalize_name` — `BESTÅ`→`besta` question side, `BÅsta.md`→`basta` path side; A2 match classes/ranking/caps/bonus untouched; FTS/cosine/A8 untouched.
- Verified: `grep_document` folds both sides via the imported `_fold` (no second copy); regex-teaching lines byte-identical; 3 copy sites updated.
- Verified: `regenerate_overview` appends the deterministic `Sources: <name> (<k> doc|docs)` tail (happy path only; zero-doc + `LLMError` rows byte-identical; `build_overview_prompt`/`load_kb_overview`/`build_kb_section` untouched).
- Verified: `TOOLS_SECTION` grep sentence updated + D4 search-discipline sentence present **verbatim**, immediately after the grep sentence; `PERSONA`/LOW pins byte-identical (`test_prompt_lock.py`).
- Measured evidence (in `.agents/reports/162_…/task04_prefix_evidence_and_battery.md` + re-measured every run by `test_under_rank_is_measured_not_assumed`): pre-fix tokens `['build','best']` → IKEA doc rank 6 (outside top-5); post-fix `['build','besta']` → suggested #1, cosine unchanged 0.382.
- `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated); neighbours `test_kb_overview{,_freshness} test_name_signal_read_chips test_grep_regex_teaching test_harness_aligned_tools test_agent_document_tools test_tool_call_compact` → **24 passed**.
- `uv run pytest --cov=app --cov-report=term-missing` → **3756 passed**, app coverage **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**.
- Completion criteria: fold both ways ✅ · `Sources:` tail ✅ · TOOLS_SECTION + D4 verbatim + re-cut pins ✅ · E2E green/battery recorded/full suite/coverage/lint ✅ · docs pass ✅ · commit + dir move = harness.
- Battery: 5 phase-119 questions byte-identical to the phase-119 record (GATE PASS); the 2 Besta questions **skipped loudly** — verified the 2026-09-05 replica dump has zero `ikea` matches (`grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0); rank gate stays with the owner's live KB.
- Docs fixed: README — overview-tail doc + `BOR_KB_OVERVIEW_MAX_CHARS` tail-cut constraint, the phase-162 search-discipline paragraph, per-phase (not per-story) E2E wording, new "Retrieval regression battery" section; AGENTS.md — rule 4 per-phase wording + battery command in Quick reference. (The overview's expected "AGENTS.md §6 tool table" does not exist in this repo — the grep contract line lives in README's tool table, already updated.)
- Deviations (recorded, not assumed): `_fold` adds `casefold` (required for the pinned `ß → ss`); `①` NFKD-decomposes to `1` (the overview's parenthetical was wrong) and is excluded by `NAME_TOKEN_MIN_LEN`; pre-fix evidence lives in the reports dir because task files are read-only for this pass.
- Owner follow-up A6 (runtime, not code): add the Tuning-tab steering note "IKEA's BESTÅ assembly PDF is a live source — search it as `BESTÅ` or `AA-1272068`".
- Next pending phase: none — 162 is the only entry in `todo/` (harness moves it to `complete/`).

app/rag/agent.py

@@ -131,8 +131,10 @@ task 04):
    the only capped read (the phase-95 cap unchanged). And ``grep``
    greps the
    indexed documents (or the one document a combined ``source/path``
-   names) for a case-insensitive fixed substring and returns up to 20
-   ``source/path:line: text`` match lines (owner-locked A5, phase 68),
+   names) for a case- and accent-insensitive fixed substring (accents
+   fold — ``besta`` matches ``BESTÅ``; phase 162, LOCKED A3) and
+   returns up to 20 ``source/path:line: text`` match lines
+   (owner-locked A5, phase 68),
    each line truncated to 200 chars. A grep is a **locator**, not a
    context-adder: it never appends to the answer context (only
    ``read`` does — ``holder.read_docs`` is untouched by a grep).
@@ -312,7 +314,7 @@ from app.rag.llm import (
     ToolResultPiece,
     chat_stream_retried,
 )
-from app.rag.retriever import TRUNCATION_MARKER
+from app.rag.retriever import TRUNCATION_MARKER, _fold
 from app.rag.scaffolding import ScaffoldingFilter
 from app.rag.source_removal import resolve_source_name
 
@@ -437,7 +439,8 @@ AGENT_TOOLS: list[dict[str, Any]] = [
             "name": "grep",
             "description": (
                 "Search the indexed documents for an exact string "
-                "(case-insensitive) and return up to 20 matching lines "
+                "(case- and accent-insensitive — accents fold: `besta` "
+                "matches `BESTÅ`) and return up to 20 matching lines "
                 "as `source/path:line: text` — a locator, not a "
                 "context-adder: read the winner with `read` — for a "
                 "large document, use the reported line number as "
@@ -458,7 +461,9 @@ AGENT_TOOLS: list[dict[str, Any]] = [
                         "type": "string",
                         "description": (
                             "The exact text to search for (a plain "
-                            "substring, not a regex — no '.*', no "
+                            "substring — case- and accent-insensitive, "
+                            "accents fold: `besta` matches `BESTÅ` — "
+                            "not a regex — no '.*', no "
                             "'\\.', no character classes)"
                         ),
                     },
@@ -1560,16 +1565,21 @@ def format_covered_spans(spans: Sequence[tuple[int, int]]) -> str:
 def grep_document(content: str, pattern: str) -> list[tuple[int, str]]:
     """Every line of *content* that contains *pattern*, in file order.
 
-    Case-insensitive **fixed substring** (owner-locked A5: no regex — no
-    ReDoS surface, a simple contract for the model). Returns
-    ``(1-based line number, line.rstrip())`` pairs; an empty *content*
-    never matches a non-empty pattern.
+    Case- **and accent-insensitive fixed substring** (owner-locked A5:
+    no regex — no ReDoS surface, a simple contract for the model). Both
+    sides fold through the shared :func:`app.rag.retriever._fold` plus
+    lowercase (phase 162, LOCKED A3 — the fold changes the alphabet it
+    ignores, not the matching semantics: ``grep "besta"`` matches a
+    content line ``The BESTÅ frame``, and a pattern carrying accents
+    folds too), so grep and the name-hit path can never drift apart.
+    Returns ``(1-based line number, line.rstrip())`` pairs; an empty
+    *content* never matches a non-empty pattern.
     """
-    needle = pattern.lower()
+    needle = _fold(pattern.lower())
     return [
         (number, line.rstrip())
         for number, line in enumerate(content.split("\n"), start=1)
-        if needle in line.lower()
+        if needle in _fold(line.lower())
     ]
 
 

app/rag/overview.py

@@ -28,6 +28,21 @@ an import added/updated at least one document (or no row exists yet),
 and a ``lite`` failure logs and leaves the previous outline intact — an
 old outline is better than none (phase locked decisions).
 
+After the model's outline, :func:`regenerate_overview` appends ONE
+deterministic, non-LLM line — ``Sources: <name> (<k> doc|docs), …``
+(phase 162, LOCKED A2). The 2026-10-07 live finding: the ``lite``
+outline is lossy — it listed 3 groups of a 14-source index, so the
+1-doc sources (``ikea_instructions``, ``players_handbook``) were
+invisible to ``<knowledge_base>``. Small sources must be visible
+without a model call, so the line is computed mechanically from the
+catalogue rows (:func:`_source_inventory`) — the mock/real model
+outputs never affect it, and ``build_overview_prompt`` never sees the
+tail (the model input stays byte-identical). The longest-prefix cap in
+``build_kb_section`` applies to the WHOLE stored text: at the default
+``BOR_KB_OVERVIEW_MAX_CHARS`` (4000) the ~1500-char outline plus the
+one-line tail (~15 chars/source) always fit; a smaller owner-set cap
+cuts the tail first — accepted, the cap is owner-tunable.
+
 The ``KB_OVERVIEW_MODE`` marker follows the ``SUMMARY_MODE`` /
 ``DEFLECT_MODE`` convention: the deterministic E2E mock LLM keys on it
 in the system prompt (wired in phase 31, task 05).
@@ -101,6 +116,35 @@ def _first_summary_line(summary: str | None) -> str:
     return ""
 
 
+def _source_inventory(rows: Sequence[tuple[str, str, str, str | None]]) -> str:
+    """The phase-162 deterministic source inventory (LOCKED A2).
+
+    Mechanical, non-LLM: one entry per **distinct** source in catalog
+    order (``source, path`` — first occurrence in the ``source,
+    path``-ordered rows), each ``<name> (<k> doc|docs)`` — singular
+    ``doc`` at k=1, ``docs`` otherwise — joined with ``, `` behind the
+    ``Sources: `` prefix. The ``lite`` outline stays the model's job;
+    this line guarantees no source — not even a 1-doc one like the
+    2026-10-07 ``ikea_instructions`` case — can be invisible to
+    ``<knowledge_base>``, whatever the model outputs. The mock/real
+    model outputs never affect it.
+
+    *rows* is the existing ``(source, path, title, summary)``
+    projection :func:`regenerate_overview` already reads — no new
+    query. At the default ``BOR_KB_OVERVIEW_MAX_CHARS`` (4000) the
+    outline plus this one line always fit; a smaller owner-set cap
+    cuts the tail first (accepted — the cap is owner-tunable).
+    """
+    counts: dict[str, int] = {}
+    for source, _path, _title, _summary in rows:
+        counts[source] = counts.get(source, 0) + 1
+    entries = ", ".join(
+        f"{name} ({count} {'doc' if count == 1 else 'docs'})"
+        for name, count in sorted(counts.items())
+    )
+    return f"Sources: {entries}"
+
+
 def build_overview_prompt(
     rows: Sequence[tuple[str, str, str, str | None]], max_chars: int | None = None
 ) -> tuple[str, str]:
@@ -166,10 +210,16 @@ async def regenerate_overview(
 
     * Zero documents → the existing row (if any) is left untouched and
       ``False`` is returned — no KB, no outline, no wasted model call.
-    * Happy path → the model's text is upserted into the single row
-      (``id = 1``, fresh UTC ``updated_at``), committed, and
-      ``overview: regenerated docs=… chars=…`` is logged; returns
-      ``True``. Idempotent by construction: one row, always ``id = 1``.
+    * Happy path → the model's text (trailing newlines rstripped) plus
+      the deterministic ``Sources: …`` inventory line (phase 162,
+      LOCKED A2 — :func:`_source_inventory`) is upserted into the
+      single row (``id = 1``, fresh UTC ``updated_at``), committed, and
+      ``overview: regenerated docs=… chars=…`` is logged (chars = the
+      FULL stored content, tail included); returns ``True``. Idempotent
+      by construction: one row, always ``id = 1``. The model input
+      never sees the tail; the zero-doc and ``LLMError`` paths below
+      leave a previous row byte-identical — an existing tail is part of
+      the stored content and is preserved on failure.
     * :class:`LLMError` → ``overview: regeneration failed — …`` is
       logged and ``False`` is returned with the previous row intact —
       an old outline is better than none.
@@ -209,15 +259,21 @@ async def regenerate_overview(
             logger.error("overview: regeneration failed — %s", e)
             return False
 
+        # Phase 162 (LOCKED A2): the deterministic source inventory is
+        # appended to the STORED content after the model's outline —
+        # mechanical, non-LLM, so no source can be invisible to
+        # `<knowledge_base>`. The model input above never sees the tail.
+        content = text.rstrip("\n") + "\n" + _source_inventory(rows)
+
         now = datetime.now(UTC)
         row = session.get(KbOverview, 1)
         if row is None:
-            session.add(KbOverview(id=1, content=text, updated_at=now))
+            session.add(KbOverview(id=1, content=content, updated_at=now))
         else:
-            row.content = text
+            row.content = content
             row.updated_at = now
         session.commit()
-        logger.info("overview: regenerated docs=%d chars=%d", len(rows), len(text))
+        logger.info("overview: regenerated docs=%d chars=%d", len(rows), len(content))
         return True
     finally:
         if owns_session:

app/rag/prompts.py

@@ -265,6 +265,31 @@ ORIGIN_RULE: str = (
     "again for the source."
 )
 
+#: The search-discipline rule (phase 162, task 03, LOCKED A4) — appended
+#: to the ``TOOLS_SECTION`` tools paragraph immediately after the grep
+#: sentence (i.e. after the ``…retry with the plain text you expect to
+#: see.`` clause, the placement from the 2026-10-07 owner conversation
+#: where the agent answered from the lossy ``<knowledge_base>`` outline
+#: instead of running ``ls``, and where the exact-substring grep silently
+#: missed the accented ``BESTÅ`` in the IKEA PDF's text layer). Adapted
+#: verbatim from the conversation's proposed system-prompt patch for the
+#: folded grep (LOCKED A3). The three lessons: (1) an outline/summary is
+#: a hint, never an inventory — ``ls`` (no ``path``) before claiming the
+#: KB lacks something; (2) grep folds accents but one pattern is one
+#: string — split filename tokens and the part number need separate
+#: greps; (3) a gitignored note describes repo status, not index status.
+#: The mock E2E is unaffected (it keys on markers, not this sentence).
+SEARCH_DISCIPLINE_RULE: str = (
+    "A summary or outline is a hint, never an inventory: before claiming "
+    "the knowledge base lacks something, run `ls` with no `path` to list "
+    "every source, and `grep` the artifact's part number or a filename "
+    "token. `grep` folds accents — `besta` matches `BESTÅ` — but one "
+    "pattern is one string: a name's split tokens (`frame`, `white`) and "
+    "its part number (`AA-1272068`) need separate greps. A note saying a "
+    "file is gitignored describes its repo status, not whether it is "
+    "indexed as a source."
+)
+
 TOOLS_SECTION: str = (
     "<tools>\n"
     "You may extend your context with three tools. `ls` lists the "
@@ -305,12 +330,15 @@ TOOLS_SECTION: str = (
     "path as plain text otherwise. The user should be able to open the "
     "complete original from your answer in one step, not ask again for "
     "the source. `grep` locates an exact string "
-    "(case-insensitive) in the indexed documents and returns up to 20 "
+    "(case- and accent-insensitive — accents fold: `besta` matches "
+    "`BESTÅ`) in the indexed documents and returns up to 20 "
     "matching `source/path:line: text` lines — a locator, not a "
     "context-adder: read the winner with `read`. A grep pattern is a "
     "plain substring, NEVER a regex — '.*' and '\\.' are literal text "
     "there; if such a pattern returns no matches, retry with the plain "
-    "text you expect to see. For a normal search pass only `pattern` — "
+    "text you expect to see. "
+    + SEARCH_DISCIPLINE_RULE
+    + " For a normal search pass only `pattern` — "
     "its optional `path` argument "
     "limits the search to one document you already know, by the same "
     "combined `source/path` string; never a source name — a bare "

app/rag/retriever.py

@@ -28,9 +28,21 @@
   order ``(source, path)`` — the old total-matched-length tie-break
   is retired, it outranked 5-char product names by 6-char common
   tokens; capped at :data:`NAME_HIT_LIMIT`),
-  the FTS rows follow. This is what finds name-your-tool questions
-  ("gitlab") that vector similarity buries — and, the 2026-09-05
-  incident, the versioned-name case the default parser lexes
+  the FTS rows follow. NAME NORMALIZATION IS ACCENT-FOLDED (phase 162,
+  LOCKED A1): :func:`_normalize_name` runs :func:`_fold` (NFKD
+  decomposition, combining marks dropped — stdlib only) before the
+  alnum scan, and it is the SINGLE normalization both sides of name
+  matching pass through (question tokens and path parts alike) — the
+  2026-10-07 live gap this closes: the IKEA filename is ASCII
+  (``besta-frame-white__AA-1272068-11-100``) while its content layer
+  spells the name only as ``BESTÅ`` (U+00C5), and the old lower-only
+  normalization turned the accented question token into ``best`` —
+  never a match. With the fold, ``BESTÅ`` → ``besta`` (was ``best``)
+  and an accented filename (``BÅsta.md`` → ``basta``) answers to the
+  ASCII question — one function, both directions. This is what finds
+  name-your-tool questions ("gitlab") that vector similarity buries —
+  and, the 2026-09-05 incident, the versioned-name case the default
+  parser lexes
   incompatibly (question "Qwen 3.8" → ``qwen``/``3``/``8`` can never
   match a document's ``qwen3``/``8``/``27b`` tokens, while every
   unrelated llama.cpp quadlet out-ranks the target on the shared
@@ -81,6 +93,7 @@ from __future__ import annotations
 
 import math
 import re
+import unicodedata
 import uuid
 from collections.abc import Sequence
 from dataclasses import dataclass, replace
@@ -130,9 +143,42 @@ _ALNUM_RE = re.compile(r"[a-z0-9]+")
 _STEM_SPLIT_RE = re.compile(r"[^a-z0-9]+")
 
 
+def _fold(s: str) -> str:
+    """The shared accent fold (phase 162, LOCKED A1): NFKD decomposition,
+    combining marks dropped (stdlib ``unicodedata`` + ``str.casefold``
+    only — no new dependency).
+
+    ``Å/å → a``, ``Ä/Ö/Ü → a/o/u``, ``ß → ss``, ``fi → fi``, ``ç → c``
+    (``casefold`` is the Unicode fold — the ``ß → ss`` case rides it:
+    NFKD alone leaves ``ß`` intact — and ``str.lower`` leaves the ``fi``
+    ligature). Characters with no decomposition (CJK) survive the fold
+    and still drop out under the caller's alnum scan (behavior
+    unchanged for them); compatibility-decomposable characters (``①`` →
+    ``1``) fold to their ASCII base, which :data:`NAME_TOKEN_MIN_LEN`
+    already excludes as a name token (a digit base can still feed the
+    versioned join as the numeric NEXT token — the only route left is a
+    prefix-class candidate no real path matches). Case is folded too
+    (the caller lowercases anyway). Imported by ``app.rag.agent`` for
+    the folded ``grep`` (phase 162, LOCKED A3) — one fold, never
+    drifting apart.
+    """
+    decomposed = unicodedata.normalize("NFKD", s.casefold())
+    return "".join(c for c in decomposed if not unicodedata.combining(c))
+
+
 def _normalize_name(s: str) -> str:
-    """Lowercased, alnum-only form of *s* (``Qwen 3.8`` → ``qwen38``)."""
-    return "".join(_ALNUM_RE.findall(s.lower()))
+    """Lowercased, ACCENT-FOLDED, alnum-only form of *s* (``Qwen 3.8`` →
+    ``qwen38`` unchanged; ``BESTÅ`` → ``besta`` — the phase-162 fold).
+
+    This is the SINGLE normalization both sides of name matching pass
+    through — the :func:`name_hit_tokens` question tokens and the
+    :func:`_name_parts` path parts / file stem / stem sub-components —
+    so the one change bridges both directions (the ASCII question
+    ``besta`` keeps matching the ASCII filename, the accented question
+    ``BESTÅ`` now matches it too, and an accented filename ``BÅsta.md``
+    answers to the ASCII question ``basta``).
+    """
+    return "".join(_ALNUM_RE.findall(_fold(s.lower())))
 
 
 def name_hit_tokens(question: str) -> list[str]:
@@ -166,7 +212,10 @@ def name_hit_tokens(question: str) -> list[str]:
     Order of first appearance, de-duplicated. Matching applies the
     A2 rule: digit-bearing candidates prefix a normalized path part
     or file stem; digitless candidates equal a part, stem, or stem
-    sub-component — titles are never matched.
+    sub-component — titles are never matched. Candidates are
+    ACCENT-FOLDED through :func:`_normalize_name` (phase 162) — the
+    token ``besta`` drawn from ``BESTÅ`` name-matches ``BESTÅ``-class
+    names both ways.
     """
     tokens = question.split()
     out: list[str] = []
@@ -456,6 +505,10 @@ def _name_parts(path: str) -> tuple[set[str], set[str]]:
       (the stem lowercased, split on non-alphanumeric runs, each piece
       normalized, empties dropped: ``kubernetes_gitea`` →
       ``kubernetes`` / ``gitea``): DIGITLESS tokens exact-match these.
+
+    The parts are ACCENT-FOLDED through :func:`_normalize_name` (phase
+    162) — ``besta`` name-matches ``BESTÅ``-class names both ways (the
+    accented stem ``BÅsta`` enters the equal set as ``basta``).
     """
     p = Path(path)
     prefix = {_normalize_name(part) for part in p.parts}
@@ -479,6 +532,10 @@ def _name_token_matches(token: str, prefix: set[str], equal: set[str]) -> bool:
     * the token HAS NO DIGIT → it EQUALS a normalized path part, file
       stem, or stem sub-component (``gitea`` → the ``gitea/`` folder,
       ``gitea.md``, ``kubernetes_gitea``, ``gitea-values``).
+
+    Both the token and the path sets arrive ACCENT-FOLDED from
+    :func:`_normalize_name` (phase 162) — ``besta`` matches
+    ``BESTÅ``-class names both ways.
     """
     if any(ch.isdigit() for ch in token):
         return any(part.startswith(token) for part in prefix)