diff --git a/.agents/phases/complete/162_search_recall_small_sources/00_phase.md b/.agents/phases/complete/162_search_recall_small_sources/00_phase.md new file mode 100644 index 0000000..4fd0682 --- /dev/null +++ b/.agents/phases/complete/162_search_recall_small_sources/00_phase.md @@ -0,0 +1,70 @@ +# Phase 162 — Search recall: accent-folded name hits & grep, KB-overview source inventory, search-discipline teaching + +**Source:** owner live report, 2026-10-07 — brain.reeseapps.com, shared conversation "How do I build a Besta?" (share token `58525626-4462-4baf-bcf2-3abc81f18284`). The owner asked "Why did you miss it? Let's try to improve your search techniques and system prompt." The agent's post-mortem identified **four stacked failures**, three of which are code-fixable: + +1. **The `` outline is lossy and is the only map the model gets.** The outline listed 3 groups (Gufo, Brain-of-Reese, Deployments); the live index had 14 sources. `ikea_instructions` (1 doc) and `players_handbook` (1 doc) were **invisible** — a whole class of source-level questions could not be surfaced by the outline at all. The model answered from the outline instead of running `ls`. +2. **Notes poisoned the model with a true-but-different fact** (the phase-140/148 note "the IKEA tar is gitignored, on disk only" was over-generalized to "the brain has no copy of this PDF"). Model behavior — mitigated here by prompt teaching (D4), plus an owner runtime follow-up (A6, recorded, not code). +3. **The Å/ASCII accent-folding gap.** `grep "besta"` returned only phase docs and never the PDF, because the only product name in the PDF's text layer is `BESTÅ` (U+00C5): the filename is ASCII (`besta-frame-white__…`) but the content is not, and the exact-substring grep cannot bridge that. On the name-hit side the gap is symmetric: the question token `BESTÅ` normalizes (lower + alnum-only) to `best` — not `besta` — so an accented question never name-matches the ASCII filename's stem sub-components. +4. **Retrieval under-ranked the PDF (the phase-119 failure class).** The doc's title is the generic "Assembly Instructions" (title ranking gives nothing); by phase-119 LOCKED A2 the digitless token `besta` (≥4 chars) equals a stem sub-component of `besta-frame-white__AA-1272068-11-100` and should have name-matched, but the name-hit row lost the fused walk to the phase docs whose bodies literally contain "besta". Flagged by the post-mortem as a hypothesis needing a test — **this phase pins it with the battery (D5)** rather than assuming a finding. + +**Story:** n/a (owner-directed RAG behavior change; extends the phase-119 name-hit mechanism, the phase-31 KB overview, the phase-68/70 grep, and the phase-117 prompt pins). + +## Objective +Accented names (`BESTÅ` ↔ `besta`) are found **both ways** — by the name-hit path (question side) and by `grep` (content side); **no source can be invisible** to `` (a deterministic, non-LLM source-inventory line is appended to the stored overview); the model is **taught the search discipline** the post-mortem adopted (outline ≠ inventory; `ls` first; part-number/filename-token greps; gitignored ≠ unindexed); and the battery question "How do I build a Besta?" **pins** the IKEA doc into the suggested top-5 — turning the anecdote into a regression test. + +## Dependencies +- `119_name_signal_read_chips` (complete) — the phase-119 name-hit rule (LOCKED A2), the `name_hit` flag, the bonus (LOCKED A3), and the `suggested:` eval-battery line this phase extends. +- `31`-line KB overview (complete; `app/rag/overview.py`) — the `regenerate_overview` best-effort/change-gated generation this phase appends the inventory to. +- `68_search_tool` / `70_harness_aligned_tools` (complete) — the `grep` contract (case-insensitive fixed substring, locator, teaching refusals) this phase extends. +- `117_tool_call_compact` (complete) — the `TOOLS_SECTION` byte-pins this phase re-cuts. + +## Design (shared by all tasks — the executor reads this, not the chat) + +**D1 — accent-folded name normalization (task 01, LOCKED A1).** One new helper in `app/rag/retriever.py`: `_fold(s)` = `unicodedata` NFKD normalize + drop every char with a non-zero combining class (stdlib only; no new dependency). `_normalize_name` becomes the existing `[a-z0-9]+` scan of `_fold(s.lower())`. Because `_normalize_name` is the **single** normalization both sides of name matching pass through — the question tokens in `name_hit_tokens` AND the path parts / file stem / stem sub-components in `_name_parts` — the one change bridges both directions: the ASCII question `besta` keeps matching the ASCII filename's sub-components (unchanged behavior), the accented question `BESTÅ` now normalizes to `besta` (was `best` — the 2026-10-07 gap), and an accented **filename** (`BÅsta.md` → `basta`) matches an ASCII question. Folding cases the tests pin: `Å/å → a`, `Ä/ä/Ö/ö/Ü/ü → a/o/u` (the letter, decomposition preserved), `ß → ss`, the `fi` ligature → `fi`, `ç → c`; characters with no NFKD decomposition (CJK, `①`) still drop out under the alnum scan, exactly as today. **Everything else in the A2 mechanism is untouched**: the digit-bearing-prefix / digitless-equality match classes, titles-never-matched, the (matched-token count, catalog order) ranking, `NAME_TOKEN_MIN_LEN`, the versioned-name join, `NAME_HIT_LIMIT`, the `name_hit` flag, the phase-119 bonus (LOCKED A3), and the FTS / cosine / A8 gate (LOCKED A7/A8 — the `tsvector` 'english' pipeline is NOT modified; same-accents FTS already bridges, accent-vs-ASCII FTS bridging is out of scope by the lock). + +**D2 — the KB-overview source inventory (task 02, LOCKED A2).** The overview is `lite`-generated and lossy (D1-failure #1); the fix is **deterministic, non-LLM**: `regenerate_overview` (happy path only) appends one line to the stored content, after the model's outline: `Sources: ( doc|docs), …` — one entry per DISTINCT source, catalog order (`source, path`), exact per-source document counts. Computed from the same catalogue rows the prompt builder already reads (no new query). `build_overview_prompt` / `OVERVIEW_INSTRUCTION` are **byte-identical** (the model never sees the tail — it is mechanical, not a model burden); `load_kb_overview` / `build_kb_section` are untouched (the existing longest-prefix cap applies to the whole stored text). Budget: the model caps the outline at ~1500 chars per the instruction; one inventory line is ~15 chars/source — at the default `BOR_KB_OVERVIEW_MAX_CHARS` (4000) the tail always fits; a smaller owner-set cap cuts the tail first (accepted; the cap is owner-tunable, the record here is the constraint). The zero-doc early return and the `LLMError` path are **byte-identical** (the previous row — tail included, it is part of the stored content — stays intact; an old outline is better than none). + +**D3 — accent-folding grep (task 03, LOCKED A3).** `grep_document` in `app/rag/agent.py` folds **both sides** — the pattern and each content line — through the SAME `_fold` as D1 (imported from `app.rag.retriever`, which `agent.py` already imports from: grep and the name-hit path can never drift apart) plus lowercase (replacing the current lower-only comparison): `grep "besta"` now matches a content line `The BESTÅ frame`. The contract remains a **fixed substring** (the phase-68 owner lock) — the fold changes the alphabet it ignores, not the matching semantics; still a locator, still ≤20 lines, still line-truncated at 200 chars. The regex teaching (`NO_MATCHES_REGEX` / `_SCOPED`, `looks_like_regex`, the plain-form retry hint) is **byte-identical** — the plain-substring claim it teaches is still true. Copy updates (all in this task): the `agent.py` module docstring grep paragraph, the `AGENT_TOOLS` grep function description + the `pattern` parameter description, and the `TOOLS_SECTION` grep sentence in `app/rag/prompts.py` ("case-insensitive" → "case- and accent-insensitive"), plus D4's new sentence. The phase-117 `test_prompt_lock.py` / `test_prompts.py` / `test_agent.py` pins re-cut in the same task (the phase-124 pattern: copy change + re-cut pins in one task). + +**D4 — the search-discipline sentence (task 03, LOCKED A4).** Appended to the `TOOLS_SECTION` tools paragraph **immediately after the grep sentence** (the conversation's placement), LOCKED verbatim: + +> `A summary or outline is a hint, never an inventory: before claiming the knowledge base lacks something, run `ls` with no `path` to list every source, and `grep` the artifact's part number or a filename token. `grep` folds accents — `besta` matches `BESTÅ` — but one pattern is one string: a name's split tokens (`frame`, `white`) and its part number (`AA-1272068`) need separate greps. A note saying a file is gitignored describes its repo status, not whether it is indexed as a source.` + +(Adapted from the conversation's proposed patch for the D3 fold — the conversation's original sentence assumed a non-folding grep; the owner asked for "improve your search techniques and system prompt", which is the permission source for this copy change.) `PERSONA`, the LOW/`DEFLECT_MODE` body, and every other `TOOLS_SECTION` sentence stay byte-identical. The mock LLM is unaffected (it keys on markers, not on the sentence). + +**D5 — verification (task 04).** A new dedicated Playwright suite `tests/e2e/test_search_recall_small_sources.py` (run in isolation; the phase-119 local-directory-source fixture pattern) with a synthetic replica of the live miss: source `ikea_instructions` (ONE doc, `besta-frame-white__AA-1272068-11-100.md`, body containing the name **only** in accented form — no ASCII "besta") + 2–3 distractor docs in other sources whose bodies contain ASCII "besta" (the phase-docs stand-in). The **pre-fix evidence is measured and recorded** in the task file: with the task-01 fold temporarily reverted (git stash), the ACCENTED question "How do I build the BESTÅ?" must lose the accented doc from the suggested top-5 (record the measured ranks); restore the fix. Gated assertions: (1) the accented question's `done`-frame `suggestions` include the `ikea_instructions` doc (the D1 question-side fold); (2) the ASCII question "How do I build a Besta?" includes it too (the phase-119 path regression); (3) the stored `kb_overview` row (read through the fixture session) ends with the `Sources: …` tail naming **every** fixture source with exact counts — the 1-doc source visible (D2); (4) the out-of-KB question's gate verdict is unchanged (D1–D3 must not move the A8 gate). `tests/fixtures/retrieval_battery.txt` (the tracked phase-119 battery) gains `How do I build a Besta?` with its gate comment: on the 1,000-doc live replica (the `restore-test-db` skill; the dump is a local gitignored artifact — if missing, STOP and ask the owner, never fabricate numbers) the battery runs when the replica's `documents` table contains an `ikea_instructions`-class source (check `SELECT DISTINCT source FROM documents`) and the suggested top-5 must include `ikea_instructions/besta-frame-white__AA-1272068-11-100.pdf`; if the replica lacks it (the snapshot may predate the import), the five existing phase-119 questions still run (their gate must NOT move — the name-hit change is behavior-preserving for them) and the Besta question is recorded as **skipped loudly (source absent in the replica snapshot)** — the phase-140 "skipped loudly" precedent. The under-rank hypothesis (D1-failure #4) is thus either pinned green or handed to the owner with measured ranks — never assumed. + +**Invariants (all tasks).** The A8 honesty gate, the FTS `tsvector` pipeline, the cosine path, every SSE frame shape, the LOW prompt, `PERSONA`, the `NO_MATCHES_REGEX`/`_SCOPED` teaching lines, the mock-LLM markers, and the `build_overview_prompt` / `OVERVIEW_INSTRUCTION` inputs are byte-identical. `PLAN.md` untouched (Protocol B — A5). + +## Tasks +1. `01_accent_fold_name_norms.md` — the shared `_fold` helper + the `_normalize_name` fold (D1). +2. `02_kb_overview_source_inventory.md` — the deterministic `Sources: …` tail (D2). +3. `03_grep_folding_search_discipline.md` — the folded `grep_document` + the three copy updates + the D4 sentence + the re-cut pins (D3+D4). +4. `04_e2e_battery_pins.md` — the dedicated E2E, the pre-fix evidence, the battery question + replica run, the final gates (D5). + +## Testing & Quality +- Unit: `tests/unit/test_retriever.py` (D1: the bridge both ways, the fold cases, the A2 non-regression set), `tests/unit/test_overview.py` (D2: the tail format/order/counts, the 1-doc survival, the zero-doc + LLM-failure paths, the byte-identical `build_overview_prompt`), `tests/unit/test_agent.py` (D3: `grep_document` fold both sides, the scoped + unscoped paths, the copy pins), `tests/unit/test_prompts.py` + `tests/unit/test_prompt_lock.py` (D3+D4: the re-cut pins; the LOW + `PERSONA` byte-identity). +- E2E: `tests/e2e/test_search_recall_small_sources.py`, **in isolation** (`uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov`); the phase-119/117/31 regression suites green (the re-cut pins included). +- Battery (task 04): per D5 — the replica run recorded in the task file; the five phase-119 questions' gate unchanged. +- Coverage: **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean. + +## Completion Criteria +- [ ] `grep "besta"` (unit + the E2E fixture's content layer) matches an accented `BESTÅ` content line, and the accented question "How do I build the BESTÅ?" name-matches the ASCII filename's stem sub-components — the fold is one function, both sides, measured (the pre-fix ranks recorded in `04_…md`). +- [ ] The stored `kb_overview` row ends with the deterministic `Sources: …` line naming every source with exact counts; a 1-doc source is visible; the zero-doc and LLM-failure paths are byte-identical to the pre-phase behavior. +- [ ] The `TOOLS_SECTION` carries the folded grep contract + the LOCKED D4 sentence (all other pins byte-identical; the re-cut pins green). +- [ ] `tests/e2e/test_search_recall_small_sources.py` green in isolation; the battery question is recorded per D5; `uv run pytest` green; coverage >90%; `uv run ruff check . && uv run pyright` clean. +- [ ] `README.md` + `AGENTS.md` reviewed in full in the final pass (the grep contract line in `AGENTS.md`'s §6 quick reference and `PLAN.md` §6's tool table mention "case-insensitive fixed substring" — the final pass updates `AGENTS.md`'s tool description to "case- and accent-insensitive"; `PLAN.md` stays untouched per A5, folded at the next redo). +- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate. + +## Locked decisions +- **A1 — the fold**: NFKD normalize + combining-class strip (stdlib `unicodedata`) applied inside `_normalize_name` — the single normalization both sides of name matching use; the A2 match classes, ranking, caps, flags, the phase-119 bonus, and the FTS/cosine/A8 pipeline are untouched (owner-confirmed 2026-10-07, from the conversation's post-mortem). +- **A2 — the overview tail**: `Sources: ( doc|docs), …` appended server-side to the stored outline at generation time (happy path only); deterministic, non-LLM, catalog order; `build_overview_prompt` / `OVERVIEW_INSTRUCTION` / `load_kb_overview` / `build_kb_section` byte-identical; the zero-doc and `LLMError` paths untouched (owner-confirmed 2026-10-07). +- **A3 — the grep fold**: `grep_document` compares the `_fold`+lowercase of the pattern against the `_fold`+lowercase of each line — the SAME `_fold` as A1 (imported); still a fixed substring (phase-68 lock), still a locator, regex teaching byte-identical (owner-confirmed 2026-10-07 — the conversation's "optional grep-side accent folding", accepted). +- **A4 — the D4 sentence** is appended to the `TOOLS_SECTION` tools paragraph immediately after the grep sentence, verbatim as written in D4; `PERSONA` + LOW body + the other `TOOLS_SECTION` sentences byte-identical (owner-confirmed 2026-10-07 — the owner asked for the system-prompt improvement in the conversation). +- **A5 — PLAN.md untouched (Protocol B)**: the grep-contract extension (A3) and the prompt sentence (A4) are recorded here (owner-confirmed) and folded into the anchor table at the next redo (the phase-119 A6 precedent). +- **A6 — the steering-note follow-up is an owner runtime action, not code**: the conversation's proposed tuning rule ("IKEA's BESTÅ assembly PDF is a live source — search it as BESTÅ or AA-1272068, not besta") is a `steering_notes` row the owner adds via the Tuning tab after this phase ships; the phase records it here and does not touch `steering_notes` data. + +## Commit +```bash +git add app/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(rag): accent-fold name hits and grep (BESTÅ <-> besta), a deterministic source inventory in the KB overview, and search-discipline prompt teaching" +``` diff --git a/.agents/phases/complete/162_search_recall_small_sources/01_accent_fold_name_norms.md b/.agents/phases/complete/162_search_recall_small_sources/01_accent_fold_name_norms.md new file mode 100644 index 0000000..285366d --- /dev/null +++ b/.agents/phases/complete/162_search_recall_small_sources/01_accent_fold_name_norms.md @@ -0,0 +1,28 @@ +# Task 01 — The accent-folded name normalization (phase-162 D1, LOCKED A1) + +**Phase:** `162_search_recall_small_sources` · **Source:** phase overview D1 — the 2026-10-07 live finding: `grep "besta"` never hit the IKEA PDF (its text layer carries only `BESTÅ`, U+00C5), and the symmetric name-hit gap: the question token `BESTÅ` normalizes (lower + alnum-only) to `best`, never `besta`, so an accented question cannot name-match the ASCII filename `besta-frame-white__AA-1272068-11-100` (whose stem sub-components include `besta` under phase-119 LOCKED A2). +**Story:** n/a + +## Objective +`_normalize_name` in `app/rag/retriever.py` folds accents (NFKD + combining-class strip, stdlib `unicodedata`) before the existing alnum scan, so both sides of name matching bridge `BESTÅ` ↔ `besta` — with the A2 mechanism (match classes, ranking, caps, flags, bonus) and the FTS/cosine/A8 pipeline byte-identical. + +## Work +1. `app/rag/retriever.py` — new module-level `_fold(s: str) -> str`: `unicodedata.normalize("NFKD", s)` with every char whose `unicodedata.combining(c)` is non-zero dropped (stdlib only — no new dependency). Docstring: the shared accent fold (phase 162) — NFKD decomposition, combining marks dropped; `Å/å → a`, `ß → ss`, `fi → fi`, `ç → c`; characters with no decomposition (CJK, `①`) survive the fold and still drop out under the caller's alnum scan (behavior unchanged for them). +2. `app/rag/retriever.py` — `_normalize_name`: `"".join(_ALNUM_RE.findall(_fold(s.lower())))` (was: `…findall(s.lower())`). Docstring update: the folded form (`Qwen 3.8` → `qwen38` unchanged; `BESTÅ` → `besta` new) — this is the **single** normalization both sides pass through (the `name_hit_tokens` question tokens and the `_name_parts` path parts / stem / stem sub-components), so the one change bridges both directions. +3. `app/rag/retriever.py` — `name_hit_tokens` / `_name_parts` / `_name_token_matches`: **no code change** (they consume `_normalize_name`); their docstrings gain one clause each: candidates/parts are accent-folded (phase 162) — `besta` matches `BESTÅ`-class names both ways. +4. `app/rag/retriever.py` — module docstring: the normalization paragraph now states the fold (the 2026-10-07 gap: the ASCII filename vs the accented content layer). +5. `tests/unit/test_retriever.py` — the fold, on the file's existing fixture pattern: + - **question side**: the accented question "How do I build the BESTÅ?" produces a `besta` token (was `best` pre-phase — pin the exact token list); `Ä`/`Ö`/`Ü`/`ß`/`fi`/`ç` fold cases (`straße`-class → `strasse`-class, ligature `fine` → `fine`); + - **path side**: `_name_parts` on `besta-frame-white__AA-1272068-11-100` yields `besta` in the equal set (the phase-119 case, unchanged) AND on the accented filename `BÅsta.md` yields `basta` — the ASCII question `basta` name-matches it; + - **the bridge, end to end**: `_name_hit_chunks`-class assertion — an accented-question token hits the ASCII-stem doc AND an ASCII token hits the accented-stem doc; + - **A2 non-regression (all existing behavior, re-pinned after the change)**: digitless precision (the `deploy`-titled `Deployments/` doc does NOT hit for `deploy`), digit-bearing prefix (`qwen38` → `qwen3.8-27b-…`, and the mid-word non-match), the versioned join (`Qwen 3.8` → `qwen38`), min-length on the **folded** form, ranking (matched-token count, catalog order), `NAME_HIT_LIMIT`, `name_hit=True`/`False` convention; + - **no-decomposition characters**: a CJK / `①` token still drops out (the alnum scan behavior for non-folding chars is unchanged); + - the FTS/`retrieve()`/gate rows are untouched (the existing `test_retriever.py` / `test_retriever_recency.py` suite stays green without modification). + +## Testing & Quality +- Unit: the cases above; coverage **>90%** on `app/`. + +## Completion Criteria +- [ ] `BESTÅ` (question) → `besta` token → name-hits the ASCII filename's sub-components, and `besta` (question) → name-hits the accented filename `BÅsta.md` — one function, both directions. +- [ ] Every phase-119 A2 case (precision, prefix, join, ranking, cap, flag) passes UNCHANGED after the fold; the FTS/gate suites are untouched. +- [ ] `uv run pytest tests/unit/test_retriever.py tests/unit/test_retriever_recency.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean. diff --git a/.agents/phases/complete/162_search_recall_small_sources/02_kb_overview_source_inventory.md b/.agents/phases/complete/162_search_recall_small_sources/02_kb_overview_source_inventory.md new file mode 100644 index 0000000..e491094 --- /dev/null +++ b/.agents/phases/complete/162_search_recall_small_sources/02_kb_overview_source_inventory.md @@ -0,0 +1,29 @@ +# Task 02 — The deterministic source inventory in the KB overview (phase-162 D2, LOCKED A2) + +**Phase:** `162_search_recall_small_sources` · **Source:** phase overview D2 — the 2026-10-07 live finding: the `lite`-generated `` outline listed 3 groups of a 14-source index; the 1-doc sources `ikea_instructions` and `players_handbook` were invisible, so source-level questions ("You have ikea_instructions in your RAG" — the owner's push-back) could not be surfaced by the outline at all. +**Story:** n/a + +## Objective +`regenerate_overview` in `app/rag/overview.py` appends one **deterministic, non-LLM** line to the stored outline — `Sources: ( doc|docs), …` — so no source can be invisible to ``; the model input, the load/inject path, and the zero-doc / LLM-failure paths are byte-identical. + +## Work +1. `app/rag/overview.py` — a new pure helper `_source_inventory(rows: Sequence[tuple[str, str, str, str | None]]) -> str` next to `_first_summary_line`: one entry per **distinct** source, in catalog order (first occurrence in the `source, path`-ordered rows), each ` ( doc|docs)` — singular `doc` at k=1, `docs` otherwise — joined with `, ` behind the prefix `Sources: `. (The `rows` shape is the existing `(source, path, title, summary)` projection `regenerate_overview` already reads — no new query.) Docstring: the phase-162 deterministic inventory (LOCKED A2) — mechanical, non-LLM; the `lite` outline stays the model's job. +2. `app/rag/overview.py` — `regenerate_overview` **happy path only**: after the model text is obtained and before the upsert, the stored content = the model text (trailing newlines rstripped) + `"\n"` + the inventory line. The upsert, the `overview: regenerated docs=… chars=…` log (chars = the FULL stored content, tail included), the `True` return, and the change-gating contract are otherwise unchanged. +3. `app/rag/overview.py` — the **zero-doc early return** (`False`, row untouched) and the **`LLMError` path** (logged, `False`, previous row intact) are UNCHANGED — the previous row keeps its earlier tail, it is part of the stored content; an old outline is better than none. `build_overview_prompt`, `OVERVIEW_INSTRUCTION`, `SYSTEM_PROMPT`, `load_kb_overview`, and `app/rag/prompts.py::build_kb_section` are **untouched** (the model never sees the tail; the existing longest-prefix cap applies to the whole stored text — at the default `BOR_KB_OVERVIEW_MAX_CHARS` (4000) the ~1500-char outline + the one-line tail always fit; a smaller owner-set cap cuts the tail first — accepted, recorded in the docstring). +4. `app/rag/overview.py` — module docstring: a paragraph on the inventory line (phase 162 — the 2026-10-07 finding: the `lite` outline is lossy; small sources must be visible without a model call; the line is deterministic so the mock/real model outputs never affect it). +5. `tests/unit/test_overview.py` — the helper + the paths (the file's existing duck-typed-`OverviewLLM` fake pattern): + - the format: `Sources: a (1 doc), b (2 docs), c (3 docs)` — exact, incl. singular/plural and the catalog order; + - the **1-doc survival** (the `ikea_instructions` case): a 1-doc source among many appears with `(1 doc)`; + - the happy path: the stored row = model text + newline + the tail (exact content pin on the fake LLM's fixed output); + - zero-doc: `False`, the existing row (a pre-phase row, tail-less) byte-identical; + - `LLMError`: `False`, the **previous** row byte-identical — INCLUDING a row that already carried a tail (the tail is content, preserved on failure); + - `build_overview_prompt` byte-identical to the pre-phase output for the same rows (the model input never gains the tail); + - the regenerated log line carries the full stored-content char count. + +## Testing & Quality +- Unit: the cases above; coverage **>90%** on `app/`. + +## Completion Criteria +- [ ] A regenerated `kb_overview` row ends with `Sources: …` naming every source with exact counts (a 1-doc source visible); the model input is byte-identical to the pre-phase prompt. +- [ ] The zero-doc and `LLMError` paths leave the row byte-identical (an existing tail preserved on failure); the char log is exact. +- [ ] `uv run pytest tests/unit/test_overview.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean. diff --git a/.agents/phases/complete/162_search_recall_small_sources/03_grep_folding_search_discipline.md b/.agents/phases/complete/162_search_recall_small_sources/03_grep_folding_search_discipline.md new file mode 100644 index 0000000..5816cd5 --- /dev/null +++ b/.agents/phases/complete/162_search_recall_small_sources/03_grep_folding_search_discipline.md @@ -0,0 +1,27 @@ +# Task 03 — The accent-folding grep + the search-discipline prompt teaching (phase-162 D3+D4, LOCKED A3+A4) + +**Phase:** `162_search_recall_small_sources` · **Source:** phase overview D3+D4 — the 2026-10-07 live finding: `grep "besta"` never hit the PDF whose text layer carries only `BESTÅ` ("the tool is documented as 'a plain substring' and silently missing Å is the nastiest of the four"), and the owner's ask: "Let's try to improve your search techniques and system prompt." +**Story:** n/a + +## Objective +`grep_document` folds accents on **both** pattern and content (the SAME `_fold` as task 01, imported — grep and the name-hit path can never drift); the contract copy in all three places states the folded contract; the LOCKED D4 search-discipline sentence joins the `TOOLS_SECTION` tools paragraph; the phase-117 pins re-cut in this task. + +## Work +1. `app/rag/agent.py` — `grep_document(content, pattern)`: the comparison is `_fold(pattern.lower()) in _fold(line.lower())` per (rstripped) line — importing `_fold` from `app.rag.retriever` (the module already imports `TRUNCATION_MARKER` from there). Everything else UNCHANGED: the rstrip, the `lineno` numbering, the 20-match cap, the 200-char line truncation, the scoped/unscoped call sites (`all_documents` / the one-doc path at the existing call site), the locator semantics (no context add — `holder.read_docs` untouched). Docstring update: case- AND accent-insensitive fixed substring (phase 162, LOCKED A3) — the fold changes the alphabet it ignores, not the matching semantics; still NEVER a regex. +2. `app/rag/agent.py` — the module docstring grep paragraph (the "case-insensitive fixed substring" sentence): "case- and accent-insensitive fixed substring (accents fold — `besta` matches `BESTÅ`)". +3. `app/rag/agent.py` — `AGENT_TOOLS` the grep function description: "(case-insensitive)" → "(case- and accent-insensitive — accents fold: `besta` matches `BESTÅ`)", and the `pattern` parameter description gains the same clause (the "plain substring, NEVER a regex" teaching unchanged). The `NO_MATCHES_REGEX` / `NO_MATCHES_REGEX_SCOPED` teaching lines and `looks_like_regex` / `plain_form` are **byte-identical** (the plain-substring claim they teach is still true). +4. `app/rag/prompts.py` — `TOOLS_SECTION`, the grep sentence: "locates an exact string (case-insensitive)" → "locates an exact string (case- and accent-insensitive — accents fold: `besta` matches `BESTÅ`)" (the sentence's remainder, incl. the plain-substring + retry teaching, unchanged). +5. `app/rag/prompts.py` — `TOOLS_SECTION`: the LOCKED D4 sentence (verbatim from the overview) appended to the tools paragraph **immediately after the grep sentence**, i.e. after the `…retry with the plain text you expect to see.` clause: + `A summary or outline is a hint, never an inventory: before claiming the knowledge base lacks something, run `ls` with no `path` to list every source, and `grep` the artifact's part number or a filename token. `grep` folds accents — `besta` matches `BESTÅ` — but one pattern is one string: a name's split tokens (`frame`, `white`) and its part number (`AA-1272068`) need separate greps. A note saying a file is gitignored describes its repo status, not whether it is indexed as a source.` + `PERSONA`, the LOW/`DEFLECT_MODE` body, the `` blocks, and every other `TOOLS_SECTION` sentence are byte-identical. (The E2E mock is unaffected — it keys on markers, not on the sentence.) +6. Pins re-cut, **in this task** (the phase-124 pattern): `tests/unit/test_prompt_lock.py` (the `TOOLS_SECTION` pin + its derived cross-pins), `tests/unit/test_prompts.py` (the grounded-prompt pins that embed the grep sentence), `tests/unit/test_agent.py` (the `AGENT_TOOLS` description pin, if present — grep the file for the old description text; the `NO_MATCHES_REGEX` pins stay green unmodified). +7. `tests/unit/test_agent.py` — `grep_document` fold coverage (the existing case-insensitive test's pattern): `grep_document("The BESTÅ frame (AA-1272068-11-100)", "besta")` hits; `grep_document("…BÅsta…", "basta")` hits; `grep_document("café", "cafe")` hits; the pattern's own accent folds too — `grep_document("The besta frame", "BESTÅ")` ALSO hits (the pattern `BESTÅ` folds to `besta`); a genuinely non-matching folded pattern still returns no lines (`grep_document("The besta frame", "bestb")` → `[]`); line numbers / rstrip / cap / truncation unchanged (existing tests stay green unmodified). + +## Testing & Quality +- Unit: the cases above; coverage **>90%** on `app/`. + +## Completion Criteria +- [ ] `grep_document` matches accented ↔ ASCII both ways through the imported `_fold` (no second copy of the fold); the regex teaching lines are byte-identical. +- [ ] The three copy sites (module docstring, `AGENT_TOOLS`, `TOOLS_SECTION`) state the folded contract; the D4 sentence sits immediately after the grep sentence, LOCKED verbatim. +- [ ] The re-cut `test_prompt_lock.py` / `test_prompts.py` / `test_agent.py` pins green; the LOW + `PERSONA` byte-identity pins green unmodified. +- [ ] `uv run pytest tests/unit/test_agent.py tests/unit/test_prompt_lock.py tests/unit/test_prompts.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean. diff --git a/.agents/phases/complete/162_search_recall_small_sources/04_e2e_battery_pins.md b/.agents/phases/complete/162_search_recall_small_sources/04_e2e_battery_pins.md new file mode 100644 index 0000000..6c76404 --- /dev/null +++ b/.agents/phases/complete/162_search_recall_small_sources/04_e2e_battery_pins.md @@ -0,0 +1,31 @@ +# Task 04 — The dedicated E2E, the measured pre-fix evidence, and the battery pin (phase-162 D5) + +**Phase:** `162_search_recall_small_sources` · **Source:** phase overview D5 — the 2026-10-07 finding must turn from anecdote into a regression test: the accented question "How do I build the BESTÅ?" surfaces the accented-content doc whose ASCII filename carries the name's sub-components, over ASCII "besta"-mentioning distractors; the 1-doc source is visible in the stored overview; the A8 gate and the five phase-119 battery questions do not move. +**Story:** n/a + +## Objective +`tests/e2e/test_search_recall_small_sources.py` (run in isolation) gates the fold end-to-end on a synthetic replica of the live miss; the pre-fix ranks are **measured and recorded** in this file (the under-rank hypothesis is pinned green or handed to the owner with evidence — never assumed); `tests/fixtures/retrieval_battery.txt` gains the Besta question per D5; the final gates run. + +## Work +1. `tests/e2e/test_search_recall_small_sources.py` — new dedicated suite, **run in isolation** (`uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov`), on the phase-119 local-directory-source fixture pattern (fixture module truncates/re-imports; the deterministic mock LLM by default). Fixture KB — a synthetic replica of the live miss: + - source **`ikea_instructions`** (ONE doc): `besta-frame-white__AA-1272068-11-100.md` — body carrying the product name **only in accented form** (e.g. "The BESTÅ frame (AA-1272068-11-100)… cam bolt 130618 … wall bracket 141413 … leveling foot …") — **no ASCII "besta" anywhere in the body** (the live PDF's exact situation); + - 2–3 **distractor docs** in other sources whose bodies contain ASCII "besta" (the phase-docs stand-in — e.g. `brain-of-reese/.agents/phases/complete/140_pictorial_pdf_pages/00_phase.md`-shaped text: "the besta fixture drove the pictorial phases"). Tune the distractor count/bodies so the pre-phase accented question loses the `ikea_instructions` doc from the suggested top-5 (step 2 measures this — if it does NOT lose it, add distractors until it does, and record the final fixture shape here). + Assertions (mock LLM; the `done`-frame `suggestions` list is the observable tier — the phase-119 done-frame shape): + - **(1) the accented question** `POST /api/chat` "How do I build the BESTÅ?" → the `done` frame's `suggestions` include `ikea_instructions/besta-frame-white__AA-1272068-11-100.md` (the D1 question-side fold — `BESTÅ` → `besta` → the stem sub-component name-hit); + - **(2) the ASCII regression** "How do I build a Besta?" → the same doc in `suggestions` (the phase-119 path, unchanged by the fold); + - **(3) the overview tail** — after the fixture import (the mock's `KB_OVERVIEW_MODE` regeneration ran), the stored `kb_overview` row (read through the fixture's DB session, the phase-119 pattern) ends with `Sources: …` naming **every** fixture source with exact counts — the 1-doc `ikea_instructions` source visible (the D2 gate); + - **(4) the gate sanity** — an out-of-KB question (e.g. "What is the capital of Mongolia?") returns the deflected terminal (the D1–D3 changes must not move the A8 gate verdict; the mock's cosine behavior makes this deterministic — follow the phase-119 deflection-sanity assertion). +2. **The pre-fix evidence — measured, recorded in THIS file (before the final gates):** `git stash` the task-01 `_normalize_name` change (tasks 02–03 changes may stay — they do not affect the name-hit path), run assertion (1) against the fixture, record the measured suggested top-5 ranks for the accented question (the `ikea_instructions` doc's rank or its absence), `git stash pop`. The recorded block must show the doc OUT of the suggested top-5 pre-fix (if the fixture as built does not show that, go back to step 1's tuning and re-measure — the recorded block is the evidence the fold fixes a real miss, not a no-op). +3. `tests/fixtures/retrieval_battery.txt` — append `How do I build a Besta?` with a gate comment (the file's existing comment style): on the 1,000-doc live replica (the `restore-test-db` skill — the dump is a local gitignored artifact; **if missing, STOP and ask the owner, never fabricate numbers**) the battery runs when `SELECT DISTINCT source FROM documents` shows an `ikea_instructions`-class source, and the printed `suggested:` top-5 must include `ikea_instructions/besta-frame-white__AA-1272068-11-100.pdf`; if the replica lacks that source (the snapshot may predate the import), the five EXISTING phase-119 questions still run — their recorded gate (the canonical docs' ranks, the qwen3.8 regression, the deflection sanity `HIGH, fts_hits=5`) must be **unchanged** by this phase — and the Besta question is recorded in the task file as **skipped loudly (source absent in the replica snapshot)** — the phase-140 "skipped loudly" precedent. The under-rank hypothesis (the overview's failure #4) is thus either pinned green or handed to the owner with the measured ranks. +4. Regression + final gates: the phase-119 (`test_name_signal_read_chips.py`), phase-117, and phase-31-era regression suites green; `uv run pytest` (full suite) green; `uv run pytest --cov=app --cov-report=term-missing` coverage **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean. + +## Testing & Quality +- E2E: the dedicated suite in isolation (per AGENTS.md rule 4 — one file per phase, run in isolation); unit pins from tasks 01–03 already in place. +- Battery: per step 3, recorded in this file. +- Coverage: **>90%** on `app/`; lint + types clean. + +## Completion Criteria +- [ ] The recorded pre-fix block in this file shows the accented question losing the `ikea_instructions` doc from the suggested top-5; the post-fix E2E (assertions 1–4) is green in isolation. +- [ ] The battery question is recorded per step 3 (pinned green, or skipped loudly with the five phase-119 questions' gate verified unchanged). +- [ ] `uv run pytest` green; coverage >90%; `uv run ruff check . && uv run pyright` clean. +- [ ] The owner follow-up (A6) is noted in the final pass's summary for the owner (the steering-note "rule 13" is a runtime Tuning-tab action, not code). diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__00_phase.a1.err b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__00_phase.a1.md b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__00_phase.a1.md new file mode 100644 index 0000000..b09e400 --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__00_phase.a1.md @@ -0,0 +1,16 @@ +**Phase 162 — final verification pass (all tasks already complete; verified + docs pass)** + +- Verified: `_fold` (NFKD+casefold, stdlib) inside `_normalize_name` — `BESTÅ`→`besta` question side, `BÅsta.md`→`basta` path side; A2 match classes/ranking/caps/bonus untouched; FTS/cosine/A8 untouched. +- Verified: `grep_document` folds both sides via the imported `_fold` (no second copy); regex-teaching lines byte-identical; 3 copy sites updated. +- Verified: `regenerate_overview` appends the deterministic `Sources: ( doc|docs)` tail (happy path only; zero-doc + `LLMError` rows byte-identical; `build_overview_prompt`/`load_kb_overview`/`build_kb_section` untouched). +- Verified: `TOOLS_SECTION` grep sentence updated + D4 search-discipline sentence present **verbatim**, immediately after the grep sentence; `PERSONA`/LOW pins byte-identical (`test_prompt_lock.py`). +- Measured evidence (in `.agents/reports/162_…/task04_prefix_evidence_and_battery.md` + re-measured every run by `test_under_rank_is_measured_not_assumed`): pre-fix tokens `['build','best']` → IKEA doc rank 6 (outside top-5); post-fix `['build','besta']` → suggested #1, cosine unchanged 0.382. +- `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated); neighbours `test_kb_overview{,_freshness} test_name_signal_read_chips test_grep_regex_teaching test_harness_aligned_tools test_agent_document_tools test_tool_call_compact` → **24 passed**. +- `uv run pytest --cov=app --cov-report=term-missing` → **3756 passed**, app coverage **99%** (>90%). +- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**. +- Completion criteria: fold both ways ✅ · `Sources:` tail ✅ · TOOLS_SECTION + D4 verbatim + re-cut pins ✅ · E2E green/battery recorded/full suite/coverage/lint ✅ · docs pass ✅ · commit + dir move = harness. +- Battery: 5 phase-119 questions byte-identical to the phase-119 record (GATE PASS); the 2 Besta questions **skipped loudly** — verified the 2026-09-05 replica dump has zero `ikea` matches (`grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0); rank gate stays with the owner's live KB. +- Docs fixed: README — overview-tail doc + `BOR_KB_OVERVIEW_MAX_CHARS` tail-cut constraint, the phase-162 search-discipline paragraph, per-phase (not per-story) E2E wording, new "Retrieval regression battery" section; AGENTS.md — rule 4 per-phase wording + battery command in Quick reference. (The overview's expected "AGENTS.md §6 tool table" does not exist in this repo — the grep contract line lives in README's tool table, already updated.) +- Deviations (recorded, not assumed): `_fold` adds `casefold` (required for the pinned `ß → ss`); `①` NFKD-decomposes to `1` (the overview's parenthetical was wrong) and is excluded by `NAME_TOKEN_MIN_LEN`; pre-fix evidence lives in the reports dir because task files are read-only for this pass. +- Owner follow-up A6 (runtime, not code): add the Tuning-tab steering note "IKEA's BESTÅ assembly PDF is a live source — search it as `BESTÅ` or `AA-1272068`". +- Next pending phase: none — 162 is the only entry in `todo/` (harness moves it to `complete/`). diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__00_phase.a1.md.status b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__00_phase.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__00_phase.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__00_phase.a1.validate b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__00_phase.a1.validate new file mode 100644 index 0000000..fee922f --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__00_phase.a1.validate @@ -0,0 +1,124 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 5%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 13%] +........................................................................ [ 15%] +........................................................................ [ 17%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 24%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 32%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 46%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 51%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 70%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 84%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 97%] +........................................................................ [ 99%] +............ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 277 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 196 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 204 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 266 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 135 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 454 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 764 4 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 156 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 470 0 100% +-------------------------------------------------- +TOTAL 5788 17 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__01_accent_fold_name_norms.a1.err b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__01_accent_fold_name_norms.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__01_accent_fold_name_norms.a1.md b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__01_accent_fold_name_norms.a1.md new file mode 100644 index 0000000..034b066 --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__01_accent_fold_name_norms.a1.md @@ -0,0 +1,13 @@ +All gates green. Final verification complete. + +**Report** + +- **Implemented (task 162/01, D1 / LOCKED A1):** + - `app/rag/retriever.py`: new module-level `_fold()` (stdlib NFKD + combining-class strip, shared with task 03's grep); `_normalize_name` = alnum scan of `_fold(s.lower())` — the single normalization both sides pass through; `name_hit_tokens`/`_name_parts`/`_name_token_matches` consumed it unchanged (docstring clauses only); module docstring normalization paragraph states the 2026-10-07 gap. + - `tests/unit/test_retriever.py`: +9 tests — `_fold` cases (Å/ÄÖÜ/ß/fi/ç, CJK, ①), `BESTÅ`→`besta` token pinned exactly (`["build", "besta"]`), `_name_parts` (`besta` in equal set unchanged; `BÅsta.md`→`basta`), end-to-end bridge both directions via `_name_hit_chunks`, min-length on the folded form, digitless-precision non-regression (`déploy` ≠ `Deployments`). +- **Tests/lint/coverage:** `uv run pytest tests/unit/test_retriever.py tests/unit/test_retriever_recency.py -v` → 94 passed; full `uv run pytest` → **3747 passed**; `--cov=app` → 99% TOTAL, retriever 99% (3 pre-existing misses); `uv run ruff check .` + `uv run pyright` clean; phase-119 E2E `tests/e2e/test_name_signal_read_chips.py` → 4 passed (FTS/gate suites untouched). +- **Docs:** README "How retrieval works" — lexical list now names the name-hit path + accent fold (phase 162). No AGENTS.md impact (its grep-contract line is task 03/final-pass scope). +- **Decisions/deviations:** + 1. NFKD alone leaves `ß` intact; the task-pinned `ß → ss` case requires `str.casefold()` — applied inside `_fold` (still stdlib-only, single shared function, A1 intent intact; documented). + 2. `①` is compat-decomposable (NFKD → `1`), contrary to the task's "no decomposition" note: it still drops as a token (<4 chars); its only footprint is a prefix-class versioned-join artifact (`the1`) no path can match — pinned as real behavior and documented. +- **Next pending task:** `162_search_recall_small_sources/02_kb_overview_source_inventory.md`. diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__01_accent_fold_name_norms.a1.md.status b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__01_accent_fold_name_norms.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__01_accent_fold_name_norms.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__01_accent_fold_name_norms.a1.validate b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__01_accent_fold_name_norms.a1.validate new file mode 100644 index 0000000..c948b2f --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__01_accent_fold_name_norms.a1.validate @@ -0,0 +1,124 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 5%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 13%] +........................................................................ [ 15%] +........................................................................ [ 17%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 24%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 32%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 46%] +........................................................................ [ 48%] +........................................................................ [ 49%] +........................................................................ [ 51%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 73%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 84%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 96%] +........................................................................ [ 97%] +........................................................................ [ 99%] +... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 277 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 196 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 204 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 266 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 135 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 454 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 764 4 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 155 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 470 0 100% +-------------------------------------------------- +TOTAL 5780 17 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__02_kb_overview_source_inventory.a1.err b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__02_kb_overview_source_inventory.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__02_kb_overview_source_inventory.a1.md b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__02_kb_overview_source_inventory.a1.md new file mode 100644 index 0000000..858d1c8 --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__02_kb_overview_source_inventory.a1.md @@ -0,0 +1,17 @@ +All gates are green. Task 02 is complete. + +**Report — task 02 `02_kb_overview_source_inventory.md`** + +- **Implemented** (verified/completed in working tree — a prior interrupted run had left the code/tests uncommitted; I audited them against D2/A2, confirmed byte-identity of `build_overview_prompt`/`OVERVIEW_INSTRUCTION`/`load_kb_overview`/`build_kb_section`, and finished the doc work): + - `_source_inventory(rows)` in `app/rag/overview.py` — deterministic, non-LLM `Sources: ( doc|docs), …`, distinct sources, catalog order, singular/plural. + - `regenerate_overview` happy path stores model text (rstripped `\n`) + `\n` + tail; log `chars=` = full stored content; zero-doc and `LLMError` paths untouched (previous row + tail preserved). + - Module + function docstrings record phase 162 / A2 / the `BOR_KB_OVERVIEW_MAX_CHARS` budget note. + - Unit tests in `tests/unit/test_overview.py`: format/order/counts, 1-doc survival (`ikea_instructions` case), exact happy-path stored content, tail-less zero-doc row preserved, `LLMError` preserves an existing tail, model input never gains the tail, exact char log; re-cut pins in `tests/integration/test_import_docs_overview.py`, `tests/e2e/test_kb_overview.py`, `tests/e2e/test_kb_overview_freshness.py`. +- **Tests / lint / coverage** (DB container started — it was down): + - `uv run pytest tests/unit/test_overview.py -v` → 29 passed + - `uv run pytest --cov=app --cov-report=term-missing` → **3753 passed**, `app/` TOTAL **99%**, `app/rag/overview.py` **100%** + - `uv run pytest tests/e2e/test_kb_overview.py -v --no-cov` → 4 passed; `tests/e2e/test_kb_overview_freshness.py` → 3 passed + - `uv run ruff check .` → clean; `uv run pyright` → 0 errors +- **Docs**: `README.md` — import section now documents the deterministic `Sources: …` tail (phase 162) and the failure-path preservation; `AGENTS.md` — no doc impact (the grep-contract line there is task 03/04's scope). +- **Decisions**: inventory order via `sorted()` on source names = catalog order regardless of the recency-first row order the generator reads (proven by the scrambled-input test); no other e2e asserts exact overview-row content (checked). +- **Next pending task**: `03_grep_folding_search_discipline.md`. diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__02_kb_overview_source_inventory.a1.md.status b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__02_kb_overview_source_inventory.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__02_kb_overview_source_inventory.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__02_kb_overview_source_inventory.a1.validate b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__02_kb_overview_source_inventory.a1.validate new file mode 100644 index 0000000..5eff6bc --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__02_kb_overview_source_inventory.a1.validate @@ -0,0 +1,124 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 5%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 13%] +........................................................................ [ 15%] +........................................................................ [ 17%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 24%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 32%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 46%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 51%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 70%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 84%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 95%] +........................................................................ [ 97%] +........................................................................ [ 99%] +......... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 277 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 196 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 204 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 266 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 135 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 454 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 764 4 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 155 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 470 0 100% +-------------------------------------------------- +TOTAL 5787 17 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__03_grep_folding_search_discipline.a1.err b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__03_grep_folding_search_discipline.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__03_grep_folding_search_discipline.a1.md b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__03_grep_folding_search_discipline.a1.md new file mode 100644 index 0000000..c8e5ebb --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__03_grep_folding_search_discipline.a1.md @@ -0,0 +1,22 @@ +Task 03 complete. Report: + +**Implemented** +- `app/rag/agent.py` — `grep_document` now folds **both** pattern and each line through the shared `_fold` imported from `app.rag.retriever` (one fold, no second copy; LOCKED A3): `_fold(pattern.lower()) in _fold(line.lower())`; rstrip/line-numbers/20-cap/200-char truncation/locator semantics untouched. +- Copy updates (LOCKED A3): module-docstring grep paragraph, `AGENT_TOOLS` grep function description, and `pattern` parameter description now state "case- and accent-insensitive — accents fold: `besta` matches `BESTÅ`"; `NO_MATCHES_REGEX`/`_SCOPED` + `looks_like_regex`/`plain_form` byte-identical. +- `app/rag/prompts.py` — `TOOLS_SECTION` grep sentence folded-contract re-worded; new `SEARCH_DISCIPLINE_RULE` constant (LOCKED A4 sentence, verbatim) inserted immediately after the grep sentence; `PERSONA`/LOW/other sentences byte-identical. +- Pins re-cut (phase-124 pattern): `test_prompt_lock.py` (new sha `988642f9…`/len 3818, pre-phase-162 anchors, chained phase-162→145 diff-style pin, discipline-rule position pin, grep-contract pin), `test_prompts.py` (grep clause + D4 boundary pins), `test_agent.py` (`AGENT_TOOLS` + pattern-param description pins). +- New fold tests in `test_agent.py`: accented↔ASCII both ways (`besta`→`BESTÅ`, `basta`→`BÅsta`, `cafe`→`café`, `BESTÅ` pattern→ASCII content, `strasse`→`Straße`), non-match still `[]`. +- `README.md` — grep tool table row updated to the folded contract. + +**Tests / lint / coverage** +- `uv run pytest tests/unit/test_agent.py tests/unit/test_prompt_lock.py tests/unit/test_prompts.py` → **292 passed** +- `uv run pytest` (full suite) → **3756 passed** +- Related E2E spot-checks: `test_grep_regex_teaching.py`, `test_tool_call_compact.py`, `test_kb_overview*.py`, `test_name_signal_read_chips.py` → **green** (11 passed in the last batch) +- `uv run pytest --cov=app` → TOTAL **99%** (`agent.py` 99%, `prompts.py` 100%) — >90% gate met +- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors + +**Docs**: `README.md` grep-tool row updated (case- and accent-insensitive, fold example); `AGENTS.md` has no grep-contract line to change (the phase-162 final pass handles the §6 wording per the phase overview). + +**Decisions**: reworked the phase-145 "only byte change" pin into a chained phase-162→145 revert pin so both historical anchors stay enforced; `SEARCH_DISCIPLINE_RULE` added as a module constant mirroring the `ORIGIN_RULE` pattern. No deviations from LOCKED A3/A4. + +**Next pending task**: `162_search_recall_small_sources/04_e2e_battery_pins.md` diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__03_grep_folding_search_discipline.a1.md.status b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__03_grep_folding_search_discipline.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__03_grep_folding_search_discipline.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__03_grep_folding_search_discipline.a1.validate b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__03_grep_folding_search_discipline.a1.validate new file mode 100644 index 0000000..fee922f --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__03_grep_folding_search_discipline.a1.validate @@ -0,0 +1,124 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 5%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 13%] +........................................................................ [ 15%] +........................................................................ [ 17%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 24%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 32%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 46%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 51%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 70%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 84%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 97%] +........................................................................ [ 99%] +............ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 277 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 196 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 204 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 266 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 135 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 454 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 764 4 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 156 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 470 0 100% +-------------------------------------------------- +TOTAL 5788 17 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__04_e2e_battery_pins.a1.err b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__04_e2e_battery_pins.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__04_e2e_battery_pins.a1.md b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__04_e2e_battery_pins.a1.md new file mode 100644 index 0000000..8d63c68 --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__04_e2e_battery_pins.a1.md @@ -0,0 +1,18 @@ +**Phase 162 / task 04 — complete, all gates green.** + +**Added** `tests/e2e/test_search_recall_small_sources.py` (7 tests), fixture KB `tests/fixtures/recallkb/` (3 sources: `ikea_instructions` 1 doc with the name **only** as `BESTÅ`, `phase_docs` 4 + `players_handbook` 1 ASCII-`besta`/`build` distractors), the Besta pair + gate comment in `tests/fixtures/retrieval_battery.txt`, and evidence in `.agents/reports/162_search_recall_small_sources/task04_prefix_evidence_and_battery.md`. + +**Commands / outcomes** +- `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated, run twice incl. after the DB restore). +- `uv run pytest tests/e2e/test_name_signal_read_chips.py tests/e2e/test_kb_overview.py -q --no-cov` → **8 passed**. +- `uv run pytest --cov=app --cov-report=term-missing` → **exit 0, 3756 tests, app coverage 99%**. +- `uv run ruff check .` → passed; `uv run pyright` → **0 errors**. +- Replica restored per the `restore-test-db` skill (fingerprint verified docs=1000/chunks=8866/sources=deploy,homelab,homepage,ServMon,skills,vibe-bot, alembic 0011→0026), then `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` → **GATE PASS 1–5**: gitea README rank 4, forgejo 5/5 with README #1, gateway README rank 3, qwen3.8-27b quadlets #1/#2, Mongolia HIGH/fts_hits=5 — the five `suggested:` lists are **byte-identical** to the phase-119 record, and a re-run with the fold reverted is **byte-identical** to the post-fix run. + +**Pre-fix evidence (measured, both directions).** With `app/rag/retriever.py::_normalize_name` reverted by a one-line reversible edit, `How do I build the BESTÅ?` loses the doc: tokens `['build','best']` → no name hit, suggested 5 = the five ASCII distractors, IKEA doc **rank 6** — visible on the wire as its `done.related` entry and its absence from the folder-map echo (gate still HIGH, cosine 0.382, on the wrong docs). Shipped code: tokens `['build','besta']` → IKEA doc **suggested #1**, related = `phase_docs/156_history_source_filter.md`, cosine unchanged (0.382). The hypothesis is now re-measured on every run by `test_under_rank_is_measured_not_assumed`. + +**Decisions / deviations** (harness mechanics win): task/phase files untouched, so the evidence lives in the report file above; no `git stash` (reversible source revert instead, verified restored, `grep -c TEMP` = 0); `done.suggestions` is the deflection-chip contract and never carries paths, so the suggested tier is asserted via the durable `query_log.sources` record (suggested 5 + remainder), `done.related`, and the mock's folder-map echo; tuning re-worded two bodies to drop stopword density so the out-of-KB control deflects (cosine 0.3345 → 0.2083) — both token controls are pinned as assertions; the battery carries both `How do I build a Besta?` (task text) and `How do I build the BESTÅ?` (the live miss). + +**Skipped loudly:** the 2026-09-05 replica snapshot has no `ikea_instructions`-class source (`grep -c -i ikea` on the dump = 0), so the Besta rank gate stays with the owner's live KB; both questions still ran without a false grounding on a doc naming the product. + +**Docs:** no README/AGENTS change needed (task 04 is tests + fixtures + battery; user-visible behavior was documented by tasks 01–03). DB left as found (pre-restore backup restored); no other pending task for this phase — task 04's file is ready for the harness to move, and the phase commit is still owed. diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__04_e2e_battery_pins.a1.md.status b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__04_e2e_battery_pins.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__04_e2e_battery_pins.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__04_e2e_battery_pins.a1.validate b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__04_e2e_battery_pins.a1.validate new file mode 100644 index 0000000..fee922f --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/162_search_recall_small_sources__04_e2e_battery_pins.a1.validate @@ -0,0 +1,124 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 5%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 13%] +........................................................................ [ 15%] +........................................................................ [ 17%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 24%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 32%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 46%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 51%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 70%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 84%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 92%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 97%] +........................................................................ [ 99%] +............ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 277 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 196 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 204 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 266 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 135 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 454 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 764 4 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 156 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 470 0 100% +-------------------------------------------------- +TOTAL 5788 17 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/162_search_recall_small_sources/task04_prefix_evidence_and_battery.md b/.agents/reports/162_search_recall_small_sources/task04_prefix_evidence_and_battery.md new file mode 100644 index 0000000..5a7e65c --- /dev/null +++ b/.agents/reports/162_search_recall_small_sources/task04_prefix_evidence_and_battery.md @@ -0,0 +1,174 @@ +# Task 04 record — pre-fix evidence + battery (phase 162, D5/D6) + +**Deviation note (mechanics):** the task file says "record the measured +pre-fix ranks in THIS file" and to revert the fold with `git stash`, but +the execution harness forbids editing phase/task files and forbids +index/history-touching Git commands. Per the conflict rule the harness +wins on mechanics and the intent is preserved: the evidence lives HERE, +the fold was reverted by a one-line, reversible edit of +`app/rag/retriever.py` (restored and byte-verified afterwards — +`grep -c TEMP app/rag/retriever.py` = 0), and the summary goes in the +task's final report. The permanent, non-throwaway form of the evidence is +`tests/e2e/test_search_recall_small_sources.py::test_under_rank_is_measured_not_assumed`, +which re-measures BOTH directions on every run. + +## 1. The fixture (tests/fixtures/recallkb — three local-directory sources) + +| source | docs | role | +| --- | --- | --- | +| `ikea_instructions` | **1** (`besta-frame-white__AA-1272068-11-100.md`) | the live miss: body names the product **only** as accented `BESTÅ` | +| `phase_docs` | 4 (`140_pictorial_pdf_pages`, `148_pdf_text_backfill`, `152_seaweedfs_blob_ingest`, `156_history_source_filter`) | ASCII `besta` + `build` distractors | +| `players_handbook` | 1 (`game_night_setup.md`) | ASCII `besta` + `build` distractor in a second small source | + +Final shape tuning (the task's "add distractors until it loses"): the +initial 4-doc draft already lost the tier, but the deflection control +failed (best cosine 0.3345 — the small source's `Assembly` section was +dense in the stopword `the`, which the mock's tokenizer keeps). Two +bodies were re-worded (`ikea_instructions` Assembly paragraph, +`players_handbook/game_night_setup.md`) to drop stopword density: the +out-of-KB cosine fell to **0.2083** (< 0.30 threshold → deflects) while +the on-topic questions kept 0.38 / 0.44. Two token controls are pinned +as assertions in `synced_kb`: (a) no ASCII `besta` and no `build` anywhere +in the small source, (b) the accented name sits **past** the mock's +24-token summary-digest window, so the ASCII-only digest never leaks a +`best` lexeme the pre-phase FTS could have hit. + +## 2. Measured pre-fix evidence — the E2E fixture, both directions + +Lens: the app's REAL `retrieve()` + `select_suggested()` over the synced +KB with the mock's embeddings. Gate: threshold 0.30, lexical-support +floor 0.15 (the conftest mock calibration). Bonus 0.005, recency 0.0007, +suggested 5 / related 2 (code defaults, `.env`-free like production). + +Question `How do I build the BESTÅ?` + +**PRE-FIX — the real revert** (`_normalize_name` → +`"".join(_ALNUM_RE.findall(s.lower()))`, the exact task-01 before-state), +`pytest tests/e2e/test_search_recall_small_sources.py` run against the +fixture (full log: `/tmp/prefix_e2e_run.log`, `/tmp/prefix_wire_run.log`): + +``` +name tokens : ['build', 'best'] → NO name hit +suggested 5 : phase_docs/140_pictorial_pdf_pages.md + phase_docs/148_pdf_text_backfill.md + phase_docs/152_seaweedfs_blob_ingest.md + players_handbook/game_night_setup.md + phase_docs/156_history_source_filter.md +related : ikea_instructions/besta-frame-white__AA-1272068-11-100.md ← RANK 6, the miss +gate : best cosine 0.382, fts_hits 8 → grounded (on the WRONG docs) +done frame : {'source': 'ikea_instructions', + 'path': 'besta-frame-white__AA-1272068-11-100.md', + 'title': 'Assembly Instructions'} ← the rank-6+ "Nearby docs" slot +folder echo : (folders: phase_docs/: 148_…, 152_…, 156_…; players_handbook/:) + → NO `ikea_instructions/` line at all: the small source never + reached the HIGH prompt +``` + +**POST-FIX (shipped code)** — same fixture, same measurement: + +``` +name tokens : ['build', 'besta'] → name hit +suggested 5 : ikea_instructions/besta-frame-white__AA-1272068-11-100.md ← #1 + phase_docs/140_pictorial_pdf_pages.md + phase_docs/148_pdf_text_backfill.md + phase_docs/152_seaweedfs_blob_ingest.md + players_handbook/game_night_setup.md +related : phase_docs/156_history_source_filter.md +gate : best cosine 0.382, fts_hits 9 (the fold adds the LEXICAL term; + cosine untouched — LOCKED A1) +folder echo : … includes `ikea_instructions/:` (header alone — its only + document IS the suggested doc) +``` + +Mechanism (measured, and why the fix is the name signal and nothing +else): every distractor matches the question's other FTS token `build`, +so each fuses TWO RRF terms (vector + FTS ≈ 0.032) while the small source +— no `build`, and `BESTÅ` lexes as `bestå`, which the ASCII-`best` +tsquery cannot match — fuses ONE (≈ 0.016) and lands rank 6. The folded +name hit is the only signal big enough to lift it: name hits LEAD the +lexical list (`fuse` rank 1 = 1/61) plus the per-document +`BOR_NAME_HIT_BONUS` (0.005). ASCII question `How do I build a Besta?` +measures the SAME tier before and after (suggested #1, cosine 0.4404, +related `phase_docs/156_history_source_filter.md`) — the phase-119 path +is untouched. + +Under-rank hypothesis (D1 failure #4): **pinned green with measured +ranks**, not assumed. + +## 3. The dedicated suite + +`uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` +→ **7 passed** (task-04 cases 1–4 plus the measurement test, the grep +fold at the content layer, and the folder-line prompt witness). + +Interpretation note: the task text says the `done`-frame `suggestions` +list is the observable tier. On the shipped wire `done.suggestions` is +the DEFLECTION contract (2–3 title-derived "Maybe try" chips, `[]` on +grounded turns — `app/api/chat.py`, phase 04) and never carries document +paths. The faithful observables of the SUGGESTED tier are (a) the durable +`query_log.sources` record = suggested 5 then the rank-6+ remainder +(phase 118 A3) and (b) `done.related` = the rank-6+ tier. The suite +asserts both (plus `done.sources == []` — read-docs-only, phase 119 +LOCKED A1 — and the mock's folder-map echo for the prompt-level witness). + +Neighbour suites re-run (the retriever + overview were touched by this +phase): `uv run pytest tests/e2e/test_name_signal_read_chips.py +tests/e2e/test_kb_overview.py -q --no-cov` → **8 passed**. + +## 4. Battery — `tests/fixtures/retrieval_battery.txt` + +The file gains the Besta pair (`How do I build a Besta?` — the task's +line — and `How do I build the BESTÅ?`, the live miss itself) with the +gate comment. The five phase-119 questions keep positions 1–5. + +Preconditions (the `restore-test-db` skill, run in-container): safety +backup → `DROP SCHEMA public CASCADE` → single-transaction load of +`data/bor_live_kb_replica.dump.sql` → **fingerprint verified: +docs=1000, chunks=8866, +sources=deploy,homelab,homepage,ServMon,skills,vibe-bot, alembic=0011** +→ `uv run alembic upgrade head` (0011 → **0026**) → docs=1000 re-verified. + +`uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` +(full output `/tmp/battery_postfix.log`, pre-fix comparison +`/tmp/battery_prefix.log`): + +| # | question | verdict | +| --- | --- | --- | +| 1 | How do I deploy gitea? | HIGH, `fts_hits=38`; suggested top-5 **byte-identical** to the phase-119 record — `gitea/README.md` **rank 4** ✓ | +| 2 | How do I deploy forgejo? | HIGH, `fts_hits=37`; forgejo tree 5/5, `forgejo/README.md` **#1** ✓ | +| 3 | How do I set up the gateway? | HIGH, `fts_hits=37`; `gateway/README.md` **rank 3** ✓ | +| 4 | Do you have notes on the qwen 3.8 27b model? | HIGH, `fts_hits=31`; both `qwen3.8-27b-*` quadlets at #1/#2 ✓ (the digit-prefix regression holds) | +| 5 | What is the capital of Mongolia? | **HIGH, `fts_hits=5`** — unchanged ✓ | +| 6 | How do I build a Besta? | **SKIPPED LOUDLY — source absent in the replica snapshot** | +| 7 | How do I build the BESTÅ? | **SKIPPED LOUDLY — source absent in the replica snapshot** | + +`SELECT DISTINCT source FROM documents` on the restored replica lists +`deploy, homelab, homepage, ServMon, skills, vibe-bot` — NO +`ikea_instructions`-class source (`grep -c -i ikea +data/bor_live_kb_replica.dump.sql` = 0: the 2026-09-05 snapshot predates +the IKEA import, exactly the case D5 anticipated; phase-140's +"skipped loudly" precedent). The two questions still ran and did not +crash; both return the `build.sh`/`bifrost`-class top-5 that a KB without +the product can only return, and neither grounds on a document that names +the product. **The rank gate therefore stays with the owner** — it can +only close on a KB that has the source (the live KB). + +Behavior-preservation, measured (not argued): + +* the five phase-119 `suggested:` lists are **byte-identical** to the + recorded 2026-09-16 run in `.agents/reports/119_name_signal_read_chips/task06_battery_and_e2e.md`; +* the battery re-run with the task-01 fold REVERTED is **byte-identical** + to the post-fix run (`diff` of the question/gate/suggested lines: no + difference) — the fold cannot move an ASCII-only corpus, as designed; +* `best_cosine` moves by ≤ 0.0004 between the September record and today + (real-`embed` model jitter, not code). + +**Verdict: GATE PASS** for questions 1–5 (unchanged), 6–7 recorded as +skipped loudly with the reason and the re-check command. + +## 5. Environment left behind + +The replica was restored only for the battery. The pre-restore DB state +(backup `/tmp/pre_restore_host.dump`, 13 leftover E2E fixture docs, +alembic 0026) is restored afterwards — reload the replica any time with +the `restore-test-db` skill. diff --git a/AGENTS.md b/AGENTS.md index dd1c1ed..14e2f27 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,8 +9,9 @@ 3. **Strictly adhere to the LOCKED DECISIONS** (PLAN §2). If you believe a LOCKED decision is wrong, stop and flag it — do not silently deviate. 4. **"One Story, One Phase":** each user story in `.agents/user_stories/` - corresponds to a distinct execution phase that includes its own dedicated - Playwright E2E test suite (one test file per story, run in isolation). + corresponds to a distinct execution phase, and **every** phase — story-driven + or owner-directed — includes its own dedicated Playwright E2E test suite + (one test file for the phase, run in isolation). 5. **"UI Structure Check":** before finalizing any UI component, verify that it follows the layout principles in `.agents/PLAN.md` §7 (proper container usage, the 72rem centered reading column — every page rides the 72rem @@ -57,4 +58,5 @@ uv run pytest --cov=app --cov-report=term-missing uv run playwright install chromium # once uv run pytest tests/e2e/test_smoke.py -v --no-cov uv run ruff check . && uv run pyright # lint + types +uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt # retrieval regression battery (phases 119/162) ``` diff --git a/README.md b/README.md index 45e7ba9..a900e0a 100644 --- a/README.md +++ b/README.md @@ -280,7 +280,12 @@ reflect the new files. - A run that changed the KB also regenerates the **KB overview** and the **folder summaries** (the drill-down catalog's stored per-folder descriptions) with the `lite` model — best-effort: a summary failure - never fails the import. Because the sync only rewrites the folders a + never fails the import. The stored overview ends with a deterministic + `Sources: ( doc|docs), …` line — every source with its exact + document count, catalog order, computed without a model call — so no + source (not even a 1-doc one) is invisible to `` + (phase 162); the `LLMError`/zero-doc paths leave the previous row + (tail included) untouched. Because the sync only rewrites the folders a change touches, a one-shot `uv run python -m scripts.regen_folder_summaries` refreshes EVERY stored folder summary in one run (full mode: every candidate folder @@ -492,11 +497,20 @@ server-side tools — on **grounded** (high-relevance) turns only: |------|-------------| | **`ls`** | Lists the knowledge base as a **drill-down tree**, one level at a time: no `path` = every source (document count + stored description), a source name = that source's top level, a `source/folder` path = one level deeper; file lines are `source: X \| path: Y \| title: Z` | | **`read(path, offset?, limit?)`** | Appends one indexed document's text to context by its combined `source/path`; optional `offset` (1-based first line) + `limit` (line count) read just that part of a large document; very large reads are truncated at `BOR_READ_MAX_CHARS` with a notice that teaches the workflow — grep reports line numbers, then read that part with `offset` + `limit` | -| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case-insensitive fixed substring); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document | +| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case- and accent-insensitive fixed substring — accents fold: `besta` matches `BESTÅ`, phase 162); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document | | **`show_image(path)`** | *(only when `BOR_ANSWER_IMAGES=1`)* shows one standalone image document as part of the answer — image documents only, at most one per answer | (See [Answer images](#answer-images-phase-124) for the `show_image` tool.) +**The prompt teaches the search discipline (phase 162).** A summary or an +outline is a hint, never an inventory: the system prompt tells the model to run +`ls` with no `path` — the full source list — before ever claiming the knowledge +base lacks something, and to `grep` the artifact's part number or a filename +token. One grep pattern is one string, so a name's split tokens (`frame`, +`white`) and its part number (`AA-1272068`) each need their own grep; and a +note saying a file is gitignored describes its **repo** status, not whether it +is indexed as a source. + **Git-source provenance (phase 145).** When a document comes from a git source, its `read` result also carries an `origin:` line (the repository's URL, credentials stripped) and, when the source pins one, a `branch:` line. @@ -583,7 +597,12 @@ stemming) and searched **twice** against Postgres: 1. **Vector** — pgvector cosine top-N (default `BOR_HYBRID_VECTOR_CANDIDATES=100`) 2. **Lexical** — a stored `tsvector` (GIN-indexed) matched with `to_tsquery`, - top-N by `ts_rank` (default `BOR_HYBRID_LEXICAL_CANDIDATES=30`) + top-N by `ts_rank` (default `BOR_HYBRID_LEXICAL_CANDIDATES=30`), led by + **name hits** — documents whose *path* matches a question token (`gitea` + → `gitea/…`, `qwen38` → `qwen3.8-27b-…`; phase 119). Name normalization + is **accent-folded** (phase 162): a question's `BESTÅ` folds to `besta` + and matches the ASCII filename, and an accented filename (`BÅsta.md`) + matches an ASCII question — both directions, stdlib fold only. The two ranked lists are fused with **Reciprocal Rank Fusion** (`score = Σ 1/(k + rank)`, `BOR_RRF_K=60`) — a chunk in both lists scores @@ -917,7 +936,7 @@ to `.env` and adjust. | `BOR_MAX_OUTPUT_TOKENS` | `32768` | Max answer length in tokens (answers must not be cut off) | | `BOR_STEERING_MAX_CHARS` | `8000` | Char budget for the `` (steering notes) prompt section | | `BOR_SUMMARY_MAX_CHARS` | `12000` | Cap on document content sent to the `lite` summary model | -| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | Char budget for the `` prompt section | +| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | Char budget for the `` prompt section — the stored outline **plus** its deterministic `Sources: …` tail (phase 162); the cap keeps the longest-fitting prefix, so a value below the default clips the tail first (the default always holds both) | | `BOR_OVERVIEW_INPUT_MAX_CHARS` | `40000` | Cap on the document list sent to `lite` for KB overview generation | | `BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS` | `8000` | Cap on a folder's document list sent to `lite` for folder summaries (phase 94) | @@ -1058,8 +1077,9 @@ normally until you attach. Override the port with `DEBUGPY_PORT`. ## QA / Testing -Three layers — the project rule is **one story, one phase, one Playwright -suite** (see `AGENTS.md`): +Three layers — the project rule is **one phase, one Playwright suite** (a +phase may implement a user story or an owner-directed change; see +`AGENTS.md`): ```bash # Unit + integration (FastAPI TestClient) @@ -1075,12 +1095,27 @@ uv run pyright # Playwright E2E — install the browser once: uv run playwright install chromium -# Each story's E2E runs IN ISOLATION: +# Each phase's E2E runs IN ISOLATION: uv run pytest tests/e2e/test_import_documents.py -v --no-cov uv run pytest tests/e2e/test_chat_rag.py -v --no-cov -# ...one file per story in .agents/user_stories/ (see .agents/phases/todo/) +# ...one file per phase in tests/e2e/ (the stories live in +# .agents/user_stories/, see .agents/phases/todo/) ``` +**Retrieval regression battery.** `tests/fixtures/retrieval_battery.txt` is +the tracked retrieval gate (phases 119 + 162): the name-your-tool questions, +the versioned-name (`qwen 3.8 27b`) regression, the out-of-KB deflection +sanity, and the `besta` / `BESTÅ` accent-fold pair. Run it against a KB with +`scripts.eval_retrieval` (the real pipeline, real embeddings): + +```bash +uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt +``` + +The exact expected ranks are recorded in the fixture's header comments; a +question whose source is missing from the KB snapshot is recorded as skipped +loudly, never silently dropped. + **Deterministic E2E:** by default the E2E app talks to a local **mock aipi** (`tests/e2e/mock_llm.py`) whose embeddings are real token-overlap vectors — so the cosine relevance threshold behaves like production. To diff --git a/app/rag/agent.py b/app/rag/agent.py index c0a100a..4ca502c 100644 --- a/app/rag/agent.py +++ b/app/rag/agent.py @@ -131,8 +131,10 @@ task 04): the only capped read (the phase-95 cap unchanged). And ``grep`` greps the indexed documents (or the one document a combined ``source/path`` - names) for a case-insensitive fixed substring and returns up to 20 - ``source/path:line: text`` match lines (owner-locked A5, phase 68), + names) for a case- and accent-insensitive fixed substring (accents + fold — ``besta`` matches ``BESTÅ``; phase 162, LOCKED A3) and + returns up to 20 ``source/path:line: text`` match lines + (owner-locked A5, phase 68), each line truncated to 200 chars. A grep is a **locator**, not a context-adder: it never appends to the answer context (only ``read`` does — ``holder.read_docs`` is untouched by a grep). @@ -312,7 +314,7 @@ from app.rag.llm import ( ToolResultPiece, chat_stream_retried, ) -from app.rag.retriever import TRUNCATION_MARKER +from app.rag.retriever import TRUNCATION_MARKER, _fold from app.rag.scaffolding import ScaffoldingFilter from app.rag.source_removal import resolve_source_name @@ -437,7 +439,8 @@ AGENT_TOOLS: list[dict[str, Any]] = [ "name": "grep", "description": ( "Search the indexed documents for an exact string " - "(case-insensitive) and return up to 20 matching lines " + "(case- and accent-insensitive — accents fold: `besta` " + "matches `BESTÅ`) and return up to 20 matching lines " "as `source/path:line: text` — a locator, not a " "context-adder: read the winner with `read` — for a " "large document, use the reported line number as " @@ -458,7 +461,9 @@ AGENT_TOOLS: list[dict[str, Any]] = [ "type": "string", "description": ( "The exact text to search for (a plain " - "substring, not a regex — no '.*', no " + "substring — case- and accent-insensitive, " + "accents fold: `besta` matches `BESTÅ` — " + "not a regex — no '.*', no " "'\\.', no character classes)" ), }, @@ -1560,16 +1565,21 @@ def format_covered_spans(spans: Sequence[tuple[int, int]]) -> str: def grep_document(content: str, pattern: str) -> list[tuple[int, str]]: """Every line of *content* that contains *pattern*, in file order. - Case-insensitive **fixed substring** (owner-locked A5: no regex — no - ReDoS surface, a simple contract for the model). Returns - ``(1-based line number, line.rstrip())`` pairs; an empty *content* - never matches a non-empty pattern. + Case- **and accent-insensitive fixed substring** (owner-locked A5: + no regex — no ReDoS surface, a simple contract for the model). Both + sides fold through the shared :func:`app.rag.retriever._fold` plus + lowercase (phase 162, LOCKED A3 — the fold changes the alphabet it + ignores, not the matching semantics: ``grep "besta"`` matches a + content line ``The BESTÅ frame``, and a pattern carrying accents + folds too), so grep and the name-hit path can never drift apart. + Returns ``(1-based line number, line.rstrip())`` pairs; an empty + *content* never matches a non-empty pattern. """ - needle = pattern.lower() + needle = _fold(pattern.lower()) return [ (number, line.rstrip()) for number, line in enumerate(content.split("\n"), start=1) - if needle in line.lower() + if needle in _fold(line.lower()) ] diff --git a/app/rag/overview.py b/app/rag/overview.py index e59f189..fea4929 100644 --- a/app/rag/overview.py +++ b/app/rag/overview.py @@ -28,6 +28,21 @@ an import added/updated at least one document (or no row exists yet), and a ``lite`` failure logs and leaves the previous outline intact — an old outline is better than none (phase locked decisions). +After the model's outline, :func:`regenerate_overview` appends ONE +deterministic, non-LLM line — ``Sources: ( doc|docs), …`` +(phase 162, LOCKED A2). The 2026-10-07 live finding: the ``lite`` +outline is lossy — it listed 3 groups of a 14-source index, so the +1-doc sources (``ikea_instructions``, ``players_handbook``) were +invisible to ````. Small sources must be visible +without a model call, so the line is computed mechanically from the +catalogue rows (:func:`_source_inventory`) — the mock/real model +outputs never affect it, and ``build_overview_prompt`` never sees the +tail (the model input stays byte-identical). The longest-prefix cap in +``build_kb_section`` applies to the WHOLE stored text: at the default +``BOR_KB_OVERVIEW_MAX_CHARS`` (4000) the ~1500-char outline plus the +one-line tail (~15 chars/source) always fit; a smaller owner-set cap +cuts the tail first — accepted, the cap is owner-tunable. + The ``KB_OVERVIEW_MODE`` marker follows the ``SUMMARY_MODE`` / ``DEFLECT_MODE`` convention: the deterministic E2E mock LLM keys on it in the system prompt (wired in phase 31, task 05). @@ -101,6 +116,35 @@ def _first_summary_line(summary: str | None) -> str: return "" +def _source_inventory(rows: Sequence[tuple[str, str, str, str | None]]) -> str: + """The phase-162 deterministic source inventory (LOCKED A2). + + Mechanical, non-LLM: one entry per **distinct** source in catalog + order (``source, path`` — first occurrence in the ``source, + path``-ordered rows), each `` ( doc|docs)`` — singular + ``doc`` at k=1, ``docs`` otherwise — joined with ``, `` behind the + ``Sources: `` prefix. The ``lite`` outline stays the model's job; + this line guarantees no source — not even a 1-doc one like the + 2026-10-07 ``ikea_instructions`` case — can be invisible to + ````, whatever the model outputs. The mock/real + model outputs never affect it. + + *rows* is the existing ``(source, path, title, summary)`` + projection :func:`regenerate_overview` already reads — no new + query. At the default ``BOR_KB_OVERVIEW_MAX_CHARS`` (4000) the + outline plus this one line always fit; a smaller owner-set cap + cuts the tail first (accepted — the cap is owner-tunable). + """ + counts: dict[str, int] = {} + for source, _path, _title, _summary in rows: + counts[source] = counts.get(source, 0) + 1 + entries = ", ".join( + f"{name} ({count} {'doc' if count == 1 else 'docs'})" + for name, count in sorted(counts.items()) + ) + return f"Sources: {entries}" + + def build_overview_prompt( rows: Sequence[tuple[str, str, str, str | None]], max_chars: int | None = None ) -> tuple[str, str]: @@ -166,10 +210,16 @@ async def regenerate_overview( * Zero documents → the existing row (if any) is left untouched and ``False`` is returned — no KB, no outline, no wasted model call. - * Happy path → the model's text is upserted into the single row - (``id = 1``, fresh UTC ``updated_at``), committed, and - ``overview: regenerated docs=… chars=…`` is logged; returns - ``True``. Idempotent by construction: one row, always ``id = 1``. + * Happy path → the model's text (trailing newlines rstripped) plus + the deterministic ``Sources: …`` inventory line (phase 162, + LOCKED A2 — :func:`_source_inventory`) is upserted into the + single row (``id = 1``, fresh UTC ``updated_at``), committed, and + ``overview: regenerated docs=… chars=…`` is logged (chars = the + FULL stored content, tail included); returns ``True``. Idempotent + by construction: one row, always ``id = 1``. The model input + never sees the tail; the zero-doc and ``LLMError`` paths below + leave a previous row byte-identical — an existing tail is part of + the stored content and is preserved on failure. * :class:`LLMError` → ``overview: regeneration failed — …`` is logged and ``False`` is returned with the previous row intact — an old outline is better than none. @@ -209,15 +259,21 @@ async def regenerate_overview( logger.error("overview: regeneration failed — %s", e) return False + # Phase 162 (LOCKED A2): the deterministic source inventory is + # appended to the STORED content after the model's outline — + # mechanical, non-LLM, so no source can be invisible to + # ``. The model input above never sees the tail. + content = text.rstrip("\n") + "\n" + _source_inventory(rows) + now = datetime.now(UTC) row = session.get(KbOverview, 1) if row is None: - session.add(KbOverview(id=1, content=text, updated_at=now)) + session.add(KbOverview(id=1, content=content, updated_at=now)) else: - row.content = text + row.content = content row.updated_at = now session.commit() - logger.info("overview: regenerated docs=%d chars=%d", len(rows), len(text)) + logger.info("overview: regenerated docs=%d chars=%d", len(rows), len(content)) return True finally: if owns_session: diff --git a/app/rag/prompts.py b/app/rag/prompts.py index 06ef067..324058d 100644 --- a/app/rag/prompts.py +++ b/app/rag/prompts.py @@ -265,6 +265,31 @@ ORIGIN_RULE: str = ( "again for the source." ) +#: The search-discipline rule (phase 162, task 03, LOCKED A4) — appended +#: to the ``TOOLS_SECTION`` tools paragraph immediately after the grep +#: sentence (i.e. after the ``…retry with the plain text you expect to +#: see.`` clause, the placement from the 2026-10-07 owner conversation +#: where the agent answered from the lossy ```` outline +#: instead of running ``ls``, and where the exact-substring grep silently +#: missed the accented ``BESTÅ`` in the IKEA PDF's text layer). Adapted +#: verbatim from the conversation's proposed system-prompt patch for the +#: folded grep (LOCKED A3). The three lessons: (1) an outline/summary is +#: a hint, never an inventory — ``ls`` (no ``path``) before claiming the +#: KB lacks something; (2) grep folds accents but one pattern is one +#: string — split filename tokens and the part number need separate +#: greps; (3) a gitignored note describes repo status, not index status. +#: The mock E2E is unaffected (it keys on markers, not this sentence). +SEARCH_DISCIPLINE_RULE: str = ( + "A summary or outline is a hint, never an inventory: before claiming " + "the knowledge base lacks something, run `ls` with no `path` to list " + "every source, and `grep` the artifact's part number or a filename " + "token. `grep` folds accents — `besta` matches `BESTÅ` — but one " + "pattern is one string: a name's split tokens (`frame`, `white`) and " + "its part number (`AA-1272068`) need separate greps. A note saying a " + "file is gitignored describes its repo status, not whether it is " + "indexed as a source." +) + TOOLS_SECTION: str = ( "\n" "You may extend your context with three tools. `ls` lists the " @@ -305,12 +330,15 @@ TOOLS_SECTION: str = ( "path as plain text otherwise. The user should be able to open the " "complete original from your answer in one step, not ask again for " "the source. `grep` locates an exact string " - "(case-insensitive) in the indexed documents and returns up to 20 " + "(case- and accent-insensitive — accents fold: `besta` matches " + "`BESTÅ`) in the indexed documents and returns up to 20 " "matching `source/path:line: text` lines — a locator, not a " "context-adder: read the winner with `read`. A grep pattern is a " "plain substring, NEVER a regex — '.*' and '\\.' are literal text " "there; if such a pattern returns no matches, retry with the plain " - "text you expect to see. For a normal search pass only `pattern` — " + "text you expect to see. " + + SEARCH_DISCIPLINE_RULE + + " For a normal search pass only `pattern` — " "its optional `path` argument " "limits the search to one document you already know, by the same " "combined `source/path` string; never a source name — a bare " diff --git a/app/rag/retriever.py b/app/rag/retriever.py index 1c9b78c..e7f4c0f 100644 --- a/app/rag/retriever.py +++ b/app/rag/retriever.py @@ -28,9 +28,21 @@ order ``(source, path)`` — the old total-matched-length tie-break is retired, it outranked 5-char product names by 6-char common tokens; capped at :data:`NAME_HIT_LIMIT`), - the FTS rows follow. This is what finds name-your-tool questions - ("gitlab") that vector similarity buries — and, the 2026-09-05 - incident, the versioned-name case the default parser lexes + the FTS rows follow. NAME NORMALIZATION IS ACCENT-FOLDED (phase 162, + LOCKED A1): :func:`_normalize_name` runs :func:`_fold` (NFKD + decomposition, combining marks dropped — stdlib only) before the + alnum scan, and it is the SINGLE normalization both sides of name + matching pass through (question tokens and path parts alike) — the + 2026-10-07 live gap this closes: the IKEA filename is ASCII + (``besta-frame-white__AA-1272068-11-100``) while its content layer + spells the name only as ``BESTÅ`` (U+00C5), and the old lower-only + normalization turned the accented question token into ``best`` — + never a match. With the fold, ``BESTÅ`` → ``besta`` (was ``best``) + and an accented filename (``BÅsta.md`` → ``basta``) answers to the + ASCII question — one function, both directions. This is what finds + name-your-tool questions ("gitlab") that vector similarity buries — + and, the 2026-09-05 incident, the versioned-name case the default + parser lexes incompatibly (question "Qwen 3.8" → ``qwen``/``3``/``8`` can never match a document's ``qwen3``/``8``/``27b`` tokens, while every unrelated llama.cpp quadlet out-ranks the target on the shared @@ -81,6 +93,7 @@ from __future__ import annotations import math import re +import unicodedata import uuid from collections.abc import Sequence from dataclasses import dataclass, replace @@ -130,9 +143,42 @@ _ALNUM_RE = re.compile(r"[a-z0-9]+") _STEM_SPLIT_RE = re.compile(r"[^a-z0-9]+") +def _fold(s: str) -> str: + """The shared accent fold (phase 162, LOCKED A1): NFKD decomposition, + combining marks dropped (stdlib ``unicodedata`` + ``str.casefold`` + only — no new dependency). + + ``Å/å → a``, ``Ä/Ö/Ü → a/o/u``, ``ß → ss``, ``fi → fi``, ``ç → c`` + (``casefold`` is the Unicode fold — the ``ß → ss`` case rides it: + NFKD alone leaves ``ß`` intact — and ``str.lower`` leaves the ``fi`` + ligature). Characters with no decomposition (CJK) survive the fold + and still drop out under the caller's alnum scan (behavior + unchanged for them); compatibility-decomposable characters (``①`` → + ``1``) fold to their ASCII base, which :data:`NAME_TOKEN_MIN_LEN` + already excludes as a name token (a digit base can still feed the + versioned join as the numeric NEXT token — the only route left is a + prefix-class candidate no real path matches). Case is folded too + (the caller lowercases anyway). Imported by ``app.rag.agent`` for + the folded ``grep`` (phase 162, LOCKED A3) — one fold, never + drifting apart. + """ + decomposed = unicodedata.normalize("NFKD", s.casefold()) + return "".join(c for c in decomposed if not unicodedata.combining(c)) + + def _normalize_name(s: str) -> str: - """Lowercased, alnum-only form of *s* (``Qwen 3.8`` → ``qwen38``).""" - return "".join(_ALNUM_RE.findall(s.lower())) + """Lowercased, ACCENT-FOLDED, alnum-only form of *s* (``Qwen 3.8`` → + ``qwen38`` unchanged; ``BESTÅ`` → ``besta`` — the phase-162 fold). + + This is the SINGLE normalization both sides of name matching pass + through — the :func:`name_hit_tokens` question tokens and the + :func:`_name_parts` path parts / file stem / stem sub-components — + so the one change bridges both directions (the ASCII question + ``besta`` keeps matching the ASCII filename, the accented question + ``BESTÅ`` now matches it too, and an accented filename ``BÅsta.md`` + answers to the ASCII question ``basta``). + """ + return "".join(_ALNUM_RE.findall(_fold(s.lower()))) def name_hit_tokens(question: str) -> list[str]: @@ -166,7 +212,10 @@ def name_hit_tokens(question: str) -> list[str]: Order of first appearance, de-duplicated. Matching applies the A2 rule: digit-bearing candidates prefix a normalized path part or file stem; digitless candidates equal a part, stem, or stem - sub-component — titles are never matched. + sub-component — titles are never matched. Candidates are + ACCENT-FOLDED through :func:`_normalize_name` (phase 162) — the + token ``besta`` drawn from ``BESTÅ`` name-matches ``BESTÅ``-class + names both ways. """ tokens = question.split() out: list[str] = [] @@ -456,6 +505,10 @@ def _name_parts(path: str) -> tuple[set[str], set[str]]: (the stem lowercased, split on non-alphanumeric runs, each piece normalized, empties dropped: ``kubernetes_gitea`` → ``kubernetes`` / ``gitea``): DIGITLESS tokens exact-match these. + + The parts are ACCENT-FOLDED through :func:`_normalize_name` (phase + 162) — ``besta`` name-matches ``BESTÅ``-class names both ways (the + accented stem ``BÅsta`` enters the equal set as ``basta``). """ p = Path(path) prefix = {_normalize_name(part) for part in p.parts} @@ -479,6 +532,10 @@ def _name_token_matches(token: str, prefix: set[str], equal: set[str]) -> bool: * the token HAS NO DIGIT → it EQUALS a normalized path part, file stem, or stem sub-component (``gitea`` → the ``gitea/`` folder, ``gitea.md``, ``kubernetes_gitea``, ``gitea-values``). + + Both the token and the path sets arrive ACCENT-FOLDED from + :func:`_normalize_name` (phase 162) — ``besta`` matches + ``BESTÅ``-class names both ways. """ if any(ch.isdigit() for ch in token): return any(part.startswith(token) for part in prefix) diff --git a/tests/e2e/test_kb_overview.py b/tests/e2e/test_kb_overview.py index 460de9a..ce97e79 100644 --- a/tests/e2e/test_kb_overview.py +++ b/tests/e2e/test_kb_overview.py @@ -307,8 +307,11 @@ def test_mock_generated_outline_is_stored_and_echoed( assert user.splitlines()[0].startswith( "docs — homelab/tables.md — Service Port Table" ), "the generator's list must lead with the NEWEST doc (recency-first)" + # Phase 162 (LOCKED A2): the stored content ends with the + # deterministic source-inventory line — one source, 13 docs. expected = ( - "Knowledge base outline:\n- docs homelab tables md service port table this" + "Knowledge base outline:\n- docs homelab tables md service port table this\n" + "Sources: docs (13 docs)" ) assert row.content == expected first_bullet = row.content.splitlines()[1].removeprefix("- ").strip() diff --git a/tests/e2e/test_kb_overview_freshness.py b/tests/e2e/test_kb_overview_freshness.py index ea40792..2251744 100644 --- a/tests/e2e/test_kb_overview_freshness.py +++ b/tests/e2e/test_kb_overview_freshness.py @@ -329,8 +329,10 @@ def test_regenerated_outline_leads_with_newest_doc( assert row is not None and row.id == 1 # The stored outline is the mock's 8-token digest of the capped list # — all 8 tokens from the NEWEST doc's source/path/title (computed - # from the fixture above, not hardcoded wording): - assert row.content == EXPECTED_DIGEST + # from the fixture above, not hardcoded wording) — and phase 162 + # (LOCKED A2) appends the deterministic source-inventory line after + # it: every fixture source named with its exact count. + assert row.content == EXPECTED_DIGEST + "\nSources: reese (5 docs)" assert "old" not in row.content # no oldest-batch token survives the cut diff --git a/tests/e2e/test_search_recall_small_sources.py b/tests/e2e/test_search_recall_small_sources.py new file mode 100644 index 0000000..4b7b7f4 --- /dev/null +++ b/tests/e2e/test_search_recall_small_sources.py @@ -0,0 +1,864 @@ +"""Phase 162 E2E (Playwright, mock-only): a 1-document source is BOTH +visible to ```` and winnable by its accented question. + +Run in isolation (DB must be up: ``podman compose up -d db``): + + uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov + +The 2026-10-07 live miss, replayed deterministically: the owner asked +``How do I build the BESTÅ?``; the KB held ``ikea_instructions/ +besta-frame-white__AA-1272068-11-100.md`` (1 document in the source, its +body carrying the name ONLY as ``BESTÅ``) and the question came back +empty-handed — ASCII ``besta`` documents outranked the one document that +actually named the product. Three separate losses, three pins here: + +* **D1 (the question-side fold)** — name matching never bridged the + accent: ``BESTÅ`` normalized to ``best``, ``besta-frame-…`` to + ``besta``, and the digitless rule EQUALS a stem component. Test 1 + pins the fix, test 2 MEASURES the miss. +* **D2 (the deterministic source inventory)** — the ``lite`` outline is + lossy, so a 1-doc source could be invisible to the model. Test 4 pins + the mechanical ``Sources:`` tail on the stored row. +* **D3 (the grep fold + search discipline)** — the stored accented body + stayed grep-matchable by the ASCII name. Test 6 pins the content + layer; the turn-level prose discipline is the prompt teaching + (``app/rag/prompts.py``, unit-pinned in task 03). + +KB fixture — ``tests/fixtures/recallkb`` (tracked), THREE local-directory +sources registered through the authenticated API + the REAL in-process +``POST /api/sync`` (the ``test_name_signal_read_chips.py`` pattern), six +markdown documents whose text is token-controlled so the hybrid gate, the +FTS list and the name-hit bonus pick the intended tiers deterministically +(``synced_kb`` pins the tiers with the app's REAL retrieval — a +fixture-text regression breaks at setup with a measured message): + +* ``ikea_instructions/`` — the small source: ONE document, the live + incident's own filename, its body naming the product ONLY as the + accented ``BESTÅ`` (no ASCII ``besta`` anywhere in the file). Two + token controls make the mechanism visible: no ``build`` (the question's + other FTS token) anywhere, and the accented name deliberately placed + PAST the 24-token digest window, so the mock summary chunk leaks no + ASCII ``best`` lexeme that a pre-fix FTS could have hit; +* ``phase_docs/`` (4 docs) + ``players_handbook/`` (1 doc) — the ASCII + distractors of the live index: every body carries ASCII ``besta`` AND + ``build``, so each of the five outranks the small source on + pre-fix retrieval (two fused terms — FTS + vector — against its one). + +Measured, never assumed (test 2 re-measures both directions; the same +numbers are recorded in +``.agents/reports/162_search_recall_small_sources/``). Accented question +``How do I build the BESTÅ?``, the app's real ``retrieve()`` + +``select_suggested()`` over this KB with the mock's embeddings (gate: +threshold 0.30, lexical-support floor 0.15 — the conftest mock +calibration; ``best cosine`` / ``fts`` are the A8 inputs): + + PRE-FIX (the phase-119 ``_normalize_name``, no accent fold) + name tokens ['build', 'best'] → NO name hit + suggested the five ASCII distractors + related ikea_instructions/besta-frame-white__… (rank 6) + best cosine 0.382, fts 8 (grounded, on the WRONG documents — + the live 2026-10-07 shape: an answer + about the ASCII ``besta`` notes) + POST-FIX (shipped) + name tokens ['build', 'besta'] → name hit + suggested THE SMALL SOURCE #1, then four distractors + related phase_docs/156_history_source_filter.md + best cosine 0.382, fts 9 (the fold adds the lexical term; the + cosine is untouched — LOCKED A1) + +The mechanism is the term COUNT: every distractor matches the question's +other FTS token ``build``, so each fuses TWO terms (vector + FTS) while +the small source — no ``build``, no accent-bridgeable lexeme — fuses ONE +and lands rank 6. The folded name hit is the only signal that can lift +it: name hits LEAD the lexical list (``fuse`` rank 1 ⇒ 1/61) and the +per-document ``BOR_NAME_HIT_BONUS`` (0.005) rides on top. The ASCII +question ``How do I build a Besta?`` keeps the same document suggested +#1 with the same cosine, before and after (the untouched phase-119 +path, test 3). + +Test → contract mapping (task 04 cases 1–4): +1. ``test_accented_question_suggests_the_small_source`` — (1): the + accented question grounds (``deflected: false``), the durable + record's SUGGESTED slice (entries 1–5, LOCKED A3) contains + ``ikea_instructions/besta-frame-white__AA-1272068-11-100.md``, and + the ``done`` frame's ``related`` list — the rank-6+ remainder — + does NOT (the doc moved from the rank-6 miss into the tier). A + zero-read grounded turn chips nothing (phase 119 LOCKED A1). +2. ``test_under_rank_is_measured_not_assumed`` — (1, the hypothesis): + with the PRE-FIX normalizer re-installed (``monkeypatch``), the same + retrieval over the same KB puts the doc OUT of the top-5 (and the + ``BESTÅ?`` token degrades to ``best``); with the shipped normalizer + it is suggested #1. The under-rank is measured in both directions, + so the pin can never silently rot. +3. ``test_ascii_question_still_name_matches`` — (2): the ASCII + regression (the phase-119 path untouched) — same document in the + suggested tier, grounded, same related remainder. +4. ``test_overview_tail_names_every_source_with_counts`` — (3): the + stored ``kb_overview`` row ENDS with the deterministic + ``Sources: …`` line, one entry per source with exact counts — the + 1-doc sources visible whatever the ``lite`` digest said. +5. ``test_out_of_kb_question_still_deflects`` — (4): the gate verdict + for the out-of-KB question is UNCHANGED (``deflected: true``, + ``fts_hits == 0``, no chips, ``Maybe try`` chips only) — the fold + must not turn a real miss into a false grounding. +6. ``test_grep_folds_the_accented_body`` — (3, content layer): + ``grep_document`` with the ASCII pattern ``besta`` matches the + stored accented line (D3), and the accented pattern matches too. +7. ``test_suggested_folder_lines_carry_the_small_source`` — the + prompt-level witness: the mock's ``repeat your folder map`` echo + ends with the suggested-folder lines, and the 1-doc source's line + ``ikea_instructions/:`` is among them (its suggested doc owns the + line, so the header stands alone). +""" +from __future__ import annotations + +import json +import os +import re +import subprocess +import sys +import time +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import httpx +import pytest +from playwright.sync_api import Locator, Page, expect +from sqlalchemy import select, text + +from app.config import Settings as _Settings +from app.db import SessionLocal +from app.models import Document, KbOverview, QueryLog +from app.rag import retriever +from app.rag.agent import grep_document +from app.rag.retriever import retrieve, select_related, select_suggested +from e2e.auth_helpers import login +from e2e.conftest import ( + ADMIN_PASSWORD, + SESSION_SECRET, + USE_REAL_LLM, + _wait_http, +) +from e2e.mock_llm import TOKEN_RE, embed_text + +REPO = Path(__file__).resolve().parents[2] + +#: Own port (the phase-79 full-inventory rule: a module app binds its +#: own port so a combined run never drives the session app). +APP_PORT = int(os.environ.get("E2E_APP_PORT_RECALL", "8162")) +APP_URL = f"http://127.0.0.1:{APP_PORT}" + +FIXTURES = REPO / "tests" / "fixtures" / "recallkb" +MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E" + +# -------------------------------------------------------------------------- +# The fixture KB (tracked files on disk — read back byte-for-byte) +# -------------------------------------------------------------------------- + +#: source name → document paths. THE SMALL SOURCE is the live incident: +#: one document whose body carries the product name only accented. +SOURCE_DOCS: dict[str, list[str]] = { + "ikea_instructions": ["besta-frame-white__AA-1272068-11-100.md"], + "phase_docs": [ + "140_pictorial_pdf_pages.md", + "148_pdf_text_backfill.md", + "152_seaweedfs_blob_ingest.md", + "156_history_source_filter.md", + ], + "players_handbook": ["game_night_setup.md"], +} + +IKEA_SOURCE = "ikea_instructions" +IKEA_DOC = "besta-frame-white__AA-1272068-11-100.md" +IKEA = f"{IKEA_SOURCE}/{IKEA_DOC}" + +#: Every fixture doc in ``(source, path)`` catalog order. +ALL_DOCS = [f"{source}/{path}" for source in SOURCE_DOCS for path in SOURCE_DOCS[source]] + +#: The deterministic source inventory tail (LOCKED A2 order: name-sorted, +#: singular ``doc`` at k=1) — the D2 assertion. +INVENTORY = "Sources: ikea_instructions (1 doc), phase_docs (4 docs), players_handbook (1 doc)" + +# -------------------------------------------------------------------------- +# The scripted questions +# -------------------------------------------------------------------------- + +#: The live miss, byte-for-byte (the ``Å`` is the whole point — the file +#: spells it ``BESTÅ``, the pre-phase name matcher read ``best``). +ACCENTED_QUESTION = "How do I build the BESTÅ?" +#: The phase-119 regression: the same product, spelled in ASCII. This +#: question ALWAYS name-matched (the untouched path). +ASCII_QUESTION = "How do I build a Besta?" +#: The accented question + the mock's folder-map trigger — the prompt +#: lens on the suggested tier (the folder lines are composed from the +#: suggested documents only). +FOLDER_QUESTION = ACCENTED_QUESTION + " repeat your folder map" +#: The out-of-KB control (the phase-119 battery's deflection sanity). +DEFLECT_QUESTION = "What is the capital of Mongolia?" + + +def _expected_summary(content: str, source: str, path: str) -> str: + """The mock lite model's byte-stable digest + the code pointer line + (mirrors ``mock_llm.compose_answer``'s ``SUMMARY_MODE`` branch — the + first 24 tokens of the document content — plus the summarizer's + deterministic ``Source:`` line; no model output is ever trusted).""" + digest = " ".join(TOKEN_RE.findall(content.lower())[:24]) + return f"This document covers {digest}.\nSource: {source}/{path}" + + +def _prefix_normalizer(value: str) -> str: + """The PRE-PHASE-162 name normalizer (the measured before-state): + lower + alnum scan with NO accent fold — the exact body + ``_normalize_name`` had before task 01. Installed by monkeypatch in + test 2 so the miss is MEASURED, not asserted from reasoning.""" + return "".join(retriever._ALNUM_RE.findall(value.lower())) + + +# -------------------------------------------------------------------------- +# Fixtures +# -------------------------------------------------------------------------- + + +@pytest.fixture(scope="module") +def seed_dirs() -> list[Path]: + """The three local-directory sources (the registered directory's + BASENAME is the source name, phase 38). The small source must be a + directory of its own — that is what makes it a 1-doc source.""" + dirs = [] + for source, paths in SOURCE_DOCS.items(): + directory = FIXTURES / source + assert directory.is_dir(), f"{directory} is missing" + for path in paths: + assert (directory / path).is_file(), f"fixture doc {source}/{path} is missing" + dirs.append(directory) + return dirs + + +@pytest.fixture(scope="module") +def app_server(mock_llm: int, seed_dirs: list[Path]) -> Iterator[str]: + """The real app under test — per-module app on its own port (the + phase-79 inventory pattern), the mock LLM, the mock-calibrated gate, + and the code defaults for everything the tiers depend on (suggested + 5 / related 2 / name-hit bonus 0.005 / recency 0.0007, exactly the + production ``.env``-free defaults).""" + env = dict(os.environ) + env.pop("DEBUGPY", None) + env["BOR_ENVIRONMENT"] = "e2e" + env["BOR_STATIC_DIR"] = str(REPO / "frontend") + env["BOR_LLM_BASE_URL"] = ( + "https://aipi.reeseapps.com/v1" + if USE_REAL_LLM + else f"http://127.0.0.1:{mock_llm}/v1" + ) + # Mock-calibrated gate (the conftest pattern): the accented and the + # ASCII turns ground on the distractors' cosine (≈0.35); the + # out-of-KB control stays under it. + env["BOR_RELEVANCE_THRESHOLD"] = "0.30" + env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15" + env["BOR_LLM_RETRY_DELAY"] = "0" + env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default) + env.setdefault( + "BOR_DATABASE_URL", + "postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese", + ) + env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD + env["BOR_SESSION_SECRET"] = SESSION_SECRET + # The repo's .env is the owner's — empty the git list so the registry + # holds EXACTLY the three directories this suite registers, and pin + # the leak-guarded corpus settings (the conftest pattern). + env["BOR_GIT_SOURCES"] = "" + env["BOR_DOCS_REPO"] = "" + env["BOR_SUGGESTIONS"] = json.dumps( + _Settings.model_fields["suggestions"].default + ) + env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default + env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default + proc = subprocess.Popen( + [sys.executable, "-m", "uvicorn", "app.main:app", + "--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"], + cwd=REPO, + env=env, + ) + try: + _wait_http(f"{APP_URL}/api/health") + yield APP_URL + finally: + proc.terminate() + try: + proc.wait(timeout=10) + except subprocess.TimeoutExpired: + proc.kill() + + +@pytest.fixture(scope="module") +def app_url(app_server: str) -> str: + return app_server + + +def _truncate_all() -> None: + """Fresh registry + KB (the E2E isolation pattern): the tier margins + are pinned against EXACTLY these six documents.""" + with SessionLocal() as db: + db.execute( + text( + "TRUNCATE chunks, documents, query_log, steering_notes, " + "kb_overview, git_sources, folder_summaries" + ) + ) + db.commit() + + +def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]: + """Poll the cookie-authenticated status endpoint to a terminal state + (the ``test_summary_seed_context`` pattern over plain httpx).""" + deadline = time.monotonic() + timeout_s + body: dict[str, Any] = {} + while time.monotonic() < deadline: + r = client.get("/api/sync/status") + assert r.status_code == 200, r.text + body = r.json() + if body["state"] in ("success", "failed"): + return body + time.sleep(0.5) + raise AssertionError(f"sync did not reach a terminal state: {body}") + + +def _tiers(question: str) -> tuple[list[str], list[str], float, int]: + """The measured tiers of one question over the synced KB, with the + app's REAL hybrid retrieval and the mock's embeddings (the + phase-119 measurement lens): ``(suggested, related, top cosine, + fts hits)``.""" + with SessionLocal() as db: + chunks = retrieve(db, question, embed_text(question)) + suggested = select_suggested(chunks) + related = select_related( + chunks, + {d.id for d in suggested}, + _Settings.model_fields["related_max_docs"].default, + ) + return ( + [f"{d.source}/{d.path}" for d in suggested], + [f"{d.source}/{d.path}" for d in related], + max((c.cosine for c in chunks), default=0.0), + sum(1 for c in chunks if c.fts_hit), + ) + + +def _assert_tiers( + question: str, suggested: list[str], related: list[str] +) -> None: + """Pin the engineered invariant for one scripted question, with a + MEASURED message (a fixture-text regression fails at setup, not at + the wire assertions).""" + got_suggested, got_related, cosine, fts = _tiers(question) + assert got_suggested == suggested, ( + f"suggested tier drifted for {question!r}: {got_suggested} " + f"(expected {suggested}; best cosine {cosine:.3f}, fts {fts})" + ) + assert got_related == related, ( + f"related tier drifted for {question!r}: {got_related} " + f"(expected {related})" + ) + + +@pytest.fixture(scope="module") +def synced_kb(app_server: str, seed_dirs: list[Path]) -> None: + """The story's precondition: the three-source KB synced under the + deterministic mock. Registers the directories through the + authenticated API, runs the REAL in-process sync, pins the stored + content byte-identical to the tracked fixture files (including the + accented ``BESTÅ`` surviving the round trip untouched), pins the + fixture's own token controls (the mechanism the miss is reproduced + with), and pins the engineered tiers for all three scripted + questions.""" + _truncate_all() + with httpx.Client(base_url=app_server, timeout=30.0) as client: + r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) + assert r.status_code == 204, r.text + for directory in seed_dirs: + r = client.post( + "/api/git-sources", json={"kind": "local", "path": str(directory)} + ) + assert r.status_code == 201, r.text + r = client.post("/api/sync") + assert r.status_code == 202, r.text + body = _wait_sync_done_http(client) + assert body["state"] == "success", body + assert body["detail"]["added"] == len(ALL_DOCS), body["detail"] + assert body["detail"]["pruned"] == 0, body["detail"] + + with SessionLocal() as db: + for source, paths in SOURCE_DOCS.items(): + for path in paths: + on_disk = (FIXTURES / source / path).read_text(encoding="utf-8") + stored = db.scalar( + select(Document).where( + Document.source == source, Document.path == path + ) + ) + assert stored is not None, f"fixture doc {source}/{path} missing" + assert stored.content == on_disk, f"content drifted for {source}/{path}" + assert stored.summary == _expected_summary(on_disk, source, path), ( + f"summary is not the mock digest for {source}/{path}" + ) + schunks = [c for c in stored.chunks if c.is_summary] + assert len(schunks) == 1 and schunks[0].embedding is not None, ( + f"expected one embedded is_summary chunk for {source}/{path}" + ) + + # ---- The fixture's token controls (the reproduction mechanism) ---- + ikea_text = (FIXTURES / IKEA_SOURCE / IKEA_DOC).read_text(encoding="utf-8") + # (a) the name exists ONLY accented — no ASCII ``besta`` in the body, + # so nothing but the phase-162 fold can bridge question ↔ doc; + assert "BESTÅ" in ikea_text + assert "besta" not in ikea_text.lower().replace("bestå", ""), ikea_text + # (b) no ``build`` either: the accented question's only OTHER FTS + # token — the small source must win on the NAME signal, never on + # a content-word accident; + assert "build" not in ikea_text.lower() + # (c) the accented name sits PAST the 24-token digest window, so the + # mock's ASCII-only digest never leaks a ``best`` lexeme the + # pre-phase FTS could have hit (a real ``lite`` summary of the + # live PDF never wrote the name in ASCII either). + digest = TOKEN_RE.findall(ikea_text.lower())[:24] + assert "best" not in digest and "besta" not in digest, digest + + # ---- The engineered tiers (measured, stable across re-imports) ---- + _assert_tiers(ACCENTED_QUESTION, SUGGESTED_ACCENTED, RELATED_ACCENTED) + _assert_tiers(ASCII_QUESTION, SUGGESTED_ASCII, RELATED_ASCII) + _assert_tiers(FOLDER_QUESTION, SUGGESTED_FOLDER, RELATED_FOLDER) + + +@pytest.fixture(autouse=True) +def _clean(db_ready: None) -> Iterator[None]: + """Per-test query_log isolation (the KB itself is module-scoped — the + scripted turns never change it).""" + with SessionLocal() as db: + db.execute(text("TRUNCATE query_log")) + db.commit() + yield + with SessionLocal() as db: + db.execute(text("TRUNCATE query_log")) + db.commit() + + +# -------------------------------------------------------------------------- +# The pinned tiers (MEASURED with the app's real retrieval over the mock +# embeddings — see the module docstring's before/after record) +# -------------------------------------------------------------------------- + +#: POST-FIX, accented question: the small source LEADS the tier (its +#: name hit leads the lexical list ⇒ ``fuse`` rank 1 = 1/61, plus the +#: per-document name-hit bonus 0.005), the five ASCII distractors +#: follow in measured retrieval order, one of them falls to rank 6. +SUGGESTED_ACCENTED = [ + IKEA, + "phase_docs/140_pictorial_pdf_pages.md", + "phase_docs/148_pdf_text_backfill.md", + "phase_docs/152_seaweedfs_blob_ingest.md", + "players_handbook/game_night_setup.md", +] +RELATED_ACCENTED = ["phase_docs/156_history_source_filter.md"] +#: POST-FIX, ASCII question (the untouched phase-119 path): same doc, +#: same #1 slot — the only difference is the measured order of two +#: distractors that trade places on the extra ``besta`` FTS term. +SUGGESTED_ASCII = [ + IKEA, + "phase_docs/140_pictorial_pdf_pages.md", + "phase_docs/148_pdf_text_backfill.md", + "players_handbook/game_night_setup.md", + "phase_docs/152_seaweedfs_blob_ingest.md", +] +RELATED_ASCII = ["phase_docs/156_history_source_filter.md"] +#: The folder-map turn (the trigger's own tokens dilute the cosine to +#: 0.296 — lexical support carries the gate) keeps the same tier. +SUGGESTED_FOLDER = list(SUGGESTED_ACCENTED) +RELATED_FOLDER = list(RELATED_ACCENTED) +#: PRE-FIX (the measured miss, test 2): the folded token never reaches +#: ``besta``, so the small source has ONE fused term (vector) against +#: every distractor's TWO (vector + FTS ``build``) and the five +#: distractors fill the tier — the IKEA doc is rank 6, i.e. the rank-6+ +#: ``related`` remainder. Exactly the live 2026-10-07 under-rank. +PRE_SUGGESTED_ACCENTED = [ + "phase_docs/140_pictorial_pdf_pages.md", + "phase_docs/148_pdf_text_backfill.md", + "phase_docs/152_seaweedfs_blob_ingest.md", + "players_handbook/game_night_setup.md", + "phase_docs/156_history_source_filter.md", +] +PRE_RELATED_ACCENTED = [IKEA] + + +# -------------------------------------------------------------------------- +# Page helpers (the phase-119 house patterns) +# -------------------------------------------------------------------------- + +#: Captures the raw SSE ``data:`` payloads of the /api/chat stream (a +#: response clone read in the background) — wire-level assertions on the +#: ``done`` frame, independent of the rendering. +SSE_HOOK = """ +() => { + if (window.__sseInstalled) return; + window.__sseInstalled = true; + window.__sseFrames = []; + const origFetch = window.fetch; + window.fetch = async function (...args) { + const res = await origFetch.apply(this, args); + try { + const url = typeof args[0] === 'string' ? args[0] : args[0].url; + if (url.includes('/api/chat')) { + res.clone().text().then((bodyText) => { + for (const block of bodyText.split('\\n\\n')) { + const line = block.trim(); + if (line.startsWith('data: ')) { + window.__sseFrames.push(line.slice(6)); + } + } + }); + } + } catch (e) { /* non-clonable responses: ignored */ } + return res; + }; +} +""" + + +def _install_page_hooks(page: Page) -> None: + page.evaluate(SSE_HOOK) + + +def _frames(page: Page) -> list[dict[str, Any]]: + """The SSE frames captured since the last submit, once the hook's + background read settles.""" + deadline = time.monotonic() + 30.0 + while True: + raw = page.evaluate("() => window.__sseFrames || []") + parsed = [json.loads(line) for line in raw if line] + if any(f.get("type") == "done" for f in parsed): + return parsed + if time.monotonic() > deadline: + raise AssertionError( + f"SSE hook captured no `done` frame (frames so far: " + f"{len(parsed)}) — hook install failed?" + ) + time.sleep(0.05) + + +def _done(page: Page) -> dict[str, Any]: + return next(f for f in _frames(page) if f.get("type") == "done") + + +def _tool_frames(frames: list[dict[str, Any]]) -> list[dict[str, Any]]: + return [f for f in frames if f.get("type") == "tool"] + + +def _submit(page: Page, question: str) -> None: + page.evaluate("window.__sseFrames = []") + page.fill("#message-input", question) + page.click("#send-btn") + expect(page.locator(".msg.user .bubble").last).to_contain_text(question) + + +def _wait_settled(page: Page) -> None: + expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000) + expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000) + expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000) + + +def _last_brain(page: Page) -> Locator: + return page.locator(".msg.brain").last + + +def _last_query_log() -> QueryLog: + with SessionLocal() as db: + rows = db.scalars(select(QueryLog)).all() + assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}" + return rows[0] + + +def _recorded_tiers(row: QueryLog) -> tuple[list[str], list[str]]: + """The durable record's tiers (LOCKED A3: the record is the suggested + tier followed by the rank-6+ remainder — the wire's only faithful + witness of the suggested set, since ``done.sources`` is the READ-DOCS + citation surface and ``done.suggestions`` is the deflection chips).""" + recorded = [s for s in row.sources.split(", ") if s] + return recorded[:5], recorded[5:] + + +# -------------------------------------------------------------------------- +# 1. (task 04 case 1) the accented question puts the 1-doc source in the +# suggested top-5 — and it is no longer the rank-6 miss +# -------------------------------------------------------------------------- + + +def test_accented_question_suggests_the_small_source( + page: Page, app_url: str, synced_kb: None, db_ready: None +) -> None: + """``How do I build the BESTÅ?`` grounds, and the suggested tier of + the turn carries ``ikea_instructions/besta-frame-white__AA-1272068- + 11-100.md`` — SUGGESTED #1, measured. The suggested set is pinned on + the wire through the durable record (LOCKED A3: ``sources`` = the + suggested 5, then the rank-6+ remainder): the IKEA doc sits inside + the first five entries, and the ``done`` frame's ``related`` list — + the rank-6+ tier, the slot it occupied as the live miss — is EMPTY + of it. A zero-read grounded turn chips nothing (phase 119 LOCKED + A1); the answer renders grounded.""" + page.set_default_timeout(30_000) + login(page, app_url, next="/") + _install_page_hooks(page) + + _submit(page, ACCENTED_QUESTION) + _wait_settled(page) + + frames = _frames(page) + assert _tool_frames(frames) == [], _tool_frames(frames) + done = _done(page) + assert done["deflected"] is False, done + assert done["sources"] == [], done["sources"] + # The miss was the IKEA doc landing in the rank-6+ remainder. It is + # not there anymore — the tier it belongs to is the suggested one. + related = [(s["source"], s["path"]) for s in done["related"]] + assert (IKEA_SOURCE, IKEA_DOC) not in related, done["related"] + # The ``suggestions`` field is the DEFLECTION contract (title chips + # under a weak turn) — a grounded turn carries none, so it can never + # be the tier witness (the durable record is, and is asserted below). + assert done["suggestions"] == [], done["suggestions"] + + bubble = _last_brain(page).locator(".bubble") + expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000) + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) + # The de-emphasized rank-6+ row renders — and the doc it used to + # hold (the miss) is a distractor now. + row = _last_query_log() + related_links = page.locator(".msg.brain .related-docs .related-doc") + expect(related_links).to_have_count(len(RELATED_ACCENTED)) + for i, path in enumerate(RELATED_ACCENTED): + expect(related_links.nth(i)).to_contain_text(path) + expect(related_links.filter(has_text=IKEA)).to_have_count(0) + + assert row.question == ACCENTED_QUESTION + assert row.deflected is False + suggested, related_record = _recorded_tiers(row) + assert suggested == SUGGESTED_ACCENTED, row.sources + assert related_record == RELATED_ACCENTED, row.sources + assert suggested[0] == IKEA, row.sources + + +# -------------------------------------------------------------------------- +# 2. (task 04 case 1, the hypothesis) the under-rank is MEASURED in both +# directions: pre-fix the doc loses the tier, post-fix it wins it +# -------------------------------------------------------------------------- + + +def test_under_rank_is_measured_not_assumed( + monkeypatch: pytest.MonkeyPatch, synced_kb: None, db_ready: None +) -> None: + """The phase-162 hypothesis, measured rather than asserted: with the + PRE-phase normalizer re-installed (``lower`` + alnum scan, no accent + fold) the accented question degrades to the token ``best`` — no name + hit — and the SAME retrieval over the SAME KB drops + ``ikea_instructions/besta-frame-white__AA-1272068-11-100.md`` out of + the suggested top-5 into the rank-6+ remainder (the live miss, + reproduced). With the shipped normalizer the folded ``besta`` token + name-matches the filename and the doc is suggested #1. Re-running + the measurement is the guard: if a future fixture or retriever + change ever makes the pre-phase path win, this test fails and the + recorded evidence is re-examined instead of trusted.""" + # The token side of the miss, measured. + assert retriever.name_hit_tokens(ACCENTED_QUESTION) == ["build", "besta"] + monkeypatch.setattr(retriever, "_normalize_name", _prefix_normalizer) + assert retriever.name_hit_tokens(ACCENTED_QUESTION) == ["build", "best"] + + pre_suggested, pre_related, _cosine, _fts = _tiers(ACCENTED_QUESTION) + assert IKEA not in pre_suggested, ( + f"the pre-phase path WINS the tier — the fixture no longer " + f"reproduces the miss: {pre_suggested}" + ) + assert pre_suggested == PRE_SUGGESTED_ACCENTED, pre_suggested + assert IKEA in pre_related, ( + f"the miss should surface as the rank-6+ remainder: {pre_related}" + ) + assert pre_related == PRE_RELATED_ACCENTED, pre_related + + monkeypatch.undo() + # And the same measurement with the shipped fold: the doc is #1. + post_suggested, post_related, _cosine, _fts = _tiers(ACCENTED_QUESTION) + assert post_suggested == SUGGESTED_ACCENTED + assert post_related == RELATED_ACCENTED + + +# -------------------------------------------------------------------------- +# 3. (task 04 case 2) the ASCII question keeps its phase-119 name hit +# -------------------------------------------------------------------------- + + +def test_ascii_question_still_name_matches( + page: Page, app_url: str, synced_kb: None, db_ready: None +) -> None: + """``How do I build a Besta?`` — the question the phase-119 rule + already answered — still grounds on the same document (the fold must + be behavior-preserving for the ASCII path: ``besta`` normalizes to + ``besta`` either way). Same tier position, same durable record, + grounded, nothing chipped on a zero-read turn.""" + page.set_default_timeout(30_000) + login(page, app_url, next="/") + _install_page_hooks(page) + + _submit(page, ASCII_QUESTION) + _wait_settled(page) + + frames = _frames(page) + assert _tool_frames(frames) == [], _tool_frames(frames) + done = _done(page) + assert done["deflected"] is False, done + assert done["sources"] == [], done["sources"] + expect(_last_brain(page).locator(".bubble")).to_contain_text( + MOCK_ANSWER_MARKER, timeout=30_000 + ) + expect(page.locator(".msg.brain .source-chip")).to_have_count(0) + + row = _last_query_log() + assert row.question == ASCII_QUESTION + assert row.deflected is False + suggested, related_record = _recorded_tiers(row) + assert suggested == SUGGESTED_ASCII, row.sources + assert suggested[0] == IKEA, row.sources + assert related_record == RELATED_ASCII, row.sources + + +# -------------------------------------------------------------------------- +# 4. (task 04 case 3) the deterministic source inventory — the D2 tail +# -------------------------------------------------------------------------- + + +def test_overview_tail_names_every_source_with_counts( + synced_kb: None, db_ready: None +) -> None: + """The stored ``kb_overview`` row ENDS with the mechanical + ``Sources: …`` line — one entry per source, exact counts, singular + ``doc`` at k=1 — so the 1-document sources are visible to + ```` whatever the lossy ``lite`` digest said (the + 2026-10-07 live finding: a 14-source index outlined as 3 groups). + The tail is ONE line, appears exactly once, and sits after the + model's outline (the model input never sees it).""" + with SessionLocal() as db: + rows = db.scalars(select(KbOverview)).all() + assert len(rows) == 1, rows + content = rows[0].content + lines = content.splitlines() + assert lines[-1] == INVENTORY, content + assert content.count("Sources: ") == 1, content + # The ``lite`` outline still leads (the tail is appended, never a + # replacement). + assert len(lines) >= 2, content + assert lines[0].startswith("Knowledge base outline:"), content + for source in SOURCE_DOCS: + assert source in lines[-1], content + + +# -------------------------------------------------------------------------- +# 5. (task 04 case 4) the out-of-KB control — the gate verdict unchanged +# -------------------------------------------------------------------------- + + +def test_out_of_kb_question_still_deflects( + page: Page, app_url: str, synced_kb: None, db_ready: None +) -> None: + """``What is the capital of Mongolia?`` matches nothing in this KB: + the A8 verdict stays DEFLECTED (``deflected: true``, ``fts_hits == + 0``, weak cosine under the mock-calibrated threshold), the answer + carries no citation chips, and the only chips are the ``Maybe try`` + suggestions. The accent fold must never turn a real miss into a + false grounding (the phase-119 battery's deflection sanity, + localised).""" + page.set_default_timeout(30_000) + login(page, app_url, next="/") + _install_page_hooks(page) + + _submit(page, DEFLECT_QUESTION) + _wait_settled(page) + + done = _done(page) + assert done["deflected"] is True, done + assert done["sources"] == [], done["sources"] + assert 2 <= len(done["suggestions"]) <= 3, done["suggestions"] + + bubble = _last_brain(page) + expect(bubble).to_have_class(re.compile(r"is-deflected")) + expect(bubble.locator(".source-chip")).to_have_count(0) + expect( + bubble.locator(".maybe-try .suggestion-chip") + ).to_have_count(len(done["suggestions"])) + + row = _last_query_log() + assert row.question == DEFLECT_QUESTION + assert row.deflected is True + assert row.fts_hits == 0, row.fts_hits + assert row.top_score < 0.30, row.top_score + + +# -------------------------------------------------------------------------- +# 6. (task 04 case 3 / D3) the stored accented body stays grep-matchable +# -------------------------------------------------------------------------- + + +def test_grep_folds_the_accented_body(synced_kb: None, db_ready: None) -> None: + """The content layer of the same miss: the stored body spells the + product ``BESTÅ`` only, and the agent's ``grep`` tool folds BOTH the + pattern and the line — so the ASCII pattern ``besta`` finds the + accented line (the pre-fold tool returned nothing, which is why the + live turn gave up on a document it had already opened), and the + accented pattern still matches itself.""" + with SessionLocal() as db: + stored = db.scalar( + select(Document).where( + Document.source == IKEA_SOURCE, Document.path == IKEA_DOC + ) + ) + assert stored is not None + assert "BESTÅ" in stored.content + + ascii_hits = grep_document(stored.content, "besta") + assert any("BESTÅ" in line for _no, line in ascii_hits), ascii_hits + accented_hits = grep_document(stored.content, "BESTÅ") + assert [line for _no, line in accented_hits] == [line for _no, line in ascii_hits] + # A pattern that is not the name still finds nothing (the fold is an + # accent bridge, not a wildcard). + assert grep_document(stored.content, "bestaa") == [] + + +# -------------------------------------------------------------------------- +# 7. the prompt-level witness — the 1-doc source's folder line rides the +# HIGH prompt with the suggested tier +# -------------------------------------------------------------------------- + + +def test_suggested_folder_lines_carry_the_small_source( + page: Page, app_url: str, synced_kb: None, db_ready: None +) -> None: + """The accented question + the mock's ``repeat your folder map`` + trigger ⇒ the answer ends with the suggested-folder lines VERBATIM. + The 1-doc source contributes ``ikea_instructions/:`` — a header with + no sibling entries, because its only document IS the suggested doc — + the visible proof the small source reached the prompt with the + tier.""" + page.set_default_timeout(30_000) + login(page, app_url, next="/") + _install_page_hooks(page) + + _submit(page, FOLDER_QUESTION) + _wait_settled(page) + + frames = _frames(page) + assert _tool_frames(frames) == [], _tool_frames(frames) + done = _done(page) + assert done["deflected"] is False, done + bubble = _last_brain(page).locator(".bubble") + expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000) + expect(bubble).to_contain_text("ikea_instructions/:") + + row = _last_query_log() + assert row.deflected is False + suggested, _related = _recorded_tiers(row) + assert suggested == SUGGESTED_FOLDER, row.sources diff --git a/tests/fixtures/recallkb/ikea_instructions/besta-frame-white__AA-1272068-11-100.md b/tests/fixtures/recallkb/ikea_instructions/besta-frame-white__AA-1272068-11-100.md new file mode 100644 index 0000000..9ee7393 --- /dev/null +++ b/tests/fixtures/recallkb/ikea_instructions/besta-frame-white__AA-1272068-11-100.md @@ -0,0 +1,25 @@ +# Assembly Instructions + +Article number 127.206.11 — revision 4 + +Cam bolt 130618 · wall bracket 141413 · leveling foot 303154 · dowel 518041 + +## Package contents + +- 1 x frame, white, 60 x 20 x 60 cm +- 2 x side panels +- 1 x back panel, hardboard +- 12 x cam bolts with sleeve nuts +- 1 bag of dowels + +## Assembly + +Insert sleeve nuts into the drilled holes, draw both side panels against the +frame, tighten each cam bolt a quarter turn. Attach back panel with nails +supplied, keeping it square and level. Fix wall bracket using plugs matched to +wall material — furniture of this depth must always be secured. + +## Product name + +BESTÅ frame, white, 60 x 20 x 60 cm, AA 1272068 11 100. Wipe with a soft +cloth dampened in water or a mild cleaner. Never soak the edges. diff --git a/tests/fixtures/recallkb/phase_docs/140_pictorial_pdf_pages.md b/tests/fixtures/recallkb/phase_docs/140_pictorial_pdf_pages.md new file mode 100644 index 0000000..54b9833 --- /dev/null +++ b/tests/fixtures/recallkb/phase_docs/140_pictorial_pdf_pages.md @@ -0,0 +1,11 @@ +# Pictorial PDF pages + +How do I build a retrievable text layer for a scanned manual? Build one page +image per PDF page, build an image description for each page, then build the +index from those descriptions. The ASCII besta assembly PDF was the first +manual to go through this path. + +- build page images at 1400 px +- build one description per page image +- build chunks from descriptions, never from pixel guesses +- besta manual: 24 scanned pages, 24 descriptions diff --git a/tests/fixtures/recallkb/phase_docs/148_pdf_text_backfill.md b/tests/fixtures/recallkb/phase_docs/148_pdf_text_backfill.md new file mode 100644 index 0000000..f12d600 --- /dev/null +++ b/tests/fixtures/recallkb/phase_docs/148_pdf_text_backfill.md @@ -0,0 +1,9 @@ +# PDF text backfill + +Backfill plan for scanned manuals. A manual that arrives without a text layer +needs one before it can answer anything: run the pictorial page path, build +descriptions for every page, keep the original page order. + +- build: render, describe, index +- besta shelf PDF: 24 scanned pages, no embedded text +- question asked twice in review: how do I build this layer? diff --git a/tests/fixtures/recallkb/phase_docs/152_seaweedfs_blob_ingest.md b/tests/fixtures/recallkb/phase_docs/152_seaweedfs_blob_ingest.md new file mode 100644 index 0000000..feb4a65 --- /dev/null +++ b/tests/fixtures/recallkb/phase_docs/152_seaweedfs_blob_ingest.md @@ -0,0 +1,8 @@ +# SeaweedFS blob ingest + +Object-store notes. Mount first, upload second. + +- mount fs://manuals before the first upload +- build the volume list, then copy the scan batch +- besta scan batch: fs://manuals/2026-09/besta-scans.tar +- checksum every archive after the copy diff --git a/tests/fixtures/recallkb/phase_docs/156_history_source_filter.md b/tests/fixtures/recallkb/phase_docs/156_history_source_filter.md new file mode 100644 index 0000000..a7a2edb --- /dev/null +++ b/tests/fixtures/recallkb/phase_docs/156_history_source_filter.md @@ -0,0 +1,8 @@ +# History source filter + +Turn history keeps the source each answer came from, so a filter can narrow +the log to one source at a time. + +- filter chip per source, sourced from turn history +- a build log entry records the source it indexed +- rerunning a question with a filter never rewrites history diff --git a/tests/fixtures/recallkb/players_handbook/game_night_setup.md b/tests/fixtures/recallkb/players_handbook/game_night_setup.md new file mode 100644 index 0000000..d403d03 --- /dev/null +++ b/tests/fixtures/recallkb/players_handbook/game_night_setup.md @@ -0,0 +1,7 @@ +# Game night setup + +Room notes, games side of the shelf wall. + +- besta unit holds box library; top shelf for dice trays +- build tray insert before next session +- keep rules binder beside clock, never stacked on besta top diff --git a/tests/fixtures/retrieval_battery.txt b/tests/fixtures/retrieval_battery.txt index 99c5fc3..54dc51c 100644 --- a/tests/fixtures/retrieval_battery.txt +++ b/tests/fixtures/retrieval_battery.txt @@ -1,4 +1,6 @@ -# Phase 119 retrieval battery (task 06, D6) — the 1,000-doc live replica. +# Retrieval battery — phase 119 (task 06, D6) + phase 162 (task 04, D5), +# run against the 1,000-doc live replica: +# uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt # # Gate (per the phase overview, post-phase measured numbers): # 1-3. each name question's suggested top-5 contains its canonical doc: @@ -10,8 +12,32 @@ # 5. deflection sanity: the out-of-KB question's gate verdict + fts # count match the pre-phase replica run (HIGH, fts_hits=5) — the # name-hit change must not move it. +# +# Phase 162 appends the Besta pair (the 2026-10-07 live miss). Gate: the +# suggested top-5 must contain the IKEA doc — ASCII filename +# `ikea_instructions/besta-frame-white__AA-1272068-11-100.pdf`, body that +# spells the product `BESTÅ` only — for BOTH spellings: the accented one +# needs the phase-162 question-side fold (D1) to bridge ASCII filename ← +# accented question, the ASCII one is the phase-119 path (regression). +# Either way ASCII `besta` documents must not outrank the doc that names +# the product. +# +# SKIPPED LOUDLY on the local replica (the phase-140 precedent): the +# 2026-09-05 snapshot carries NO `ikea_instructions`-class source — +# `grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0, and +# `SELECT DISTINCT source FROM documents` after the restore lists the +# sources without it (verify, don't assume). The two questions still +# run and must behave like any out-of-KB ask (no crash, no false +# grounding on a doc that never named the product); the gate on the +# rank itself can only close on a KB that has the source (owner's live +# KB). The fixture that DOES reproduce the miss end-to-end is +# tests/e2e/test_search_recall_small_sources.py over +# tests/fixtures/recallkb (pre-fix ranks measured there, both +# directions). How do I deploy gitea? How do I deploy forgejo? How do I set up the gateway? Do you have notes on the qwen 3.8 27b model? What is the capital of Mongolia? +How do I build a Besta? +How do I build the BESTÅ? diff --git a/tests/integration/test_import_docs_overview.py b/tests/integration/test_import_docs_overview.py index bb078de..c75b432 100644 --- a/tests/integration/test_import_docs_overview.py +++ b/tests/integration/test_import_docs_overview.py @@ -239,10 +239,11 @@ def test_changed_import_writes_overview_row( by_role = {m["role"]: m["content"] for m in llm.chat_calls[3]} assert "FOLDER_SUMMARY_MODE" in by_role["system"] assert by_role["user"].splitlines()[0] == "Folder: MyDocs" - # The model's outline lands in the single row. + # The model's outline lands in the single row, followed by the + # phase-162 deterministic source inventory (LOCKED A2). row = _row(db) assert row is not None - assert row.content == "Summary of MyDocs" + assert row.content == "Summary of MyDocs\nSources: MyDocs (2 docs)" assert row.updated_at is not None # The phase's required log line (PLAN §9). assert any("overview: regenerated docs=2 chars=" in r.getMessage() for r in records) @@ -274,7 +275,8 @@ def test_unchanged_reimport_does_not_call_lite( ) assert len(llm.chat_calls) == 4 # no new lite call row = _row(db) - assert row is not None and row.content == "Summary of MyDocs" + assert row is not None + assert row.content == "Summary of MyDocs\nSources: MyDocs (2 docs)" assert _version(db) == 1 # phase 53: an unchanged re-run never bumps @@ -345,7 +347,8 @@ def test_limit_run_skips_overview( # lite call; the overview + folder gates skip under --limit. assert len(llm.chat_calls) == 5 row = _row(db) - assert row is not None and row.content == "Summary of MyDocs" + assert row is not None + assert row.content == "Summary of MyDocs\nSources: MyDocs (2 docs)" assert _version(db) == 1 # phase 53: --limit debug runs never bump @@ -462,5 +465,8 @@ def test_overview_outline_names_the_newest_docs_when_the_cap_bites( # would have been "mydocs old 01 md old 01 summary of" (the oldest # batch) — the "summaries only up to phase 113" shape this pin # forbids. - assert row.content == "Knowledge base outline:\n- mydocs zz new 03 md zz new 03" + assert row.content == ( + "Knowledge base outline:\n- mydocs zz new 03 md zz new 03\n" + "Sources: MyDocs (6 docs)" + ) assert "old" not in row.content # no oldest-batch token may survive the cut diff --git a/tests/unit/test_agent.py b/tests/unit/test_agent.py index ef8b33a..9739dba 100644 --- a/tests/unit/test_agent.py +++ b/tests/unit/test_agent.py @@ -367,7 +367,8 @@ def test_agent_tools_names_and_parameters() -> None: # teaching no-match line instead of a trusted miss. assert grep["description"] == ( "Search the indexed documents for an exact string " - "(case-insensitive) and return up to 20 matching lines " + "(case- and accent-insensitive — accents fold: `besta` " + "matches `BESTÅ`) and return up to 20 matching lines " "as `source/path:line: text` — a locator, not a " "context-adder: read the winner with `read` — for a " "large document, use the reported line number as " @@ -387,8 +388,11 @@ def test_agent_tools_names_and_parameters() -> None: assert set(grep_params["properties"]) == {"pattern", "path"} assert all(p["type"] == "string" for p in grep_params["properties"].values()) assert grep_params["properties"]["pattern"]["description"] == ( - "The exact text to search for (a plain substring, " - "not a regex — no '.*', no '\\.', no character classes)" + "The exact text to search for (a plain " + "substring — case- and accent-insensitive, " + "accents fold: `besta` matches `BESTÅ` — " + "not a regex — no '.*', no " + "'\\.', no character classes)" ) # Phase 72 (task 02): the bare-path contract is stated up front; # task 05 (live gate iterations 1-8): the one-known-document clause @@ -4000,6 +4004,27 @@ def test_origin_lines_image_doc_keeps_marker_position( # ---------- grep (the phase-68 A5 contract under the new name) ---------- +def test_grep_document_folds_accents_both_ways() -> None: + """Phase 162 (LOCKED A3): grep folds BOTH the pattern and each line + through the shared ``app.rag.retriever._fold`` plus lowercase — an + ASCII pattern hits accented content and an accented pattern hits + ASCII content, one fold, never drifting from the name-hit path.""" + # ASCII pattern, accented content. + assert agent.grep_document("The BESTÅ frame (AA-1272068-11-100)", "besta") == [ + (1, "The BESTÅ frame (AA-1272068-11-100)") + ] + assert agent.grep_document("line one\nBÅsta shelf\nline three", "basta") == [ + (2, "BÅsta shelf") + ] + assert agent.grep_document("café-au-lait", "cafe") == [(1, "café-au-lait")] + # Accented pattern, ASCII content — the pattern's own accents fold. + assert agent.grep_document("The besta frame", "BESTÅ") == [(1, "The besta frame")] + # A genuinely non-matching folded pattern still returns no lines. + assert agent.grep_document("The besta frame", "bestb") == [] + # The fold bridges Ä/Ö/Ü/ß too (the same _fold as name matching). + assert agent.grep_document("Straße", "strasse") == [(1, "Straße")] + + def test_grep_document_case_insensitive_line_numbers() -> None: """Case-insensitive fixed substring, 1-based line numbers, file order, repeated matches within a line collapse to one match (grep semantics).""" diff --git a/tests/unit/test_overview.py b/tests/unit/test_overview.py index 9015faf..53b5db9 100644 --- a/tests/unit/test_overview.py +++ b/tests/unit/test_overview.py @@ -5,7 +5,11 @@ per-document user lines (source/path/title/first summary line), and the input cap with the shared ``[…truncated…]`` marker. The loader and regenerator tests run against the local compose Postgres (preferred — real single-row upsert), skipping with clear instructions when the stack -is not up — same pattern as ``test_importer.py``. +is not up — same pattern as ``test_importer.py``. The phase-162 section +pins the deterministic ``Sources: …`` inventory tail (LOCKED A2): the +format/order/counts, the 1-doc survival, the happy-path stored content, +the byte-identical zero-doc / ``LLMError`` paths, and the byte-identical +model input. """ from __future__ import annotations @@ -15,7 +19,7 @@ import uuid from datetime import UTC, datetime, timedelta import pytest -from sqlalchemy import text +from sqlalchemy import select, text from sqlalchemy.orm import Session from app.config import Settings, get_settings @@ -25,6 +29,7 @@ from app.rag.overview import ( KB_OVERVIEW_MODE, OVERVIEW_INSTRUCTION, SYSTEM_PROMPT, + _source_inventory, build_overview_prompt, load_kb_overview, regenerate_overview, @@ -302,11 +307,13 @@ def test_regenerate_happy_path_creates_single_row_id_1( row = db.get(KbOverview, 1) assert row is not None, "the single row must be upserted" assert row.id == 1 - assert row.content == REPLY + # Phase 162 (LOCKED A2): the stored content carries the + # deterministic source-inventory tail after the model text. + assert row.content == REPLY + "\nSources: Homelab (2 docs)" assert row.updated_at is not None age = datetime.now(UTC) - row.updated_at assert age.total_seconds() < 300, "updated_at must be a fresh UTC timestamp" - assert f"overview: regenerated docs=2 chars={len(REPLY)}" in caplog.text + assert f"overview: regenerated docs=2 chars={len(row.content)}" in caplog.text finally: _truncate_documents(db) @@ -324,7 +331,7 @@ def test_regenerate_updates_existing_row_in_place(db: Session, clean_overview) - assert count == 1, "still exactly one row after the re-generation" row = db.get(KbOverview, 1) assert row is not None - assert row.content == REPLY # replaced, not appended + assert row.content == REPLY + "\nSources: Homelab (1 doc)" # replaced, not appended assert row.updated_at is not None and row.updated_at > past finally: _truncate_documents(db) @@ -511,4 +518,135 @@ def test_regenerate_opens_own_session_when_none(db: Session, clean_overview) -> assert ok is True row = db.get(KbOverview, 1) # fresh lookup via the test session - assert row is not None and row.content == REPLY + assert row is not None and row.content == REPLY + "\nSources: Homelab (1 doc)" + + +# ---------- phase 162: the deterministic source inventory (LOCKED A2) ------- + + +def test_source_inventory_format_counts_and_catalog_order() -> None: + """Exact format, singular/plural, and catalog order: one entry per + DISTINCT source — `` ( doc|docs)`` joined with ``, `` + behind ``Sources: ``, ordered ``source, path`` (the scrambled + input rows prove the catalog order is the OUTPUT order, not the + recency-first order of the rows the generator reads).""" + rows = [ + ("b", "x.md", "X", None), + ("a", "y.md", "Y", None), + ("c", "p1.md", "P1", None), + ("b", "z.md", "Z", None), + ("c", "p2.md", "P2", None), + ("c", "p3.md", "P3", None), + ] + assert _source_inventory(rows) == "Sources: a (1 doc), b (2 docs), c (3 docs)" + + +def test_source_inventory_keeps_a_one_doc_source_visible() -> None: + """The 2026-10-07 ``ikea_instructions`` case: a 1-doc source among + many appears with ``(1 doc)`` — no source is invisible to + ````, whatever the ``lite`` outline says.""" + rows = [ + ("gufo", "a.md", "A", None), + ( + "ikea_instructions", + "besta-frame-white__AA-1272068-11-100.md", + "Assembly Instructions", + None, + ), + ("gufo", "b.md", "B", None), + ("players_handbook", "core.md", "Core", None), + ] + assert _source_inventory(rows) == ( + "Sources: gufo (2 docs), ikea_instructions (1 doc), " + "players_handbook (1 doc)" + ) + + +def test_regenerate_appends_the_inventory_tail_to_the_stored_content( + db: Session, clean_overview, caplog: pytest.LogCaptureFixture +) -> None: + """Happy path, exact pin on the fake's fixed output: the stored row + = model text (trailing newlines rstripped) + ``\\n`` + the inventory + line; the log's chars = the FULL stored content, tail included.""" + _truncate_documents(db) + try: + _add_doc(db, "players_handbook", "rpg.md", "RPG") + _add_doc(db, "Homelab", "b.md", "B") + _add_doc(db, "Homelab", "a.md", "A") + llm = _FakeLLM(reply=REPLY + "\n\n") + expected = REPLY + "\nSources: Homelab (2 docs), players_handbook (1 doc)" + with caplog.at_level(logging.INFO, logger="app.rag.overview"): + ok = asyncio.run(regenerate_overview(llm, db)) + assert ok is True + row = db.get(KbOverview, 1) + assert row is not None + assert row.content == expected # no blank line before the tail + assert f"overview: regenerated docs=3 chars={len(expected)}" in caplog.text + finally: + _truncate_documents(db) + + +def test_regenerate_model_input_never_gains_the_inventory_tail( + db: Session, clean_overview +) -> None: + """The model input is byte-identical to the pre-phase prompt: the + user message the fake received equals ``build_overview_prompt`` for + the same catalogue rows — the tail is never sent to ``lite``.""" + _truncate_documents(db) + try: + _add_doc(db, "Homelab", "a.md", "A", "A summary.\nSource: Homelab/a.md") + _add_doc(db, "Deployments", "b.yaml", "B") + llm = _FakeLLM() + assert asyncio.run(regenerate_overview(llm, db)) is True + rows = db.execute( + select(Document.source, Document.path, Document.title, Document.summary) + .order_by(Document.created_at.desc(), Document.source, Document.path) + ).all() + finally: + _truncate_documents(db) + + user = next(m["content"] for m in llm.messages if m["role"] == "user") + _, expected_user = build_overview_prompt([tuple(r) for r in rows]) + assert user == expected_user + assert "Sources:" not in user + + +def test_regenerate_zero_docs_keeps_tailless_pre_phase_row( + db: Session, clean_overview +) -> None: + """Zero-doc early return: byte-identical pre-phase behavior — the + pre-phase (tail-less) row survives untouched, no tail is invented.""" + db.execute(text("DELETE FROM kb_overview")) + pre_phase_row = "- Old outline\n- from before phase 162" + db.add(KbOverview(id=1, content=pre_phase_row)) + db.commit() + _truncate_documents(db) + llm = _FakeLLM() + ok = asyncio.run(regenerate_overview(llm, db)) + assert ok is False + assert llm.calls == 0 + row = db.get(KbOverview, 1) + assert row is not None and row.content == pre_phase_row + assert "Sources:" not in row.content + + +def test_regenerate_llm_error_preserves_an_existing_inventory_tail( + db: Session, clean_overview +) -> None: + """The tail is CONTENT: a previous row that already carried a + ``Sources: …`` tail survives an ``LLMError`` byte-identically — + including the tail (an old outline is better than none).""" + db.execute(text("DELETE FROM kb_overview")) + previous = "- Old outline\nSources: Homelab (3 docs)" + db.add(KbOverview(id=1, content=previous)) + db.commit() + _truncate_documents(db) + try: + _add_doc(db, "Homelab", "a.md", "A") + llm = _FakeLLM(fail=LLMError("simulated lite-model failure")) + ok = asyncio.run(regenerate_overview(llm, db)) + assert ok is False + row = db.get(KbOverview, 1) + assert row is not None and row.content == previous + finally: + _truncate_documents(db) diff --git a/tests/unit/test_prompt_lock.py b/tests/unit/test_prompt_lock.py index 0fe6186..5e33ff2 100644 --- a/tests/unit/test_prompt_lock.py +++ b/tests/unit/test_prompt_lock.py @@ -17,6 +17,7 @@ import hashlib from app.rag.prompts import ( ORIGIN_RULE, PERSONA, + SEARCH_DISCIPLINE_RULE, SUGGEST_INTRO, TOOLS_SECTION, _base, @@ -87,9 +88,34 @@ def test_high_and_low_bases_byte_locked() -> None: #: inserted — the prefix (the ``ls``-clause opening) and the suffix #: (the discipline-rules ending) survived byte-identical, so they are #: the same anchors as pre-phase-145; the pre-phase-145 sha/length are -#: kept below for the diff-style pin. -TOOLS_SECTION_SHA256 = "41a84885be98d4e15b95205694d59d364296a30e27758d39ae737881d5bacaa7" -TOOLS_SECTION_LEN = 3279 +#: kept below for the chained diff-style pin. RE-CUT again for phase 162 +#: (task 03, LOCKED A3+A4): the ``grep`` sentence's ``(case-insensitive)`` +#: parenthetical widened to the folded contract ``(case- and +#: accent-insensitive — accents fold: besta matches BESTÅ)``, and the +#: LOCKED search-discipline sentence (:data:`SEARCH_DISCIPLINE_RULE`) was +#: inserted immediately after the grep sentence (before the ``For a normal +#: search pass`` clause). Only those two byte-groups moved — the prefix +#: (the ``ls``-clause opening) and the suffix (the discipline-rules +#: ending) survive byte-identical; the pre-phase-162 sha/length are kept +#: below for the chained diff-style pin. +TOOLS_SECTION_SHA256 = "988642f9dc6bb2a4a1d44a83b5368ff4cdb9df4e181b2fe3dc2fb241b1857595" +TOOLS_SECTION_LEN = 3818 + +# The pre-phase-162 anchors — the section BEFORE phase 162's grep-fold + +# search-discipline insertion. Kept for the chained diff-style pin. +TOOLS_SECTION_PRE_PHASE_162_SHA256 = ( + "41a84885be98d4e15b95205694d59d364296a30e27758d39ae737881d5bacaa7" +) +TOOLS_SECTION_PRE_PHASE_162_LEN = 3279 + +# The grep contract parenthetical exactly as it changed in phase 162 +# (LOCKED A3): the pre-phase ``(case-insensitive)`` vs the folded one now +# embedded in ``TOOLS_SECTION``. Used to revert phase 162 for the diff pin. +_GREP_CONTRACT_PRE_PHASE_162 = "(case-insensitive)" +_GREP_CONTRACT_PHASE_162 = ( + "(case- and accent-insensitive — accents fold: `besta` matches `BESTÅ`)" +) + TOOLS_SECTION_PREFIX = ( "\n" "You may extend your context with three tools. `ls` lists the " @@ -159,19 +185,46 @@ def test_tools_section_origin_rule_verbatim_at_its_position() -> None: ) in TOOLS_SECTION -def test_tools_section_origin_rule_is_the_only_byte_change() -> None: - """Phase 145 (task 02) diff-style pin: the section's OTHER bytes are - byte-identical to pre-phase — the pre-phase section with ONLY the new - sentence-group inserted. Removing the rule (and the single space - that now separates it from the ``grep`` clause) reproduces the - pre-phase-145 section byte-for-byte (sha + length), and the length - delta is exactly the rule's length plus that one inserted space.""" - assert len(TOOLS_SECTION) == ( - TOOLS_SECTION_PRE_PHASE_145_LEN + len(ORIGIN_RULE) + 1 - ) - pre_phase = TOOLS_SECTION.replace(ORIGIN_RULE + " ", "") - assert len(pre_phase) == TOOLS_SECTION_PRE_PHASE_145_LEN - assert _sha256(pre_phase) == TOOLS_SECTION_PRE_PHASE_145_SHA256 +def test_tools_section_search_discipline_rule_verbatim_at_its_position() -> None: + """Phase 162 (task 03, LOCKED A4): the search-discipline rule is + LOCKED verbatim and sits at its EXACT position — immediately after + the grep sentence (the ``…retry with the plain text you expect to + see.`` clause) and BEFORE the ``For a normal search pass`` clause. + The rule + its exact boundaries are pinned as one contiguous span, so + a move or re-spelling of ANY byte fails.""" + assert SEARCH_DISCIPLINE_RULE in TOOLS_SECTION + assert TOOLS_SECTION.count(SEARCH_DISCIPLINE_RULE) == 1 # exactly once + assert ( + "retry with the plain text you expect to see. " + + SEARCH_DISCIPLINE_RULE + + " For a normal search pass only `pattern`" + ) in TOOLS_SECTION + + +def test_tools_section_grep_fold_contract_present() -> None: + """Phase 162 (task 03, LOCKED A3): the grep sentence now states the + FOLDED contract — the ``(case-insensitive)`` parenthetical is replaced + byte-for-byte by the folded one, and the old one is gone.""" + assert _GREP_CONTRACT_PHASE_162 in TOOLS_SECTION + assert _GREP_CONTRACT_PRE_PHASE_162 not in TOOLS_SECTION + + +def test_tools_section_phase162_and_145_are_the_only_byte_changes() -> None: + """Chained diff-style pin: the ONLY byte changes are phase 162's (the + folded grep contract + :data:`SEARCH_DISCIPLINE_RULE`) and phase 145's + (:data:`ORIGIN_RULE`). Reverting phase 162 (drop the rule + its + separating space, restore the pre-phase grep parenthetical) reproduces + the pre-phase-162 section byte-for-byte; reverting the origin rule on + top of that reproduces the pre-phase-145 section byte-for-byte.""" + # Revert phase 162 -> pre-phase-162. + pre_162 = TOOLS_SECTION.replace(SEARCH_DISCIPLINE_RULE + " ", "") + pre_162 = pre_162.replace(_GREP_CONTRACT_PHASE_162, _GREP_CONTRACT_PRE_PHASE_162) + assert len(pre_162) == TOOLS_SECTION_PRE_PHASE_162_LEN + assert _sha256(pre_162) == TOOLS_SECTION_PRE_PHASE_162_SHA256 + # Revert phase 145 -> pre-phase-145. + pre_145 = pre_162.replace(ORIGIN_RULE + " ", "") + assert len(pre_145) == TOOLS_SECTION_PRE_PHASE_145_LEN + assert _sha256(pre_145) == TOOLS_SECTION_PRE_PHASE_145_SHA256 # ---------- the LOW prompt's locked DEFLECT_MODE body ---------- diff --git a/tests/unit/test_prompts.py b/tests/unit/test_prompts.py index 0a2f2f2..2abd611 100644 --- a/tests/unit/test_prompts.py +++ b/tests/unit/test_prompts.py @@ -50,6 +50,7 @@ from app.models import Document from app.rag.prompts import ( ORIGIN_RULE, PERSONA, + SEARCH_DISCIPLINE_RULE, SHOW_IMAGE_CLAUSE, SUGGEST_INTRO, TOOLS_SECTION, @@ -286,14 +287,14 @@ def test_tools_section_markers_and_new_tool_names() -> None: def test_tools_section_teaches_the_harness_shapes() -> None: """Copy pins: ``ls``'s phase-63 catalog-line format, ``grep``'s -case-insensitive exact-string locator contract (up to 20 +case- and accent-insensitive exact-string locator contract (up to 20 ``source/path:line: text`` lines, a locator not a context-adder), and ``read``'s combined ``source/path`` + full content. (Phase 72: the source-name scope clause and the combined-identity clause are pinned byte-for-byte in :func:`test_tools_section_phase72_contract_clauses`.)""" assert "source: X | path: Y | title: Z" in TOOLS_SECTION - assert "case-insensitive" in TOOLS_SECTION + assert "case- and accent-insensitive" in TOOLS_SECTION # phase 162 fold assert "up to 20" in TOOLS_SECTION assert "source/path:line: text" in TOOLS_SECTION assert "locator, not a context-adder" in TOOLS_SECTION @@ -414,20 +415,28 @@ def test_tools_section_phase118_summary_seed_read_clause() -> None: "| path: Y | title: Z`; to find one specific document without " "listing, use `grep`." ) in TOOLS_SECTION - # The grep clause — byte-identical. + # The grep clause — folded contract (phase 162, LOCKED A3), up to the + # search-discipline sentence, byte-identical. assert ( - "`grep` locates an exact string (case-insensitive) in the " + "`grep` locates an exact string (case- and accent-insensitive — " + "accents fold: `besta` matches `BESTÅ`) in the " "indexed documents and returns up to 20 matching " "`source/path:line: text` lines — a locator, not a " "context-adder: read the winner with `read`. A grep pattern " "is a plain substring, NEVER a regex — '.*' and '\\.' are " "literal text there; if such a pattern returns no matches, " - "retry with the plain text you expect to see. For a normal " - "search pass only `pattern` — its optional `path` argument " - "limits the search to one document you already know, by the " - "same combined `source/path` string; never a source name — a " - "bare document path (without the source name) will not " - "resolve there either." + "retry with the plain text you expect to see. " + ) in TOOLS_SECTION + # The LOCKED search-discipline sentence sits immediately after the + # grep sentence, before the path clause (phase 162, LOCKED A4). + assert ( + "retry with the plain text you expect to see. " + + SEARCH_DISCIPLINE_RULE + + " For a normal search pass only `pattern` — its optional " + "`path` argument limits the search to one document you already " + "know, by the same combined `source/path` string; never a " + "source name — a bare document path (without the source name) " + "will not resolve there either." ) in TOOLS_SECTION # The discipline rules — byte-identical. assert ( diff --git a/tests/unit/test_retriever.py b/tests/unit/test_retriever.py index cf6357a..03822ad 100644 --- a/tests/unit/test_retriever.py +++ b/tests/unit/test_retriever.py @@ -1346,3 +1346,156 @@ def test_bonus_lives_in_the_selection_layer_only() -> None: rc.chunk_id: (rc.score, rc.cosine, rc.fts_hit, rc.name_hit) for rc in chunks } assert before == after + + +# --------------------------------------------------------------------------- +# Phase 162, D1 — the accent-folded name normalization (LOCKED A1): the +# 2026-10-07 live gap, the ASCII filename vs the accented content layer +# (``besta`` <-> ``BESTÅ``, both directions, ONE function) +# --------------------------------------------------------------------------- + +from app.rag.retriever import _fold, _name_parts # noqa: E402 + + +def test_fold_accent_ligature_and_sharp_s_cases() -> None: + """The shared accent fold (phase 162, LOCKED A1): NFKD decomposition, + combining marks dropped — ``Å/å → a``, ``Ä/Ö/Ü → a/o/u`` (the letter + survives, the mark drops), ``ß → ss`` (the Unicode fold — + ``str.casefold``; NFKD alone leaves ``ß`` intact), the ``fi`` ligature + ``→ fi``, ``ç → c``.""" + assert _fold("BESTÅ") == "besta" + assert _fold("Åsa") == "asa" + assert _fold("ÄÖÜäöü") == "aouaou" + assert _fold("Straße") == "strasse" + assert _fold("fine") == "fine" + assert _fold("garçon") == "garcon" + + +def test_fold_survives_non_decomposing_characters() -> None: + """Characters with NO decomposition (CJK) survive the fold unchanged — + the caller's alnum scan drops them, exactly as pre-phase. Compat-only + characters (``①``) NFKD to their ASCII base (``1``); that base is + below :data:`NAME_TOKEN_MIN_LEN`, so it never becomes a name token + (the name-signal behavior for such characters is unchanged).""" + assert _fold("漢字") == "漢字" + assert _fold("①") == "1" + + +def test_normalize_name_is_accent_folded() -> None: + """``_normalize_name`` = the alnum scan of ``_fold(s.lower())`` — + ``Qwen 3.8 → qwen38`` UNCHANGED, ``BESTÅ → besta`` NEW (pre-phase it + was ``best``); non-folding characters still drop out (``漢字 → ""``).""" + assert _normalize_name("Qwen 3.8") == "qwen38" + assert _normalize_name("BESTÅ") == "besta" + assert _normalize_name("BÅsta") == "basta" + assert _normalize_name("straße") == "strasse" + assert _normalize_name("fine") == "fine" + assert _normalize_name("garçon") == "garcon" + assert _normalize_name("漢字") == "" + # Compat-only base survives (``①`` → ``1``) but is a 1-char token — + # below the minimum, never a name candidate (see the token test). + assert _normalize_name("①") == "1" + + +def test_name_hit_tokens_accented_question_bridges() -> None: + """The question side of the bridge: the 2026-10-07 battery question + produces the ``besta`` token (pre-phase: ``best`` — the exact list is + pinned; ``how``/``do``/``i``/``the`` are below NAME_TOKEN_MIN_LEN). + The fold covers every pinned case: ``BESTÅ``, umlauts, ``ß``, the + ``fi`` ligature, ``ç``.""" + assert name_hit_tokens("How do I build the BESTÅ?") == ["build", "besta"] + assert name_hit_tokens("how do I fine the Über strasse") == [ + "fine", "uber", "strasse", + ] + assert name_hit_tokens("MÖRK ÖVER BÄSTA GARÇON") == [ + "mork", "over", "basta", "garcon", + ] + + +def test_name_hit_tokens_min_length_on_the_folded_form() -> None: + """NAME_TOKEN_MIN_LEN is measured on the FOLDED form (``BÄST`` → + ``bast`` (4, kept) while ``BÄ`` → ``ba`` (2, dropped)), and + non-folding tokens still drop out: ``漢字`` normalizes to ``""`` and + ``①`` to the 1-char ``1`` — neither can ever be a candidate. The + ``1`` is compat-decomposition (NFKD — LOCKED A1): the ``①`` token + itself never survives, and the only footprint a digit-base can leave + is a versioned-JOIN artifact (``the`` + ``1`` → ``the1``) whose sole + match route is PREFIXING a normalized path part — no real path does, + so the name signal for such characters is unchanged.""" + assert name_hit_tokens("BÄST BÄ") == ["bast"] + assert name_hit_tokens("tell me about it ① 漢字") == ["tell", "about"] + # The compat-decomposed ``1`` acts as a version token for the join — + # pinned as the real behavior (a prefix-class candidate no path hits). + assert name_hit_tokens("tell me about the ① 漢字") == [ + "tell", "about", "the1", + ] + + +def test_name_parts_fold_both_the_ascii_and_accented_stem() -> None: + """The path side of the bridge: the ASCII IKEA filename yields + ``besta`` in the EQUAL set (the stem sub-component — the phase-119 + A2 case, unchanged), and the accented filename ``BÅsta.md`` yields + ``basta`` (the folded stem — already in the equal set via the + normalized-stem member, no _name_parts code change).""" + _prefix, equal = _name_parts( + "ikea/besta-frame-white__AA-1272068-11-100.pdf" + ) + assert "besta" in equal # sub-component, unchanged + _prefix2, equal2 = _name_parts("manuals/BÅsta.md") + assert "basta" in equal2 # folded stem — an ASCII question matches it + + +def test_name_hit_chunks_accented_question_hits_ascii_doc() -> None: + """The bridge, end to end (direction 1): the accented question + ``BESTÅ`` name-hits the ASCII-stem document — the token ``besta`` + EQUALS the stem sub-component of + ``besta-frame-white__AA-1272068-11-100`` (the 2026-10-07 live miss; + pre-phase the token was ``best`` and the hit was impossible).""" + ikea = _doc( + "ikea/besta-frame-white__AA-1272068-11-100.pdf", + "MONTERING BÄSTA", + source="ikea_instructions", + ) + distractor = _doc("notes/meeting.md", "we discussed besta", source="Brain-of-Reese") + session = _FakeSession( + [_name_row(ikea), _name_row(distractor)], + [_name_hit_lateral_row(ikea)], + ) + out = _name_hit_chunks(session, "How do I build the BESTÅ?") # pyright: ignore[reportArgumentType] + assert [rc.document.path for rc in out] == [ikea.path] + assert out[0].name_hit is True + assert out[0].fts_hit is True + assert out[0].cosine == 0.0 + + +def test_name_hit_chunks_ascii_question_hits_accented_doc() -> None: + """The bridge, end to end (direction 2): the ASCII question token + ``basta`` name-hits the ACCENTED filename ``BÅsta.md`` — the folded + stem enters the equal set as ``basta`` (pre-phase the accented stem + normalized to ``bsta`` — the ``å`` simply dropped out of the alnum + scan — so the match was impossible).""" + accented = _doc("manuals/BÅsta.md", "MONTERING", source="ikea_instructions") + miss = _doc("manuals/hesta.md", "MONTERING", source="ikea_instructions") + session = _FakeSession( + [_name_row(accented), _name_row(miss)], + [_name_hit_lateral_row(accented)], + ) + out = _name_hit_chunks(session, "How do I assemble the basta shelf?") # pyright: ignore[reportArgumentType] + assert [rc.document.path for rc in out] == [accented.path] + assert out[0].name_hit is True + + +def test_name_hit_chunks_fold_preserves_digitless_precision() -> None: + """The fold does NOT relax the A2 match classes: the accented token + ``déploy`` folds to ``deploy`` and STILL does not hit the + ``Deployments/`` document (the part normalizes to ``deployments`` ≠ + ``deploy`` — titles are never matched, digits never enter); only the + exact ``gitea`` component hits.""" + precision = _doc("Deployments/reeseapps/README.md", "body", title="Deployments") + gitea = _doc("notes/gitea.md", "body") + session = _FakeSession( + [_name_row(precision), _name_row(gitea)], + [_name_hit_lateral_row(gitea)], + ) + out = _name_hit_chunks(session, "how do I déploy gitea") # pyright: ignore[reportArgumentType] + assert [rc.document.path for rc in out] == [gitea.path]